2024-04-27T03:05:07.703 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-27T03:05:07.703 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650 --name teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi --description rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} -- /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/orig.config.yaml 2024-04-27T03:05:07.736 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T03:05:07.813 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650 branch: squid description: rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7675650' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug rgw: 20 setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a rgw: ec-data-pool: true frontend: beast s3tests: slow_backend: true thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - osd.0 - - mgr.x - client.0 - osd.1 - osd.2 - osd.3 seed: 7340 sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a sleep_before_teardown: 0 subset: 0/1 suite: rgw suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHwh1HbopmUaqcyPIkoGT2EHqG1tebqFa/9riak5kf46Y1NVmqOVfiFSBvqjmEAg56m6gIJy6c1SqCp4YqJafN8= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC/wIFkg13J1eN7/FIQ6IiAuhaFcolUIVxsW/eOzj9LSwIBwiLSMtFcCQ29aZ4gqASebXvt1eD4xzCFKrd5hjn0= tasks: - install: null - ceph: conf: client: debug ms: 1 rgw gc obj min wait: 15 osd: debug ms: 1 debug objclass: 20 - rgw: client.0: null - radosgw-admin: null - radosgw-admin-rest: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_21:40:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-27T03:05:07.813 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-27T03:05:07.814 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-27T03:05:07.814 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-27T03:05:07.816 INFO:teuthology.task.internal:Checking packages... 2024-04-27T03:05:07.839 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a' 2024-04-27T03:05:07.839 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-27T03:05:07.839 INFO:teuthology.packaging:ref: None 2024-04-27T03:05:07.839 INFO:teuthology.packaging:tag: None 2024-04-27T03:05:07.839 INFO:teuthology.packaging:branch: squid 2024-04-27T03:05:07.839 INFO:teuthology.packaging:sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T03:05:07.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-27T03:05:08.060 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:05:08.061 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-27T03:05:08.071 INFO:teuthology.task.internal:no buildpackages task found 2024-04-27T03:05:08.071 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-27T03:05:08.126 INFO:teuthology.task.internal:Saving configuration 2024-04-27T03:05:08.138 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-27T03:05:08.192 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T03:05:08.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 02:59:25.381067', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFE9YQ4VDu030uKrrmOVTruZd8KorDVwyCSvM3xhSNZ8'} 2024-04-27T03:05:08.241 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-27 02:59:25.382499', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH9zNm6jUBOEC2DVbnYX5C/63T9CO/G2uaApqe/BLiwJ'} 2024-04-27T03:05:08.241 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-27T03:05:08.251 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'osd.0'] 2024-04-27T03:05:08.251 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mgr.x', 'client.0', 'osd.1', 'osd.2', 'osd.3'] 2024-04-27T03:05:08.251 INFO:teuthology.run_tasks:Running task console_log... 2024-04-27T03:05:08.347 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7eff8449d280>, signals=[15]) 2024-04-27T03:05:08.347 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-27T03:05:08.354 INFO:teuthology.task.internal:Opening connections... 2024-04-27T03:05:08.354 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T03:05:08.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:05:08.440 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:05:08.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:05:08.518 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-27T03:05:08.528 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-04-27T03:05:08.533 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-04-27T03:05:08.533 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-04-27T03:05:08.581 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-27T03:05:08.581 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Ubuntu" 2024-04-27T03:05:08.582 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="22.04" 2024-04-27T03:05:08.582 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-27T03:05:08.582 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_CODENAME=jammy 2024-04-27T03:05:08.582 INFO:teuthology.orchestra.run.smithi077.stdout:ID=ubuntu 2024-04-27T03:05:08.582 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE=debian 2024-04-27T03:05:08.583 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-27T03:05:08.583 INFO:teuthology.orchestra.run.smithi077.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-27T03:05:08.583 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-27T03:05:08.583 INFO:teuthology.orchestra.run.smithi077.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-27T03:05:08.583 INFO:teuthology.orchestra.run.smithi077.stdout:UBUNTU_CODENAME=jammy 2024-04-27T03:05:08.584 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-04-27T03:05:08.608 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-04-27T03:05:08.613 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-04-27T03:05:08.613 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-04-27T03:05:08.664 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-27T03:05:08.664 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2024-04-27T03:05:08.664 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="22.04" 2024-04-27T03:05:08.664 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-27T03:05:08.664 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=jammy 2024-04-27T03:05:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2024-04-27T03:05:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2024-04-27T03:05:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-27T03:05:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-27T03:05:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-27T03:05:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-27T03:05:08.665 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=jammy 2024-04-27T03:05:08.666 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-04-27T03:05:08.699 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-27T03:05:08.709 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-27T03:05:08.739 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-27T03:05:08.740 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-04-27T03:05:08.742 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-04-27T03:05:08.747 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-27T03:05:08.773 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-27T03:05:08.773 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-04-27T03:05:08.790 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-04-27T03:05:08.810 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-27T03:05:08.855 INFO:teuthology.run_tasks:Running task kernel... 2024-04-27T03:05:08.871 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-27T03:05:08.871 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-27T03:05:08.872 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T03:05:08.872 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-27T03:05:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:05:08.877 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-04-27T03:05:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:05:08.878 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-04-27T03:05:08.924 INFO:teuthology.orchestra.run.smithi110.stdout:5.15.0-56-generic 2024-04-27T03:05:08.924 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.15.0-56-generic 2024-04-27T03:05:08.925 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2024-04-27T03:05:08.925 INFO:teuthology.orchestra.run.smithi077.stdout:5.15.0-56-generic 2024-04-27T03:05:08.926 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.15.0-56-generic 2024-04-27T03:05:08.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get clean 2024-04-27T03:05:09.057 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2024-04-27T03:05:09.067 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2024-04-27T03:05:09.203 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-27T03:05:09.306 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-27T03:05:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-27T03:05:09.346 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-27T03:05:09.389 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-27T03:05:09.405 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-27T03:05:09.410 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-27T03:05:09.511 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-27T03:05:09.556 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-27T03:05:09.560 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-27T03:05:09.560 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-27T03:05:09.561 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-27T03:05:09.589 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-27T03:05:09.593 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-27T03:05:09.593 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-27T03:05:09.601 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-27T03:05:09.614 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-27T03:05:09.615 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-27T03:05:09.621 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-27T03:05:09.622 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-27T03:05:09.622 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-27T03:05:09.658 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-27T03:05:09.741 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-27T03:05:09.768 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-27T03:05:09.772 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-27T03:05:09.880 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-27T03:05:09.906 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-27T03:05:10.192 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-27T03:05:10.220 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-27T03:05:10.231 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-27T03:05:10.247 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-27T03:05:10.248 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-27T03:05:10.248 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-27T03:05:10.249 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-27T03:05:10.260 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-27T03:05:10.276 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-27T03:05:10.277 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-27T03:05:10.286 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-27T03:05:10.310 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-27T03:05:10.311 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-27T03:05:10.312 INFO:teuthology.orchestra.run.smithi077.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-27T03:05:10.318 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-27T03:05:10.338 INFO:teuthology.orchestra.run.smithi077.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-27T03:05:10.339 INFO:teuthology.orchestra.run.smithi077.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-27T03:05:10.346 INFO:teuthology.orchestra.run.smithi077.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-27T03:05:10.416 INFO:teuthology.orchestra.run.smithi077.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-27T03:05:10.417 INFO:teuthology.orchestra.run.smithi077.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-27T03:05:10.423 INFO:teuthology.orchestra.run.smithi077.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-27T03:05:10.423 INFO:teuthology.orchestra.run.smithi077.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-27T03:05:10.438 INFO:teuthology.orchestra.run.smithi077.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-27T03:05:10.482 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-27T03:05:10.518 INFO:teuthology.orchestra.run.smithi077.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-27T03:05:10.611 INFO:teuthology.orchestra.run.smithi077.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-27T03:05:11.062 INFO:teuthology.orchestra.run.smithi077.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-27T03:05:11.069 INFO:teuthology.orchestra.run.smithi077.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-27T03:05:11.071 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-27T03:05:11.092 INFO:teuthology.orchestra.run.smithi077.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-27T03:05:11.093 INFO:teuthology.orchestra.run.smithi077.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-27T03:05:11.105 INFO:teuthology.orchestra.run.smithi077.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-27T03:05:11.182 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-27T03:05:11.202 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-27T03:05:11.203 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-27T03:05:11.207 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-27T03:05:11.208 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-27T03:05:11.209 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-27T03:05:11.209 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-27T03:05:11.213 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-27T03:05:11.217 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-27T03:05:11.381 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-27T03:05:11.463 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-27T03:05:11.572 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-27T03:05:11.626 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-27T03:05:11.656 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-27T03:05:11.900 INFO:teuthology.orchestra.run.smithi077.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-27T03:05:12.205 INFO:teuthology.orchestra.run.smithi077.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-27T03:05:12.293 INFO:teuthology.orchestra.run.smithi077.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-27T03:05:12.315 INFO:teuthology.orchestra.run.smithi077.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-27T03:05:12.316 INFO:teuthology.orchestra.run.smithi077.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-27T03:05:12.385 INFO:teuthology.orchestra.run.smithi077.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-27T03:05:12.402 INFO:teuthology.orchestra.run.smithi077.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-27T03:05:12.402 INFO:teuthology.orchestra.run.smithi077.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-27T03:05:12.521 INFO:teuthology.orchestra.run.smithi077.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-27T03:05:12.564 INFO:teuthology.orchestra.run.smithi077.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-27T03:05:12.743 INFO:teuthology.orchestra.run.smithi077.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-27T03:05:12.826 INFO:teuthology.orchestra.run.smithi077.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-27T03:05:12.938 INFO:teuthology.orchestra.run.smithi077.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-27T03:05:12.991 INFO:teuthology.orchestra.run.smithi077.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-27T03:05:13.025 INFO:teuthology.orchestra.run.smithi077.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-27T03:05:16.018 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 13.2 MB in 3s (5,224 kB/s) 2024-04-27T03:05:17.202 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 13.2 MB in 4s (3,409 kB/s) 2024-04-27T03:05:17.514 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-27T03:05:17.532 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-27T03:05:17.594 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-27T03:05:17.794 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2024-04-27T03:05:17.794 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2024-04-27T03:05:17.969 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2024-04-27T03:05:17.969 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-27T03:05:17.969 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-27T03:05:17.970 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-27T03:05:17.970 INFO:teuthology.orchestra.run.smithi110.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-27T03:05:17.970 INFO:teuthology.orchestra.run.smithi110.stdout: python2.7-minimal 2024-04-27T03:05:17.970 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-27T03:05:17.971 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2024-04-27T03:05:17.971 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-27T03:05:17.971 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-27T03:05:17.972 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-27T03:05:17.973 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2024-04-27T03:05:17.973 INFO:teuthology.orchestra.run.smithi110.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-27T03:05:17.973 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2024-04-27T03:05:17.973 INFO:teuthology.orchestra.run.smithi110.stdout: thermald 2024-04-27T03:05:18.025 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2024-04-27T03:05:18.025 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-27T03:05:18.025 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-27T03:05:18.026 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-27T03:05:18.027 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2024-04-27T03:05:18.029 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-27T03:05:18.207 INFO:teuthology.orchestra.run.smithi110.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-27T03:05:18.207 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 113 MB of archives. 2024-04-27T03:05:18.207 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-27T03:05:18.207 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-27T03:05:18.750 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-04-27T03:05:18.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-27T03:05:18.823 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-04-27T03:05:19.033 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2024-04-27T03:05:19.034 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2024-04-27T03:05:19.199 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages were automatically installed and are no longer required: 2024-04-27T03:05:19.200 INFO:teuthology.orchestra.run.smithi077.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-27T03:05:19.200 INFO:teuthology.orchestra.run.smithi077.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-27T03:05:19.200 INFO:teuthology.orchestra.run.smithi077.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-27T03:05:19.201 INFO:teuthology.orchestra.run.smithi077.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-27T03:05:19.201 INFO:teuthology.orchestra.run.smithi077.stdout: python2.7-minimal 2024-04-27T03:05:19.201 INFO:teuthology.orchestra.run.smithi077.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-27T03:05:19.202 INFO:teuthology.orchestra.run.smithi077.stdout:The following additional packages will be installed: 2024-04-27T03:05:19.202 INFO:teuthology.orchestra.run.smithi077.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-27T03:05:19.202 INFO:teuthology.orchestra.run.smithi077.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-27T03:05:19.202 INFO:teuthology.orchestra.run.smithi077.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-27T03:05:19.204 INFO:teuthology.orchestra.run.smithi077.stdout:Suggested packages: 2024-04-27T03:05:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-27T03:05:19.204 INFO:teuthology.orchestra.run.smithi077.stdout:Recommended packages: 2024-04-27T03:05:19.204 INFO:teuthology.orchestra.run.smithi077.stdout: thermald 2024-04-27T03:05:19.254 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2024-04-27T03:05:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-27T03:05:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-27T03:05:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-27T03:05:19.254 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages will be upgraded: 2024-04-27T03:05:19.254 INFO:teuthology.orchestra.run.smithi077.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-27T03:05:19.309 INFO:teuthology.orchestra.run.smithi077.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-27T03:05:19.309 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 113 MB of archives. 2024-04-27T03:05:19.309 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-27T03:05:19.309 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-27T03:05:19.626 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-27T03:05:19.777 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-27T03:05:19.888 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-27T03:05:19.977 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-27T03:05:20.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-27T03:05:20.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-27T03:05:20.884 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-27T03:05:21.274 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-27T03:05:21.325 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-27T03:05:21.762 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 113 MB in 2s (54.8 MB/s) 2024-04-27T03:05:22.051 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-27T03:05:22.237 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-27T03:05:22.237 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-27T03:05:22.238 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-27T03:05:22.988 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-27T03:05:23.107 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-27T03:05:23.541 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 113 MB in 5s (22.3 MB/s) 2024-04-27T03:05:23.726 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-27T03:05:24.068 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-27T03:05:24.072 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:24.216 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:25.734 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-27T03:05:25.738 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:25.808 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:26.343 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-27T03:05:26.365 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:26.449 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:26.896 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-27T03:05:26.917 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:26.946 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:27.860 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-27T03:05:27.881 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:27.965 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:28.430 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-27T03:05:28.451 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:28.488 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:34.021 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-27T03:05:34.127 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-27T03:05:34.451 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-27T03:05:34.580 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-27T03:05:34.875 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-27T03:05:34.886 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-27T03:05:34.925 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-27T03:05:35.408 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-27T03:05:35.519 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-27T03:05:35.852 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-27T03:05:35.973 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-27T03:05:36.285 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-27T03:05:36.306 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-27T03:05:36.351 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-27T03:05:41.154 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-27T03:05:41.192 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:41.229 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:42.731 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-27T03:05:42.768 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-27T03:05:42.806 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:44.092 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-27T03:05:44.242 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-27T03:05:44.634 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-27T03:05:45.358 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:45.749 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-27T03:05:45.837 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-27T03:05:45.903 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:45.937 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-27T03:05:46.278 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-27T03:05:46.408 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:46.535 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-27T03:05:46.646 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:47.243 INFO:teuthology.orchestra.run.smithi077.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-27T03:05:47.243 INFO:teuthology.orchestra.run.smithi077.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-27T03:05:47.243 INFO:teuthology.orchestra.run.smithi077.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-27T03:05:47.243 INFO:teuthology.orchestra.run.smithi077.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-27T03:05:47.405 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:47.940 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-27T03:05:47.940 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-27T03:05:47.941 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-27T03:05:47.941 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-27T03:05:48.099 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:48.783 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:49.477 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:50.192 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-27T03:05:50.326 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-27T03:05:50.478 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:50.543 INFO:teuthology.orchestra.run.smithi077.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-27T03:05:50.544 INFO:teuthology.orchestra.run.smithi077.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-27T03:05:51.286 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-27T03:05:51.704 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-27T03:05:51.805 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-27T03:05:51.864 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-27T03:05:51.864 INFO:teuthology.orchestra.run.smithi110.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-27T03:06:04.574 INFO:teuthology.orchestra.run.smithi077.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-27T03:06:04.575 INFO:teuthology.orchestra.run.smithi077.stdout:Sourcing file `/etc/default/grub' 2024-04-27T03:06:04.593 INFO:teuthology.orchestra.run.smithi077.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-27T03:06:04.620 INFO:teuthology.orchestra.run.smithi077.stdout:Generating grub configuration file ... 2024-04-27T03:06:04.937 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-27T03:06:04.968 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-27T03:06:05.305 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-27T03:06:05.310 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-27T03:06:05.380 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-27T03:06:05.385 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-27T03:06:05.539 INFO:teuthology.orchestra.run.smithi077.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-27T03:06:05.539 INFO:teuthology.orchestra.run.smithi077.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-27T03:06:05.539 INFO:teuthology.orchestra.run.smithi077.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-27T03:06:05.571 INFO:teuthology.orchestra.run.smithi077.stdout:done 2024-04-27T03:06:06.030 DEBUG:teuthology.orchestra.run.smithi077:> dpkg -s linux-image-generic 2024-04-27T03:06:06.096 INFO:teuthology.orchestra.run.smithi077.stdout:Package: linux-image-generic 2024-04-27T03:06:06.096 INFO:teuthology.orchestra.run.smithi077.stdout:Status: install ok installed 2024-04-27T03:06:06.096 INFO:teuthology.orchestra.run.smithi077.stdout:Priority: optional 2024-04-27T03:06:06.096 INFO:teuthology.orchestra.run.smithi077.stdout:Section: kernel 2024-04-27T03:06:06.096 INFO:teuthology.orchestra.run.smithi077.stdout:Installed-Size: 21 2024-04-27T03:06:06.096 INFO:teuthology.orchestra.run.smithi077.stdout:Maintainer: Ubuntu Kernel Team 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: amd64 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout:Source: linux-meta 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout:Version: 5.15.0.105.102 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout:Recommends: thermald 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout:Description: Generic Linux kernel image 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout: This package will always depend on the latest generic kernel image 2024-04-27T03:06:06.097 INFO:teuthology.orchestra.run.smithi077.stdout: available. 2024-04-27T03:06:06.098 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-27T03:06:06.098 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-27T03:06:06.098 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-27T03:06:06.098 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-27T03:06:06.098 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2024-04-27T03:06:06.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get clean 2024-04-27T03:06:06.189 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-27T03:06:06.190 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub' 2024-04-27T03:06:06.202 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-27T03:06:06.219 INFO:teuthology.orchestra.run.smithi110.stdout:Generating grub configuration file ... 2024-04-27T03:06:06.282 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2024-04-27T03:06:06.524 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-27T03:06:06.525 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-27T03:06:06.543 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-27T03:06:06.563 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-27T03:06:06.597 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-27T03:06:06.679 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-27T03:06:06.899 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-27T03:06:06.905 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-27T03:06:06.975 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-27T03:06:06.981 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-27T03:06:07.149 INFO:teuthology.orchestra.run.smithi110.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-27T03:06:07.149 INFO:teuthology.orchestra.run.smithi110.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-27T03:06:07.149 INFO:teuthology.orchestra.run.smithi110.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-27T03:06:07.200 INFO:teuthology.orchestra.run.smithi110.stdout:done 2024-04-27T03:06:07.646 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 21 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2024-04-27T03:06:07.672 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.15.0.105.102 2024-04-27T03:06:07.673 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-27T03:06:07.673 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-27T03:06:07.673 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2024-04-27T03:06:07.673 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2024-04-27T03:06:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2024-04-27T03:06:07.673 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2024-04-27T03:06:07.673 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-27T03:06:07.673 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-27T03:06:07.673 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-27T03:06:07.673 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-27T03:06:07.673 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-04-27T03:06:07.673 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2024-04-27T03:06:07.913 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-04-27T03:06:07.930 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-27T03:06:07.940 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2024-04-27T03:06:07.984 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-04-27T03:06:08.070 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-27T03:06:08.071 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-27T03:06:08.082 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-27T03:06:08.102 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-27T03:06:08.160 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2024-04-27T03:06:08.160 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2024-04-27T03:06:08.316 INFO:teuthology.orchestra.run.smithi077.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-27T03:06:08.316 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages were automatically installed and are no longer required: 2024-04-27T03:06:08.316 INFO:teuthology.orchestra.run.smithi077.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-27T03:06:08.316 INFO:teuthology.orchestra.run.smithi077.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-27T03:06:08.316 INFO:teuthology.orchestra.run.smithi077.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-27T03:06:08.317 INFO:teuthology.orchestra.run.smithi077.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-27T03:06:08.317 INFO:teuthology.orchestra.run.smithi077.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-27T03:06:08.317 INFO:teuthology.orchestra.run.smithi077.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-27T03:06:08.355 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-27T03:06:08.357 DEBUG:teuthology.orchestra.run.smithi077:> dpkg -s linux-image-generic 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Package: linux-image-generic 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Status: install ok installed 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Priority: optional 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Section: kernel 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Installed-Size: 21 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Maintainer: Ubuntu Kernel Team 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: amd64 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Source: linux-meta 2024-04-27T03:06:08.415 INFO:teuthology.orchestra.run.smithi077.stdout:Version: 5.15.0.105.102 2024-04-27T03:06:08.416 INFO:teuthology.orchestra.run.smithi077.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-27T03:06:08.416 INFO:teuthology.orchestra.run.smithi077.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-27T03:06:08.416 INFO:teuthology.orchestra.run.smithi077.stdout:Recommends: thermald 2024-04-27T03:06:08.416 INFO:teuthology.orchestra.run.smithi077.stdout:Description: Generic Linux kernel image 2024-04-27T03:06:08.416 INFO:teuthology.orchestra.run.smithi077.stdout: This package will always depend on the latest generic kernel image 2024-04-27T03:06:08.416 INFO:teuthology.orchestra.run.smithi077.stdout: available. 2024-04-27T03:06:08.416 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-04-27T03:06:08.460 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.yBWHzLu1t9 2024-04-27T03:06:08.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub/grub.cfg /tmp/tmp.yBWHzLu1t9 2024-04-27T03:06:08.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.yBWHzLu1t9 2024-04-27T03:06:08.638 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.yBWHzLu1t9 is 10KB 2024-04-27T03:06:08.651 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.yBWHzLu1t9 2024-04-27T03:06:08.656 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-27T03:06:08.711 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:06:08.712 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-27T03:06:08.712 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-27T03:06:08.779 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-27T03:06:08.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo update-grub 2024-04-27T03:06:09.428 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-27T03:06:09.447 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-27T03:06:09.505 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-27T03:06:09.704 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2024-04-27T03:06:09.705 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2024-04-27T03:06:09.882 INFO:teuthology.orchestra.run.smithi110.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-27T03:06:09.882 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2024-04-27T03:06:09.882 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-27T03:06:09.882 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-27T03:06:09.882 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-27T03:06:09.883 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-27T03:06:09.883 INFO:teuthology.orchestra.run.smithi110.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-27T03:06:09.883 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-27T03:06:09.922 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-27T03:06:09.924 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 21 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2024-04-27T03:06:09.939 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2024-04-27T03:06:09.940 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.15.0.105.102 2024-04-27T03:06:09.940 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-27T03:06:09.940 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-27T03:06:09.940 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2024-04-27T03:06:09.940 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2024-04-27T03:06:09.940 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2024-04-27T03:06:09.940 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2024-04-27T03:06:09.940 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-04-27T03:06:09.984 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.lErLdre7OP 2024-04-27T03:06:09.984 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub/grub.cfg /tmp/tmp.lErLdre7OP 2024-04-27T03:06:10.039 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.lErLdre7OP 2024-04-27T03:06:10.046 INFO:teuthology.orchestra.run.smithi077.stderr:Sourcing file `/etc/default/grub' 2024-04-27T03:06:10.047 INFO:teuthology.orchestra.run.smithi077.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-27T03:06:10.050 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-04-27T03:06:10.152 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.lErLdre7OP is 10KB 2024-04-27T03:06:10.165 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.lErLdre7OP 2024-04-27T03:06:10.170 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-27T03:06:10.222 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:06:10.222 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-27T03:06:10.223 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-27T03:06:10.236 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-27T03:06:10.243 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-27T03:06:10.286 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-27T03:06:10.287 DEBUG:teuthology.orchestra.run.smithi110:> sudo update-grub 2024-04-27T03:06:10.529 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-27T03:06:10.535 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-27T03:06:10.579 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-27T03:06:10.584 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-27T03:06:10.731 INFO:teuthology.orchestra.run.smithi077.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-27T03:06:10.731 INFO:teuthology.orchestra.run.smithi077.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-27T03:06:10.731 INFO:teuthology.orchestra.run.smithi077.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-27T03:06:10.743 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-04-27T03:06:10.745 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-04-27T03:06:11.536 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub' 2024-04-27T03:06:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-27T03:06:11.539 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-04-27T03:06:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-27T03:06:11.718 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-27T03:06:11.937 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-27T03:06:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-27T03:06:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-27T03:06:11.986 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-27T03:06:12.112 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-27T03:06:12.113 INFO:teuthology.orchestra.run.smithi110.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-27T03:06:12.113 INFO:teuthology.orchestra.run.smithi110.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-27T03:06:12.128 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-04-27T03:06:12.147 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-04-27T03:06:40.767 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-27T03:06:40.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T03:06:40.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:06:42.151 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-27T03:06:42.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-27T03:06:42.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:06:59.237 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-04-27T03:07:00.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-04-27T03:07:08.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T03:07:08.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:07:09.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-27T03:07:09.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:07:11.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-04-27T03:07:12.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-04-27T03:07:23.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T03:07:23.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:07:24.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-27T03:07:24.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:07:41.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-04-27T03:07:43.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-04-27T03:07:56.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T03:07:56.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:07:57.363 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-04-27T03:07:58.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-27T03:07:58.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:07:58.246 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-04-27T03:07:58.528 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T03:07:58.529 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-27T03:07:58.529 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-04-27T03:07:58.535 INFO:teuthology.orchestra.run.smithi077.stdout:5.15.0-105-generic 2024-04-27T03:07:58.535 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-27T03:07:58.535 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-27T03:07:58.535 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-27T03:07:58.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-27T03:07:58.799 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-105-generic"... 2024-04-27T03:07:58.799 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-04-27T03:07:58.851 INFO:teuthology.orchestra.run.smithi110.stdout:5.15.0-105-generic 2024-04-27T03:07:58.851 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-27T03:07:58.851 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-27T03:07:58.851 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-27T03:07:59.536 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-27T03:07:59.536 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T03:07:59.839 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-04-27T03:07:59.852 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-27T03:07:59.852 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-27T03:07:59.858 DEBUG:teuthology.parallel:result is None 2024-04-27T03:08:00.191 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-04-27T03:08:00.217 DEBUG:teuthology.parallel:result is None 2024-04-27T03:08:00.217 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-27T03:08:00.225 INFO:teuthology.task.internal:Creating test directory... 2024-04-27T03:08:00.225 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T03:08:00.227 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-27T03:08:00.233 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-27T03:08:00.266 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-27T03:08:00.305 INFO:teuthology.task.internal:Creating archive directory... 2024-04-27T03:08:00.305 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T03:08:00.307 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-27T03:08:00.326 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-27T03:08:00.333 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-27T03:08:00.333 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T03:08:00.362 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-27T03:08:00.386 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:08:00.387 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:08:00.393 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:08:00.395 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-27T03:08:00.397 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-27T03:08:00.404 INFO:teuthology.task.internal:Configuring sudo... 2024-04-27T03:08:00.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T03:08:00.438 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-27T03:08:00.453 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-27T03:08:00.461 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-27T03:08:00.462 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T03:08:00.494 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-27T03:08:00.500 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T03:08:00.545 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T03:08:00.593 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:08:00.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T03:08:00.668 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-27T03:08:00.674 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-27T03:08:00.720 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:08:00.720 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-27T03:08:00.788 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-04-27T03:08:00.790 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-04-27T03:08:00.879 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-27T03:08:00.886 INFO:teuthology.task.internal:Starting timer... 2024-04-27T03:08:00.886 INFO:teuthology.run_tasks:Running task pcp... 2024-04-27T03:08:00.926 INFO:teuthology.run_tasks:Running task selinux... 2024-04-27T03:08:00.958 DEBUG:teuthology.task.selinux:Excluding smithi077: OS 'ubuntu' does not support SELinux 2024-04-27T03:08:00.959 DEBUG:teuthology.task.selinux:Excluding smithi110: OS 'ubuntu' does not support SELinux 2024-04-27T03:08:00.959 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-27T03:08:00.959 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-27T03:08:00.959 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-27T03:08:00.959 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-27T03:08:00.992 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-27T03:08:01.008 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-27T03:08:01.009 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-27T03:15:02.224 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2024-04-27T03:15:02.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T03:15:02.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:15:02.310 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-04-27T03:15:02.390 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-04-27T03:15:02.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-27T03:15:02.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-27T03:15:02.476 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-04-27T03:15:02.554 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-27T03:15:02.555 INFO:teuthology.run_tasks:Running task clock... 2024-04-27T03:15:02.566 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-27T03:15:02.566 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T03:15:02.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T03:15:02.569 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-27T03:15:02.569 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Command line: ntpd -gq 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: ---------------------------------------------------- 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: ntp-4 is maintained by Network Time Foundation, 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: corporation. Support and training for ntp-4 are 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: available at https://www.nwtime.org/support 2024-04-27T03:15:02.595 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: ---------------------------------------------------- 2024-04-27T03:15:02.596 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: proto: precision = 0.062 usec (-24) 2024-04-27T03:15:02.596 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: basedate set to 2022-02-04 2024-04-27T03:15:02.596 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: gps base set to 2022-02-06 (week 2196) 2024-04-27T03:15:02.596 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-27T03:15:02.597 INFO:teuthology.orchestra.run.smithi077.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-27T03:15:02.597 INFO:teuthology.orchestra.run.smithi077.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: restrict ::: KOD does nothing without LIMITED. 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Listen and drop on 0 v6wildcard [::]:123 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Listen normally on 2 lo 127.0.0.1:123 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Listen normally on 3 enp3s0f1 172.21.15.77:123 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Listen normally on 4 lo [::1]:123 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3d3%5]:123 2024-04-27T03:15:02.598 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:02 ntpd[17247]: Listening on routing socket on fd #22 for interface updates 2024-04-27T03:15:02.629 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-27T03:15:02.629 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Command line: ntpd -gq 2024-04-27T03:15:02.629 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: ---------------------------------------------------- 2024-04-27T03:15:02.629 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: ntp-4 is maintained by Network Time Foundation, 2024-04-27T03:15:02.629 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-27T03:15:02.629 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: corporation. Support and training for ntp-4 are 2024-04-27T03:15:02.629 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: available at https://www.nwtime.org/support 2024-04-27T03:15:02.630 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: ---------------------------------------------------- 2024-04-27T03:15:02.630 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: proto: precision = 0.060 usec (-24) 2024-04-27T03:15:02.630 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: basedate set to 2022-02-04 2024-04-27T03:15:02.630 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: gps base set to 2022-02-06 (week 2196) 2024-04-27T03:15:02.631 INFO:teuthology.orchestra.run.smithi110.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-27T03:15:02.631 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-27T03:15:02.631 INFO:teuthology.orchestra.run.smithi110.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-27T03:15:02.632 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: restrict ::: KOD does nothing without LIMITED. 2024-04-27T03:15:02.632 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Listen and drop on 0 v6wildcard [::]:123 2024-04-27T03:15:02.632 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-27T03:15:02.633 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Listen normally on 2 lo 127.0.0.1:123 2024-04-27T03:15:02.633 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Listen normally on 3 enp3s0f1 172.21.15.110:123 2024-04-27T03:15:02.633 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Listen normally on 4 lo [::1]:123 2024-04-27T03:15:02.633 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc35%5]:123 2024-04-27T03:15:02.633 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:02 ntpd[17244]: Listening on routing socket on fd #22 for interface updates 2024-04-27T03:15:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:03 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:03.595 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:03 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:03 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:03 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:03 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:03 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:03 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:03 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:04.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:04 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:04.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:04 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:04.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:04 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:04.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:04 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:04.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:04 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:04.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:04 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:04 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:05.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:05.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:05.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:05.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:05 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:06 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:06 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:06 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:06 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:06 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:06.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:06 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:06.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:06 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:06.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:06 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:06.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:06 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:06.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:06 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:06.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:06 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:06.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:06 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:07.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:07 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:07.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:07.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:07 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:08.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:08 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:08.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:08 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:08.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:08 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:08.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:08 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:08.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:08 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:08.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:08 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:08 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:08 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:08 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:08 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:08 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:08 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:09.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:09.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:09.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:09.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:09.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:09 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:09.630 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:09.631 INFO:teuthology.orchestra.run.smithi110.stdout:27 Apr 03:15:09 ntpd[17244]: ntpd: time slew -0.000306 s 2024-04-27T03:15:09.631 INFO:teuthology.orchestra.run.smithi110.stdout:ntpd: time slew -0.000306s 2024-04-27T03:15:09.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-27T03:15:09.631 INFO:teuthology.orchestra.run.smithi110.stderr:27 Apr 03:15:09 ntpd[17244]: can't open /var/log/ntpstats/loopstats.20240427: Permission denied 2024-04-27T03:15:09.686 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2024-04-27T03:15:09.687 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2024-04-27T03:15:09.687 INFO:teuthology.orchestra.run.smithi110.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:09.687 INFO:teuthology.orchestra.run.smithi110.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:09.687 INFO:teuthology.orchestra.run.smithi110.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:09.687 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: can't open /var/log/ntpstats/rawstats.20240427: Permission denied 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: can't open /var/log/ntpstats/peerstats.20240427: Permission denied 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stdout:27 Apr 03:15:10 ntpd[17247]: ntpd: time slew +0.001955 s 2024-04-27T03:15:10.596 INFO:teuthology.orchestra.run.smithi077.stdout:ntpd: time slew +0.001955s 2024-04-27T03:15:10.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-27T03:15:10.597 INFO:teuthology.orchestra.run.smithi077.stderr:27 Apr 03:15:10 ntpd[17247]: can't open /var/log/ntpstats/loopstats.20240427: Permission denied 2024-04-27T03:15:10.651 INFO:teuthology.orchestra.run.smithi077.stdout: remote refid st t when poll reach delay offset jitter 2024-04-27T03:15:10.651 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================== 2024-04-27T03:15:10.651 INFO:teuthology.orchestra.run.smithi077.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:10.651 INFO:teuthology.orchestra.run.smithi077.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:10.651 INFO:teuthology.orchestra.run.smithi077.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:10.651 INFO:teuthology.orchestra.run.smithi077.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:15:10.652 INFO:teuthology.run_tasks:Running task install... 2024-04-27T03:15:10.661 DEBUG:teuthology.task.install:project ceph 2024-04-27T03:15:10.661 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}} 2024-04-27T03:15:10.661 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'} 2024-04-27T03:15:10.661 INFO:teuthology.task.install:Using flavor: default 2024-04-27T03:15:10.668 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-27T03:15:10.668 INFO:teuthology.task.install:extra packages: [] 2024-04-27T03:15:10.668 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-key list | grep Ceph 2024-04-27T03:15:10.669 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-key list | grep Ceph 2024-04-27T03:15:10.732 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-27T03:15:10.748 INFO:teuthology.orchestra.run.smithi077.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-27T03:15:10.761 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-27T03:15:10.761 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph.com (release key) 2024-04-27T03:15:10.762 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-27T03:15:10.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T03:15:10.776 INFO:teuthology.orchestra.run.smithi077.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-27T03:15:10.776 INFO:teuthology.orchestra.run.smithi077.stdout:uid [ unknown] Ceph.com (release key) 2024-04-27T03:15:10.776 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-27T03:15:10.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T03:15:10.917 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default/ 2024-04-27T03:15:10.917 INFO:teuthology.task.install.deb:Package version is 19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:15:10.987 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default/ 2024-04-27T03:15:10.987 INFO:teuthology.task.install.deb:Package version is 19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:15:11.087 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:15:11.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-27T03:15:11.101 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2024-04-27T03:15:11.152 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:15:11.152 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-27T03:15:11.166 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2024-04-27T03:15:11.317 INFO:teuthology.orchestra.run.smithi077.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy InRelease 2024-04-27T03:15:11.338 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-04-27T03:15:11.356 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-27T03:15:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-27T03:15:11.367 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-27T03:15:11.383 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-04-27T03:15:11.387 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-27T03:15:11.414 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-27T03:15:11.446 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy InRelease 2024-04-27T03:15:11.469 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-27T03:15:11.477 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-27T03:15:11.477 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-27T03:15:11.482 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-04-27T03:15:11.519 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-27T03:15:11.550 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-27T03:15:11.555 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-04-27T03:15:11.599 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-27T03:15:11.632 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-27T03:15:11.722 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 28.3 kB in 0s (74.0 kB/s) 2024-04-27T03:15:11.825 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 28.3 kB in 1s (51.4 kB/s) 2024-04-27T03:15:12.653 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-27T03:15:12.668 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2447-gb22e2ebd-1jammy cephadm=19.0.0-2447-gb22e2ebd-1jammy ceph-mds=19.0.0-2447-gb22e2ebd-1jammy ceph-mgr=19.0.0-2447-gb22e2ebd-1jammy ceph-common=19.0.0-2447-gb22e2ebd-1jammy ceph-fuse=19.0.0-2447-gb22e2ebd-1jammy ceph-test=19.0.0-2447-gb22e2ebd-1jammy ceph-volume=19.0.0-2447-gb22e2ebd-1jammy radosgw=19.0.0-2447-gb22e2ebd-1jammy python3-rados=19.0.0-2447-gb22e2ebd-1jammy python3-rgw=19.0.0-2447-gb22e2ebd-1jammy python3-cephfs=19.0.0-2447-gb22e2ebd-1jammy python3-rbd=19.0.0-2447-gb22e2ebd-1jammy libcephfs2=19.0.0-2447-gb22e2ebd-1jammy libcephfs-dev=19.0.0-2447-gb22e2ebd-1jammy librados2=19.0.0-2447-gb22e2ebd-1jammy librbd1=19.0.0-2447-gb22e2ebd-1jammy rbd-fuse=19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:15:12.730 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2024-04-27T03:15:12.768 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-04-27T03:15:12.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2447-gb22e2ebd-1jammy cephadm=19.0.0-2447-gb22e2ebd-1jammy ceph-mds=19.0.0-2447-gb22e2ebd-1jammy ceph-mgr=19.0.0-2447-gb22e2ebd-1jammy ceph-common=19.0.0-2447-gb22e2ebd-1jammy ceph-fuse=19.0.0-2447-gb22e2ebd-1jammy ceph-test=19.0.0-2447-gb22e2ebd-1jammy ceph-volume=19.0.0-2447-gb22e2ebd-1jammy radosgw=19.0.0-2447-gb22e2ebd-1jammy python3-rados=19.0.0-2447-gb22e2ebd-1jammy python3-rgw=19.0.0-2447-gb22e2ebd-1jammy python3-cephfs=19.0.0-2447-gb22e2ebd-1jammy python3-rbd=19.0.0-2447-gb22e2ebd-1jammy libcephfs2=19.0.0-2447-gb22e2ebd-1jammy libcephfs-dev=19.0.0-2447-gb22e2ebd-1jammy librados2=19.0.0-2447-gb22e2ebd-1jammy librbd1=19.0.0-2447-gb22e2ebd-1jammy rbd-fuse=19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:15:12.847 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-04-27T03:15:12.929 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2024-04-27T03:15:12.929 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2024-04-27T03:15:13.044 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2024-04-27T03:15:13.045 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2024-04-27T03:15:13.123 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2024-04-27T03:15:13.124 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-27T03:15:13.124 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-27T03:15:13.124 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-27T03:15:13.124 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-27T03:15:13.125 INFO:teuthology.orchestra.run.smithi110.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-27T03:15:13.125 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-27T03:15:13.125 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2024-04-27T03:15:13.125 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-27T03:15:13.125 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-27T03:15:13.125 INFO:teuthology.orchestra.run.smithi110.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-27T03:15:13.125 INFO:teuthology.orchestra.run.smithi110.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-27T03:15:13.126 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-27T03:15:13.126 INFO:teuthology.orchestra.run.smithi110.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-27T03:15:13.127 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet zip 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-27T03:15:13.128 INFO:teuthology.orchestra.run.smithi110.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-27T03:15:13.129 INFO:teuthology.orchestra.run.smithi110.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-27T03:15:13.129 INFO:teuthology.orchestra.run.smithi110.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-27T03:15:13.129 INFO:teuthology.orchestra.run.smithi110.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-27T03:15:13.129 INFO:teuthology.orchestra.run.smithi110.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-27T03:15:13.129 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2024-04-27T03:15:13.129 INFO:teuthology.orchestra.run.smithi110.stdout: btrfs-tools 2024-04-27T03:15:13.228 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2024-04-27T03:15:13.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-27T03:15:13.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-27T03:15:13.228 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-27T03:15:13.229 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-27T03:15:13.229 INFO:teuthology.orchestra.run.smithi110.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-27T03:15:13.229 INFO:teuthology.orchestra.run.smithi110.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-27T03:15:13.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-27T03:15:13.231 INFO:teuthology.orchestra.run.smithi110.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-27T03:15:13.240 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages were automatically installed and are no longer required: 2024-04-27T03:15:13.240 INFO:teuthology.orchestra.run.smithi077.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-27T03:15:13.240 INFO:teuthology.orchestra.run.smithi077.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-27T03:15:13.240 INFO:teuthology.orchestra.run.smithi077.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-27T03:15:13.241 INFO:teuthology.orchestra.run.smithi077.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-27T03:15:13.241 INFO:teuthology.orchestra.run.smithi077.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-27T03:15:13.241 INFO:teuthology.orchestra.run.smithi077.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-27T03:15:13.241 INFO:teuthology.orchestra.run.smithi077.stdout:The following additional packages will be installed: 2024-04-27T03:15:13.241 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-27T03:15:13.241 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-27T03:15:13.243 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-27T03:15:13.244 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet zip 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout:Suggested packages: 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout:Recommended packages: 2024-04-27T03:15:13.245 INFO:teuthology.orchestra.run.smithi077.stdout: btrfs-tools 2024-04-27T03:15:13.312 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-27T03:15:13.312 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 179 MB of archives. 2024-04-27T03:15:13.312 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-27T03:15:13.313 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-27T03:15:13.347 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2024-04-27T03:15:13.347 INFO:teuthology.orchestra.run.smithi077.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-27T03:15:13.347 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-27T03:15:13.348 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-27T03:15:13.348 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-27T03:15:13.348 INFO:teuthology.orchestra.run.smithi077.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-27T03:15:13.348 INFO:teuthology.orchestra.run.smithi077.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-27T03:15:13.349 INFO:teuthology.orchestra.run.smithi077.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-27T03:15:13.349 INFO:teuthology.orchestra.run.smithi077.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-27T03:15:13.349 INFO:teuthology.orchestra.run.smithi077.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-27T03:15:13.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-27T03:15:13.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-27T03:15:13.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-27T03:15:13.351 INFO:teuthology.orchestra.run.smithi077.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-27T03:15:13.441 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-27T03:15:13.442 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 179 MB of archives. 2024-04-27T03:15:13.442 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-27T03:15:13.442 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-27T03:15:13.478 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-27T03:15:13.483 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-27T03:15:13.490 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2447-gb22e2ebd-1jammy [3,544 kB] 2024-04-27T03:15:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2447-gb22e2ebd-1jammy [3,544 kB] 2024-04-27T03:15:13.523 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-27T03:15:13.644 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-27T03:15:13.651 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-27T03:15:13.657 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-27T03:15:13.663 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-27T03:15:13.682 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-27T03:15:13.689 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-27T03:15:13.690 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-27T03:15:13.691 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-27T03:15:13.701 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-27T03:15:13.706 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-27T03:15:13.711 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-27T03:15:13.715 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-27T03:15:13.718 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-27T03:15:13.762 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-27T03:15:13.765 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-27T03:15:13.769 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-27T03:15:13.772 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-27T03:15:13.774 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-27T03:15:13.774 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-27T03:15:13.776 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-27T03:15:13.805 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-27T03:15:13.806 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-27T03:15:13.806 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-27T03:15:13.852 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-27T03:15:13.854 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-27T03:15:13.855 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-27T03:15:13.855 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-27T03:15:13.859 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-27T03:15:13.861 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-27T03:15:13.862 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-27T03:15:13.863 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-27T03:15:13.863 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-27T03:15:13.881 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-27T03:15:13.890 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-27T03:15:13.891 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-27T03:15:13.892 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-27T03:15:13.905 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-27T03:15:13.909 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-27T03:15:13.911 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-27T03:15:13.912 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-27T03:15:13.912 INFO:teuthology.orchestra.run.smithi077.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-27T03:15:13.913 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-27T03:15:13.914 INFO:teuthology.orchestra.run.smithi077.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-27T03:15:13.956 INFO:teuthology.orchestra.run.smithi077.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-27T03:15:13.956 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-27T03:15:13.958 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-27T03:15:13.958 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-27T03:15:13.959 INFO:teuthology.orchestra.run.smithi077.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-27T03:15:13.964 INFO:teuthology.orchestra.run.smithi077.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-27T03:15:13.967 INFO:teuthology.orchestra.run.smithi077.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-27T03:15:13.969 INFO:teuthology.orchestra.run.smithi077.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-27T03:15:13.969 INFO:teuthology.orchestra.run.smithi077.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-27T03:15:13.971 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-27T03:15:13.971 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-27T03:15:13.973 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-27T03:15:13.973 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-27T03:15:14.006 INFO:teuthology.orchestra.run.smithi077.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-27T03:15:14.007 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-27T03:15:14.007 INFO:teuthology.orchestra.run.smithi077.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-27T03:15:14.007 INFO:teuthology.orchestra.run.smithi077.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-27T03:15:14.008 INFO:teuthology.orchestra.run.smithi077.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-27T03:15:14.010 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-27T03:15:14.012 INFO:teuthology.orchestra.run.smithi110.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-27T03:15:14.034 INFO:teuthology.orchestra.run.smithi077.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2447-gb22e2ebd-1jammy [3,218 kB] 2024-04-27T03:15:14.057 INFO:teuthology.orchestra.run.smithi077.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-27T03:15:14.057 INFO:teuthology.orchestra.run.smithi077.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-27T03:15:14.058 INFO:teuthology.orchestra.run.smithi110.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-27T03:15:14.059 INFO:teuthology.orchestra.run.smithi077.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-27T03:15:14.060 INFO:teuthology.orchestra.run.smithi110.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-27T03:15:14.064 INFO:teuthology.orchestra.run.smithi077.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-27T03:15:14.065 INFO:teuthology.orchestra.run.smithi110.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-27T03:15:14.065 INFO:teuthology.orchestra.run.smithi077.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-27T03:15:14.066 INFO:teuthology.orchestra.run.smithi110.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-27T03:15:14.066 INFO:teuthology.orchestra.run.smithi077.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-27T03:15:14.067 INFO:teuthology.orchestra.run.smithi110.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-27T03:15:14.072 INFO:teuthology.orchestra.run.smithi110.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-27T03:15:14.118 INFO:teuthology.orchestra.run.smithi077.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-27T03:15:14.120 INFO:teuthology.orchestra.run.smithi077.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-27T03:15:14.120 INFO:teuthology.orchestra.run.smithi077.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-27T03:15:14.121 INFO:teuthology.orchestra.run.smithi077.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-27T03:15:14.159 INFO:teuthology.orchestra.run.smithi077.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-27T03:15:14.161 INFO:teuthology.orchestra.run.smithi077.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-27T03:15:14.162 INFO:teuthology.orchestra.run.smithi077.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-27T03:15:14.174 INFO:teuthology.orchestra.run.smithi077.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-27T03:15:14.175 INFO:teuthology.orchestra.run.smithi077.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-27T03:15:14.176 INFO:teuthology.orchestra.run.smithi077.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-27T03:15:14.188 INFO:teuthology.orchestra.run.smithi077.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-27T03:15:14.191 INFO:teuthology.orchestra.run.smithi077.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-27T03:15:14.195 INFO:teuthology.orchestra.run.smithi077.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-27T03:15:14.197 INFO:teuthology.orchestra.run.smithi077.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-27T03:15:14.220 INFO:teuthology.orchestra.run.smithi110.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-27T03:15:14.222 INFO:teuthology.orchestra.run.smithi110.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-27T03:15:14.224 INFO:teuthology.orchestra.run.smithi110.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-27T03:15:14.224 INFO:teuthology.orchestra.run.smithi110.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-27T03:15:14.225 INFO:teuthology.orchestra.run.smithi110.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-27T03:15:14.232 INFO:teuthology.orchestra.run.smithi077.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-27T03:15:14.233 INFO:teuthology.orchestra.run.smithi077.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-27T03:15:14.238 INFO:teuthology.orchestra.run.smithi077.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-27T03:15:14.240 INFO:teuthology.orchestra.run.smithi077.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-27T03:15:14.242 INFO:teuthology.orchestra.run.smithi077.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-27T03:15:14.247 INFO:teuthology.orchestra.run.smithi077.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-27T03:15:14.275 INFO:teuthology.orchestra.run.smithi110.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-27T03:15:14.279 INFO:teuthology.orchestra.run.smithi110.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-27T03:15:14.279 INFO:teuthology.orchestra.run.smithi110.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-27T03:15:14.363 INFO:teuthology.orchestra.run.smithi110.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-27T03:15:14.363 INFO:teuthology.orchestra.run.smithi110.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-27T03:15:14.365 INFO:teuthology.orchestra.run.smithi110.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-27T03:15:14.365 INFO:teuthology.orchestra.run.smithi110.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-27T03:15:14.366 INFO:teuthology.orchestra.run.smithi110.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-27T03:15:14.367 INFO:teuthology.orchestra.run.smithi110.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-27T03:15:14.368 INFO:teuthology.orchestra.run.smithi110.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-27T03:15:14.369 INFO:teuthology.orchestra.run.smithi110.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-27T03:15:14.375 INFO:teuthology.orchestra.run.smithi110.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-27T03:15:14.393 INFO:teuthology.orchestra.run.smithi077.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-27T03:15:14.393 INFO:teuthology.orchestra.run.smithi110.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-27T03:15:14.394 INFO:teuthology.orchestra.run.smithi077.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-27T03:15:14.396 INFO:teuthology.orchestra.run.smithi077.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-27T03:15:14.397 INFO:teuthology.orchestra.run.smithi077.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-27T03:15:14.399 INFO:teuthology.orchestra.run.smithi077.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-27T03:15:14.406 INFO:teuthology.orchestra.run.smithi110.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-27T03:15:14.407 INFO:teuthology.orchestra.run.smithi110.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-27T03:15:14.412 INFO:teuthology.orchestra.run.smithi110.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-27T03:15:14.416 INFO:teuthology.orchestra.run.smithi110.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-27T03:15:14.418 INFO:teuthology.orchestra.run.smithi110.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-27T03:15:14.418 INFO:teuthology.orchestra.run.smithi110.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-27T03:15:14.424 INFO:teuthology.orchestra.run.smithi077.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-27T03:15:14.426 INFO:teuthology.orchestra.run.smithi110.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-27T03:15:14.426 INFO:teuthology.orchestra.run.smithi077.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-27T03:15:14.426 INFO:teuthology.orchestra.run.smithi077.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-27T03:15:14.427 INFO:teuthology.orchestra.run.smithi110.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-27T03:15:14.432 INFO:teuthology.orchestra.run.smithi110.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-27T03:15:14.443 INFO:teuthology.orchestra.run.smithi077.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-27T03:15:14.444 INFO:teuthology.orchestra.run.smithi077.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-27T03:15:14.452 INFO:teuthology.orchestra.run.smithi110.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-27T03:15:14.453 INFO:teuthology.orchestra.run.smithi110.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-27T03:15:14.453 INFO:teuthology.orchestra.run.smithi110.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-27T03:15:14.455 INFO:teuthology.orchestra.run.smithi077.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-27T03:15:14.455 INFO:teuthology.orchestra.run.smithi077.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-27T03:15:14.456 INFO:teuthology.orchestra.run.smithi077.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-27T03:15:14.456 INFO:teuthology.orchestra.run.smithi110.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-27T03:15:14.456 INFO:teuthology.orchestra.run.smithi077.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-27T03:15:14.457 INFO:teuthology.orchestra.run.smithi077.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-27T03:15:14.457 INFO:teuthology.orchestra.run.smithi110.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-27T03:15:14.457 INFO:teuthology.orchestra.run.smithi077.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-27T03:15:14.465 INFO:teuthology.orchestra.run.smithi077.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-27T03:15:14.469 INFO:teuthology.orchestra.run.smithi110.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-27T03:15:14.469 INFO:teuthology.orchestra.run.smithi110.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-27T03:15:14.469 INFO:teuthology.orchestra.run.smithi110.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-27T03:15:14.477 INFO:teuthology.orchestra.run.smithi110.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-27T03:15:14.478 INFO:teuthology.orchestra.run.smithi110.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-27T03:15:14.483 INFO:teuthology.orchestra.run.smithi077.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-27T03:15:14.486 INFO:teuthology.orchestra.run.smithi077.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-27T03:15:14.487 INFO:teuthology.orchestra.run.smithi077.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-27T03:15:14.523 INFO:teuthology.orchestra.run.smithi077.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-27T03:15:14.528 INFO:teuthology.orchestra.run.smithi077.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-27T03:15:14.530 INFO:teuthology.orchestra.run.smithi077.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-27T03:15:14.532 INFO:teuthology.orchestra.run.smithi077.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-27T03:15:14.541 INFO:teuthology.orchestra.run.smithi077.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-27T03:15:14.542 INFO:teuthology.orchestra.run.smithi077.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-27T03:15:14.549 INFO:teuthology.orchestra.run.smithi077.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-27T03:15:14.556 INFO:teuthology.orchestra.run.smithi077.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2447-gb22e2ebd-1jammy [712 kB] 2024-04-27T03:15:14.558 INFO:teuthology.orchestra.run.smithi077.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-27T03:15:14.560 INFO:teuthology.orchestra.run.smithi077.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-27T03:15:14.560 INFO:teuthology.orchestra.run.smithi077.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-27T03:15:14.589 INFO:teuthology.orchestra.run.smithi077.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-27T03:15:14.591 INFO:teuthology.orchestra.run.smithi077.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-27T03:15:14.601 INFO:teuthology.orchestra.run.smithi077.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2447-gb22e2ebd-1jammy [356 kB] 2024-04-27T03:15:14.606 INFO:teuthology.orchestra.run.smithi077.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-27T03:15:14.607 INFO:teuthology.orchestra.run.smithi077.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-27T03:15:14.607 INFO:teuthology.orchestra.run.smithi077.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-27T03:15:14.612 INFO:teuthology.orchestra.run.smithi077.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-27T03:15:14.614 INFO:teuthology.orchestra.run.smithi077.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-27T03:15:14.644 INFO:teuthology.orchestra.run.smithi077.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2447-gb22e2ebd-1jammy [32.8 kB] 2024-04-27T03:15:14.650 INFO:teuthology.orchestra.run.smithi077.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2447-gb22e2ebd-1jammy [180 kB] 2024-04-27T03:15:14.656 INFO:teuthology.orchestra.run.smithi077.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2447-gb22e2ebd-1jammy [65.7 kB] 2024-04-27T03:15:14.658 INFO:teuthology.orchestra.run.smithi077.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2447-gb22e2ebd-1jammy [330 kB] 2024-04-27T03:15:14.706 INFO:teuthology.orchestra.run.smithi077.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2447-gb22e2ebd-1jammy [6,775 kB] 2024-04-27T03:15:15.555 INFO:teuthology.orchestra.run.smithi077.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2447-gb22e2ebd-1jammy [112 kB] 2024-04-27T03:15:15.557 INFO:teuthology.orchestra.run.smithi077.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2447-gb22e2ebd-1jammy [466 kB] 2024-04-27T03:15:15.616 INFO:teuthology.orchestra.run.smithi077.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2447-gb22e2ebd-1jammy [25.6 MB] 2024-04-27T03:15:16.571 INFO:teuthology.orchestra.run.smithi110.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2447-gb22e2ebd-1jammy [3,218 kB] 2024-04-27T03:15:19.090 INFO:teuthology.orchestra.run.smithi110.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2447-gb22e2ebd-1jammy [712 kB] 2024-04-27T03:15:19.240 INFO:teuthology.orchestra.run.smithi077.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2447-gb22e2ebd-1jammy [5,084 kB] 2024-04-27T03:15:19.595 INFO:teuthology.orchestra.run.smithi110.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2447-gb22e2ebd-1jammy [356 kB] 2024-04-27T03:15:19.784 INFO:teuthology.orchestra.run.smithi077.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2447-gb22e2ebd-1jammy [244 kB] 2024-04-27T03:15:19.808 INFO:teuthology.orchestra.run.smithi077.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2447-gb22e2ebd-1jammy [124 kB] 2024-04-27T03:15:19.810 INFO:teuthology.orchestra.run.smithi110.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2447-gb22e2ebd-1jammy [32.8 kB] 2024-04-27T03:15:19.812 INFO:teuthology.orchestra.run.smithi077.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2447-gb22e2ebd-1jammy [1,526 kB] 2024-04-27T03:15:19.817 INFO:teuthology.orchestra.run.smithi110.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2447-gb22e2ebd-1jammy [180 kB] 2024-04-27T03:15:19.946 INFO:teuthology.orchestra.run.smithi110.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2447-gb22e2ebd-1jammy [65.7 kB] 2024-04-27T03:15:19.965 INFO:teuthology.orchestra.run.smithi110.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2447-gb22e2ebd-1jammy [330 kB] 2024-04-27T03:15:20.045 INFO:teuthology.orchestra.run.smithi077.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2447-gb22e2ebd-1jammy [6,246 kB] 2024-04-27T03:15:20.174 INFO:teuthology.orchestra.run.smithi110.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2447-gb22e2ebd-1jammy [6,775 kB] 2024-04-27T03:15:20.786 INFO:teuthology.orchestra.run.smithi077.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2447-gb22e2ebd-1jammy [22.7 MB] 2024-04-27T03:15:22.055 INFO:teuthology.orchestra.run.smithi077.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2447-gb22e2ebd-1jammy [14.0 kB] 2024-04-27T03:15:22.056 INFO:teuthology.orchestra.run.smithi077.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2447-gb22e2ebd-1jammy [869 kB] 2024-04-27T03:15:22.073 INFO:teuthology.orchestra.run.smithi077.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2447-gb22e2ebd-1jammy [2,437 kB] 2024-04-27T03:15:22.168 INFO:teuthology.orchestra.run.smithi077.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2447-gb22e2ebd-1jammy [791 kB] 2024-04-27T03:15:22.205 INFO:teuthology.orchestra.run.smithi077.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2447-gb22e2ebd-1jammy [147 kB] 2024-04-27T03:15:22.210 INFO:teuthology.orchestra.run.smithi077.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2447-gb22e2ebd-1jammy [2,292 kB] 2024-04-27T03:15:22.275 INFO:teuthology.orchestra.run.smithi110.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2447-gb22e2ebd-1jammy [112 kB] 2024-04-27T03:15:22.294 INFO:teuthology.orchestra.run.smithi110.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2447-gb22e2ebd-1jammy [466 kB] 2024-04-27T03:15:22.313 INFO:teuthology.orchestra.run.smithi077.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2447-gb22e2ebd-1jammy [8,625 kB] 2024-04-27T03:15:22.398 INFO:teuthology.orchestra.run.smithi110.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2447-gb22e2ebd-1jammy [25.6 MB] 2024-04-27T03:15:22.701 INFO:teuthology.orchestra.run.smithi077.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2447-gb22e2ebd-1jammy [14.2 kB] 2024-04-27T03:15:22.741 INFO:teuthology.orchestra.run.smithi077.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2447-gb22e2ebd-1jammy [50.7 MB] 2024-04-27T03:15:25.474 INFO:teuthology.orchestra.run.smithi077.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2447-gb22e2ebd-1jammy [121 kB] 2024-04-27T03:15:25.477 INFO:teuthology.orchestra.run.smithi077.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2447-gb22e2ebd-1jammy [38.9 kB] 2024-04-27T03:15:25.478 INFO:teuthology.orchestra.run.smithi077.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2447-gb22e2ebd-1jammy [13.4 MB] 2024-04-27T03:15:26.066 INFO:teuthology.orchestra.run.smithi077.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2447-gb22e2ebd-1jammy [91.2 kB] 2024-04-27T03:15:26.623 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 179 MB in 13s (14.1 MB/s) 2024-04-27T03:15:26.789 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-27T03:15:26.884 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-27T03:15:26.889 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-27T03:15:26.979 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-27T03:15:27.007 INFO:teuthology.orchestra.run.smithi110.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2447-gb22e2ebd-1jammy [5,084 kB] 2024-04-27T03:15:27.266 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-27T03:15:27.292 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-27T03:15:27.324 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-27T03:15:27.331 INFO:teuthology.orchestra.run.smithi110.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2447-gb22e2ebd-1jammy [244 kB] 2024-04-27T03:15:27.336 INFO:teuthology.orchestra.run.smithi110.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2447-gb22e2ebd-1jammy [124 kB] 2024-04-27T03:15:27.343 INFO:teuthology.orchestra.run.smithi110.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2447-gb22e2ebd-1jammy [1,526 kB] 2024-04-27T03:15:27.453 INFO:teuthology.orchestra.run.smithi110.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2447-gb22e2ebd-1jammy [6,246 kB] 2024-04-27T03:15:27.643 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-27T03:15:27.670 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-27T03:15:27.710 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-27T03:15:27.918 INFO:teuthology.orchestra.run.smithi110.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2447-gb22e2ebd-1jammy [22.7 MB] 2024-04-27T03:15:28.146 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-27T03:15:28.173 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-27T03:15:28.221 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:15:28.566 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-27T03:15:28.593 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-27T03:15:28.624 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:15:28.944 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-27T03:15:28.970 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-27T03:15:29.010 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:15:29.355 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-27T03:15:29.381 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-27T03:15:29.422 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-27T03:15:29.629 INFO:teuthology.orchestra.run.smithi110.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2447-gb22e2ebd-1jammy [14.0 kB] 2024-04-27T03:15:29.630 INFO:teuthology.orchestra.run.smithi110.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2447-gb22e2ebd-1jammy [869 kB] 2024-04-27T03:15:29.647 INFO:teuthology.orchestra.run.smithi110.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2447-gb22e2ebd-1jammy [2,437 kB] 2024-04-27T03:15:29.741 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librados2. 2024-04-27T03:15:29.768 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../007-librados2_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:29.792 INFO:teuthology.orchestra.run.smithi110.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2447-gb22e2ebd-1jammy [791 kB] 2024-04-27T03:15:29.808 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librados2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:29.855 INFO:teuthology.orchestra.run.smithi110.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2447-gb22e2ebd-1jammy [147 kB] 2024-04-27T03:15:29.859 INFO:teuthology.orchestra.run.smithi110.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2447-gb22e2ebd-1jammy [2,292 kB] 2024-04-27T03:15:30.009 INFO:teuthology.orchestra.run.smithi110.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2447-gb22e2ebd-1jammy [8,625 kB] 2024-04-27T03:15:30.260 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librbd1. 2024-04-27T03:15:30.286 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../008-librbd1_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:30.327 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librbd1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:30.662 INFO:teuthology.orchestra.run.smithi110.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2447-gb22e2ebd-1jammy [14.2 kB] 2024-04-27T03:15:30.662 INFO:teuthology.orchestra.run.smithi110.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2447-gb22e2ebd-1jammy [50.7 MB] 2024-04-27T03:15:30.771 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libcephfs2. 2024-04-27T03:15:30.797 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:30.838 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libcephfs2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:31.116 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rados. 2024-04-27T03:15:31.132 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:31.282 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rados (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:31.801 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-27T03:15:31.828 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:15:31.868 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-ceph-argparse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:32.104 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cephfs. 2024-04-27T03:15:32.119 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:32.162 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cephfs (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:32.431 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-27T03:15:32.446 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:15:32.490 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:32.793 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-27T03:15:32.819 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-27T03:15:32.859 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-27T03:15:33.162 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-prettytable. 2024-04-27T03:15:33.188 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-27T03:15:33.228 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-27T03:15:33.506 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rbd. 2024-04-27T03:15:33.521 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:33.564 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rbd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:33.909 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-27T03:15:33.935 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-27T03:15:33.975 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-27T03:15:34.269 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-27T03:15:34.285 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-27T03:15:34.319 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-27T03:15:34.413 INFO:teuthology.orchestra.run.smithi110.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2447-gb22e2ebd-1jammy [121 kB] 2024-04-27T03:15:34.415 INFO:teuthology.orchestra.run.smithi110.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2447-gb22e2ebd-1jammy [38.9 kB] 2024-04-27T03:15:34.417 INFO:teuthology.orchestra.run.smithi110.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2447-gb22e2ebd-1jammy [13.4 MB] 2024-04-27T03:15:34.597 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-27T03:15:34.612 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-27T03:15:34.647 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-27T03:15:34.769 INFO:teuthology.orchestra.run.smithi110.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/b22e2ebdeb24376882b7bda2a7329c8cccc2276a/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2447-gb22e2ebd-1jammy [91.2 kB] 2024-04-27T03:15:34.959 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua5.1. 2024-04-27T03:15:34.985 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-27T03:15:35.025 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-27T03:15:35.329 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 179 MB in 22s (8,336 kB/s) 2024-04-27T03:15:35.345 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua-any. 2024-04-27T03:15:35.371 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-27T03:15:35.412 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-27T03:15:35.491 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-27T03:15:35.591 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-27T03:15:35.596 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-27T03:15:35.655 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package zip. 2024-04-27T03:15:35.671 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-27T03:15:35.681 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-27T03:15:35.705 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking zip (3.0-12build2) ... 2024-04-27T03:15:35.976 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package unzip. 2024-04-27T03:15:35.976 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-27T03:15:36.002 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-27T03:15:36.003 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-27T03:15:36.035 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-27T03:15:36.042 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-27T03:15:36.371 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-27T03:15:36.397 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-27T03:15:36.420 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package luarocks. 2024-04-27T03:15:36.437 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-27T03:15:36.447 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-27T03:15:36.487 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-27T03:15:36.891 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-27T03:15:36.918 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-27T03:15:36.966 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:15:36.995 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librgw2. 2024-04-27T03:15:37.021 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../025-librgw2_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:37.053 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librgw2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:37.319 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-27T03:15:37.346 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-27T03:15:37.377 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:15:37.589 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rgw. 2024-04-27T03:15:37.615 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:37.655 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:37.756 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-27T03:15:37.782 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-27T03:15:37.822 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:15:37.983 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-27T03:15:38.009 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-27T03:15:38.050 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-27T03:15:38.184 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-27T03:15:38.209 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-27T03:15:38.250 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-27T03:15:38.386 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libradosstriper1. 2024-04-27T03:15:38.412 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:38.452 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libradosstriper1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:38.562 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librados2. 2024-04-27T03:15:38.588 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../007-librados2_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:38.620 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librados2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:38.814 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-common. 2024-04-27T03:15:38.840 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:38.880 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:39.073 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librbd1. 2024-04-27T03:15:39.099 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../008-librbd1_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:39.139 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librbd1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:39.609 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs2. 2024-04-27T03:15:39.634 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:39.675 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:39.961 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rados. 2024-04-27T03:15:39.987 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:40.028 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rados (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:40.252 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-base. 2024-04-27T03:15:40.279 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:40.328 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-base (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:40.489 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-27T03:15:40.504 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:15:40.539 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-argparse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:40.791 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cephfs. 2024-04-27T03:15:40.807 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:40.850 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cephfs (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:40.856 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-27T03:15:40.871 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-27T03:15:40.906 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-27T03:15:41.178 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-27T03:15:41.204 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:15:41.209 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cheroot. 2024-04-27T03:15:41.235 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-27T03:15:41.252 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:41.275 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-27T03:15:41.564 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-27T03:15:41.590 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-27T03:15:41.603 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-27T03:15:41.619 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-27T03:15:41.630 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-27T03:15:41.662 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-27T03:15:41.958 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-prettytable. 2024-04-27T03:15:41.964 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-27T03:15:41.985 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-27T03:15:41.991 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-27T03:15:42.025 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-27T03:15:42.030 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-27T03:15:42.286 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rbd. 2024-04-27T03:15:42.301 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:42.336 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rbd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:42.475 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-27T03:15:42.502 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-27T03:15:42.542 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-27T03:15:42.655 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-27T03:15:42.682 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-27T03:15:42.722 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-27T03:15:42.844 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-tempora. 2024-04-27T03:15:42.871 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-27T03:15:42.911 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-27T03:15:43.008 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-27T03:15:43.034 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-27T03:15:43.074 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-27T03:15:43.214 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-portend. 2024-04-27T03:15:43.240 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-27T03:15:43.280 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-27T03:15:43.386 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-27T03:15:43.413 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-27T03:15:43.453 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-27T03:15:43.525 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-27T03:15:43.540 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-27T03:15:43.575 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-27T03:15:43.756 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua5.1. 2024-04-27T03:15:43.774 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-27T03:15:43.806 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-27T03:15:43.861 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-27T03:15:43.887 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-27T03:15:43.927 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-27T03:15:44.117 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua-any. 2024-04-27T03:15:44.144 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-27T03:15:44.184 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-27T03:15:44.256 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-natsort. 2024-04-27T03:15:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-27T03:15:44.322 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-27T03:15:44.437 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package zip. 2024-04-27T03:15:44.463 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-27T03:15:44.503 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking zip (3.0-12build2) ... 2024-04-27T03:15:44.567 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-logutils. 2024-04-27T03:15:44.593 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-27T03:15:44.625 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-27T03:15:44.814 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package unzip. 2024-04-27T03:15:44.841 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-27T03:15:44.881 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-27T03:15:44.928 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-27T03:15:44.954 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-27T03:15:44.994 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-27T03:15:45.243 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package luarocks. 2024-04-27T03:15:45.264 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-mako. 2024-04-27T03:15:45.270 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-27T03:15:45.290 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-27T03:15:45.330 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-27T03:15:45.477 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-27T03:15:45.659 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-27T03:15:45.685 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-27T03:15:45.725 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-27T03:15:46.030 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-27T03:15:46.054 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-27T03:15:46.094 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-27T03:15:46.389 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-webob. 2024-04-27T03:15:46.415 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-27T03:15:46.455 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-27T03:15:46.651 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librgw2. 2024-04-27T03:15:46.678 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../025-librgw2_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:46.718 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librgw2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:47.270 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rgw. 2024-04-27T03:15:47.286 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:47.329 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:47.567 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-waitress. 2024-04-27T03:15:47.593 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-27T03:15:47.615 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-27T03:15:47.628 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-27T03:15:47.642 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-27T03:15:47.673 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-27T03:15:47.886 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-tempita. 2024-04-27T03:15:47.901 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-27T03:15:47.944 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-27T03:15:48.010 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libradosstriper1. 2024-04-27T03:15:48.036 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:48.076 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libradosstriper1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:48.247 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-paste. 2024-04-27T03:15:48.274 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-27T03:15:48.314 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-27T03:15:48.530 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-common. 2024-04-27T03:15:48.556 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:48.596 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:48.625 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-27T03:15:48.652 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-27T03:15:48.683 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-27T03:15:48.970 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-27T03:15:48.996 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-27T03:15:49.036 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-27T03:15:49.339 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-webtest. 2024-04-27T03:15:49.365 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-27T03:15:49.405 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-27T03:15:49.710 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pecan. 2024-04-27T03:15:49.737 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-27T03:15:49.769 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-27T03:15:49.902 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-base. 2024-04-27T03:15:49.928 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:49.969 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-base (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:50.106 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-certifi. 2024-04-27T03:15:50.132 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-27T03:15:50.173 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-27T03:15:50.434 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-idna. 2024-04-27T03:15:50.493 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-27T03:15:50.493 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-27T03:15:50.531 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-27T03:15:50.557 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-27T03:15:50.597 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-27T03:15:50.812 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-urllib3. 2024-04-27T03:15:50.838 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-27T03:15:50.878 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-27T03:15:50.900 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cheroot. 2024-04-27T03:15:50.927 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-27T03:15:50.966 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-27T03:15:51.198 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-requests. 2024-04-27T03:15:51.225 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-27T03:15:51.267 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-27T03:15:51.295 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-27T03:15:51.322 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-27T03:15:51.361 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-27T03:15:51.526 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-27T03:15:51.552 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-27T03:15:51.583 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-27T03:15:51.622 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-27T03:15:51.638 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-27T03:15:51.672 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-27T03:15:51.837 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-27T03:15:51.863 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:15:51.903 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:51.942 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-27T03:15:51.970 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-27T03:15:52.008 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-27T03:15:52.283 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-27T03:15:52.310 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:52.311 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempora. 2024-04-27T03:15:52.338 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-27T03:15:52.349 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:52.378 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-27T03:15:52.681 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-portend. 2024-04-27T03:15:52.708 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-27T03:15:52.748 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-27T03:15:52.826 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-27T03:15:52.852 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-27T03:15:52.902 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-27T03:15:53.000 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-27T03:15:53.016 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-27T03:15:53.050 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-27T03:15:53.163 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-openssl. 2024-04-27T03:15:53.190 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-27T03:15:53.221 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-27T03:15:53.362 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-27T03:15:53.388 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-27T03:15:53.428 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-27T03:15:53.532 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr. 2024-04-27T03:15:53.560 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:53.599 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:53.740 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-natsort. 2024-04-27T03:15:53.766 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-27T03:15:53.798 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-27T03:15:53.986 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mon. 2024-04-27T03:15:54.013 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:54.052 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mon (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:54.085 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-logutils. 2024-04-27T03:15:54.111 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-27T03:15:54.151 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-27T03:15:54.446 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-27T03:15:54.473 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-27T03:15:54.513 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-27T03:15:54.631 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-osd. 2024-04-27T03:15:54.657 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:54.730 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-osd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:54.815 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-mako. 2024-04-27T03:15:54.842 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-27T03:15:54.890 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-27T03:15:55.277 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-27T03:15:55.304 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-27T03:15:55.352 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-27T03:15:55.747 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-27T03:15:55.774 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-27T03:15:55.783 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph. 2024-04-27T03:15:55.799 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../066-ceph_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:55.833 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:55.996 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-27T03:15:56.086 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-fuse. 2024-04-27T03:15:56.112 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:56.152 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:56.291 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webob. 2024-04-27T03:15:56.318 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-27T03:15:56.357 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-27T03:15:56.514 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mds. 2024-04-27T03:15:56.541 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:56.580 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mds (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:56.727 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-waitress. 2024-04-27T03:15:56.754 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-27T03:15:56.797 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-27T03:15:56.933 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package cephadm. 2024-04-27T03:15:56.960 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../069-cephadm_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:15:56.991 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:57.157 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempita. 2024-04-27T03:15:57.184 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-27T03:15:57.232 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-27T03:15:57.303 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-27T03:15:57.329 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-27T03:15:57.369 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-27T03:15:57.568 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-paste. 2024-04-27T03:15:57.595 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-27T03:15:57.634 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-27T03:15:57.680 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-27T03:15:57.695 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-27T03:15:57.738 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-27T03:15:57.938 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-27T03:15:57.953 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-27T03:15:57.996 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-27T03:15:58.299 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-27T03:15:58.325 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-27T03:15:58.365 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-27T03:15:58.627 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webtest. 2024-04-27T03:15:58.653 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-27T03:15:58.693 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-27T03:15:58.759 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-babel. 2024-04-27T03:15:58.786 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-27T03:15:58.851 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-27T03:15:58.954 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pecan. 2024-04-27T03:15:58.979 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-27T03:15:59.013 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-27T03:15:59.171 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jinja2. 2024-04-27T03:15:59.198 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-27T03:15:59.237 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-27T03:15:59.343 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-certifi. 2024-04-27T03:15:59.369 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-27T03:15:59.409 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-27T03:15:59.565 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-27T03:15:59.592 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:15:59.631 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:15:59.745 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-idna. 2024-04-27T03:15:59.772 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-27T03:15:59.812 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-27T03:15:59.962 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-27T03:15:59.987 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-27T03:16:00.026 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-27T03:16:00.132 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-urllib3. 2024-04-27T03:16:00.159 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-27T03:16:00.198 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-27T03:16:00.354 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-routes. 2024-04-27T03:16:00.380 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-27T03:16:00.420 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-27T03:16:00.526 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests. 2024-04-27T03:16:00.553 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-27T03:16:00.593 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-27T03:16:00.732 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-27T03:16:00.758 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:00.798 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:00.862 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-27T03:16:00.889 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-27T03:16:00.929 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-27T03:16:01.216 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-27T03:16:01.243 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:01.282 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:01.679 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-27T03:16:01.705 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:01.745 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:01.995 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-27T03:16:02.011 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-27T03:16:02.053 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-27T03:16:02.074 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-27T03:16:02.101 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-27T03:16:02.140 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-27T03:16:02.418 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-openssl. 2024-04-27T03:16:02.433 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-27T03:16:02.468 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-27T03:16:02.548 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-joblib. 2024-04-27T03:16:02.575 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-27T03:16:02.615 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-27T03:16:02.813 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr. 2024-04-27T03:16:02.839 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:02.879 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:02.985 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-27T03:16:03.012 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-27T03:16:03.051 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-27T03:16:03.299 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mon. 2024-04-27T03:16:03.326 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:03.366 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mon (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:03.529 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-sklearn. 2024-04-27T03:16:03.556 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-27T03:16:03.595 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-27T03:16:03.969 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-osd. 2024-04-27T03:16:03.996 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:04.035 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-osd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:04.345 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-27T03:16:04.373 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:04.497 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:05.089 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph. 2024-04-27T03:16:05.116 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../066-ceph_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:05.172 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:05.409 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cachetools. 2024-04-27T03:16:05.436 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-27T03:16:05.475 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-27T03:16:05.517 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-fuse. 2024-04-27T03:16:05.543 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:05.575 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:05.778 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-27T03:16:05.806 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-27T03:16:05.845 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-27T03:16:05.945 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mds. 2024-04-27T03:16:05.971 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:06.011 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mds (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:06.156 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-27T03:16:06.183 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-27T03:16:06.223 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-27T03:16:06.526 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rsa. 2024-04-27T03:16:06.553 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-27T03:16:06.565 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephadm. 2024-04-27T03:16:06.591 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../069-cephadm_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:06.592 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-27T03:16:06.623 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:06.912 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-google-auth. 2024-04-27T03:16:06.939 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-27T03:16:06.959 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-27T03:16:06.985 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-27T03:16:07.026 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-27T03:16:07.035 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-27T03:16:07.465 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-27T03:16:07.470 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-27T03:16:07.492 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-27T03:16:07.496 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-27T03:16:07.528 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-27T03:16:07.532 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-27T03:16:07.835 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-websocket. 2024-04-27T03:16:07.851 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-27T03:16:07.893 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-27T03:16:08.221 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-27T03:16:08.249 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-27T03:16:08.288 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-27T03:16:08.550 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-babel. 2024-04-27T03:16:08.577 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-27T03:16:08.616 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-27T03:16:08.919 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jinja2. 2024-04-27T03:16:08.946 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-27T03:16:08.986 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-27T03:16:09.134 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-27T03:16:09.150 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:09.192 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:09.347 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-27T03:16:09.374 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:09.414 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:09.512 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-27T03:16:09.539 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-27T03:16:09.579 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-27T03:16:09.759 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-27T03:16:09.785 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-27T03:16:09.825 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-27T03:16:09.915 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-27T03:16:09.942 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-27T03:16:09.981 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-27T03:16:10.161 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-routes. 2024-04-27T03:16:10.188 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-27T03:16:10.200 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package jq. 2024-04-27T03:16:10.216 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-27T03:16:10.228 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-27T03:16:10.250 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-27T03:16:10.486 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package socat. 2024-04-27T03:16:10.502 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-27T03:16:10.544 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-27T03:16:10.565 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-27T03:16:10.591 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:10.631 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:10.823 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package xmlstarlet. 2024-04-27T03:16:10.835 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-27T03:16:10.864 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-27T03:16:11.210 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-test. 2024-04-27T03:16:11.237 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:11.276 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-test (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:11.903 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-27T03:16:11.919 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-27T03:16:11.961 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-27T03:16:12.490 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-joblib. 2024-04-27T03:16:12.517 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-27T03:16:12.556 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-27T03:16:12.901 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-27T03:16:12.928 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-27T03:16:12.959 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-27T03:16:13.254 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn. 2024-04-27T03:16:13.281 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-27T03:16:13.320 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-27T03:16:13.531 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-volume. 2024-04-27T03:16:13.558 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:13.597 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-volume (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:13.892 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-27T03:16:13.920 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:13.959 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libcephfs-dev (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:14.079 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-27T03:16:14.107 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:14.231 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:14.270 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-27T03:16:14.297 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-27T03:16:14.336 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-27T03:16:14.641 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-27T03:16:14.668 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-27T03:16:14.707 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-27T03:16:15.019 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package nvme-cli. 2024-04-27T03:16:15.035 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-27T03:16:15.069 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-27T03:16:15.168 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cachetools. 2024-04-27T03:16:15.195 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-27T03:16:15.243 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-27T03:16:15.495 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-27T03:16:15.499 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package pkg-config. 2024-04-27T03:16:15.511 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-27T03:16:15.527 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-27T03:16:15.554 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-27T03:16:15.565 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-27T03:16:15.818 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-27T03:16:15.834 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-27T03:16:15.868 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-27T03:16:15.874 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-27T03:16:15.902 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-27T03:16:15.941 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-27T03:16:16.252 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rsa. 2024-04-27T03:16:16.279 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-27T03:16:16.305 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-27T03:16:16.318 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-27T03:16:16.332 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-27T03:16:16.371 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-27T03:16:16.632 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pastescript. 2024-04-27T03:16:16.660 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-27T03:16:16.690 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-27T03:16:16.814 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-google-auth. 2024-04-27T03:16:16.841 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-27T03:16:16.880 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-27T03:16:16.985 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pluggy. 2024-04-27T03:16:17.012 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-27T03:16:17.051 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-27T03:16:17.192 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-27T03:16:17.219 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-27T03:16:17.258 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-27T03:16:17.363 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-psutil. 2024-04-27T03:16:17.390 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-27T03:16:17.429 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-27T03:16:17.503 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-websocket. 2024-04-27T03:16:17.530 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-27T03:16:17.570 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-27T03:16:17.749 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-py. 2024-04-27T03:16:17.777 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-27T03:16:17.815 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-27T03:16:17.857 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-27T03:16:17.884 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-27T03:16:17.915 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-27T03:16:18.160 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pygments. 2024-04-27T03:16:18.188 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-27T03:16:18.227 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-27T03:16:18.529 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-27T03:16:18.556 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:18.595 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:18.706 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-27T03:16:18.733 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-27T03:16:18.772 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-27T03:16:18.923 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-27T03:16:18.950 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-27T03:16:18.989 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-27T03:16:19.075 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-toml. 2024-04-27T03:16:19.102 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-27T03:16:19.141 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-27T03:16:19.318 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-27T03:16:19.345 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-27T03:16:19.384 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-27T03:16:19.469 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pytest. 2024-04-27T03:16:19.496 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-27T03:16:19.535 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-27T03:16:19.620 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package jq. 2024-04-27T03:16:19.636 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-27T03:16:19.670 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-27T03:16:19.872 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-simplejson. 2024-04-27T03:16:19.900 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-27T03:16:19.939 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-27T03:16:19.939 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package socat. 2024-04-27T03:16:19.967 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-27T03:16:20.006 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-27T03:16:20.208 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-27T03:16:20.236 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-27T03:16:20.274 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-27T03:16:20.292 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package xmlstarlet. 2024-04-27T03:16:20.309 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-27T03:16:20.351 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-27T03:16:20.747 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-test. 2024-04-27T03:16:20.774 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:20.813 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-test (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:20.928 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package radosgw. 2024-04-27T03:16:20.956 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../116-radosgw_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:20.986 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking radosgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:21.773 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package rbd-fuse. 2024-04-27T03:16:21.789 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:21.831 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking rbd-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:22.118 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-27T03:16:22.369 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-27T03:16:22.504 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-27T03:16:22.764 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-27T03:16:23.093 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-volume. 2024-04-27T03:16:23.109 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2447-gb22e2ebd-1jammy_all.deb ... 2024-04-27T03:16:23.118 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-27T03:16:23.142 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-volume (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:23.413 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-27T03:16:23.429 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:23.463 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs-dev (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:23.551 INFO:teuthology.orchestra.run.smithi077.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-27T03:16:23.630 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:23.791 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-27T03:16:23.818 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-27T03:16:23.857 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-27T03:16:24.138 INFO:teuthology.orchestra.run.smithi077.stdout:Adding system user cephadm....done 2024-04-27T03:16:24.279 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-27T03:16:24.286 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-27T03:16:24.306 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-27T03:16:24.346 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-27T03:16:24.663 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-27T03:16:24.683 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package nvme-cli. 2024-04-27T03:16:24.710 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-27T03:16:24.749 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-27T03:16:24.913 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-27T03:16:25.015 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-27T03:16:25.146 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package pkg-config. 2024-04-27T03:16:25.170 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-27T03:16:25.204 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-27T03:16:25.249 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-27T03:16:25.449 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-27T03:16:25.465 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-27T03:16:25.507 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-27T03:16:25.517 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-27T03:16:25.643 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-27T03:16:25.919 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-27T03:16:25.944 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-27T03:16:25.946 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-27T03:16:25.985 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-27T03:16:26.288 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastescript. 2024-04-27T03:16:26.296 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-27T03:16:26.316 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-27T03:16:26.354 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-27T03:16:26.530 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-27T03:16:26.683 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pluggy. 2024-04-27T03:16:26.710 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-27T03:16:26.749 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-27T03:16:26.798 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-27T03:16:27.052 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-psutil. 2024-04-27T03:16:27.058 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-27T03:16:27.080 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-27T03:16:27.119 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-27T03:16:27.309 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-27T03:16:27.455 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-py. 2024-04-27T03:16:27.483 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-27T03:16:27.521 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-27T03:16:27.568 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-27T03:16:27.828 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-ceph-argparse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:27.867 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pygments. 2024-04-27T03:16:27.894 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-27T03:16:27.933 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-27T03:16:28.137 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-27T03:16:28.238 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-27T03:16:28.348 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-27T03:16:28.396 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-27T03:16:28.423 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-27T03:16:28.462 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-27T03:16:28.473 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-27T03:16:28.723 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-toml. 2024-04-27T03:16:28.724 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-27T03:16:28.750 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-27T03:16:28.790 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-27T03:16:29.084 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pytest. 2024-04-27T03:16:29.112 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-27T03:16:29.112 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-27T03:16:29.139 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-27T03:16:29.150 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-27T03:16:29.203 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-27T03:16:29.312 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-27T03:16:29.454 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplejson. 2024-04-27T03:16:29.482 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-27T03:16:29.512 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-27T03:16:29.655 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-27T03:16:29.748 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-27T03:16:29.765 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-27T03:16:29.772 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-27T03:16:29.798 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-27T03:16:30.040 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-27T03:16:30.300 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-27T03:16:30.553 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package radosgw. 2024-04-27T03:16:30.580 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../116-radosgw_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:30.592 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up zip (3.0-12build2) ... 2024-04-27T03:16:30.619 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking radosgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:30.718 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-27T03:16:31.354 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-27T03:16:31.430 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package rbd-fuse. 2024-04-27T03:16:31.458 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2447-gb22e2ebd-1jammy_amd64.deb ... 2024-04-27T03:16:31.496 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking rbd-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:31.605 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-27T03:16:31.714 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-27T03:16:31.818 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-27T03:16:31.832 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-27T03:16:32.077 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-27T03:16:32.124 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-27T03:16:32.469 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-27T03:16:32.500 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-27T03:16:32.734 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-27T03:16:32.751 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-27T03:16:33.049 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-27T03:16:33.111 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-27T03:16:33.381 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-27T03:16:33.484 INFO:teuthology.orchestra.run.smithi110.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-27T03:16:33.554 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:33.724 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-27T03:16:33.992 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-27T03:16:34.113 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user cephadm....done 2024-04-27T03:16:34.118 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:34.269 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-27T03:16:34.419 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-idna (3.3-1) ... 2024-04-27T03:16:34.537 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-27T03:16:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-27T03:16:34.780 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-27T03:16:34.881 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-27T03:16:35.140 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-27T03:16:35.392 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-27T03:16:35.519 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-27T03:16:35.828 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-27T03:16:35.901 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-27T03:16:36.169 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:16:36.179 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-27T03:16:36.303 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-27T03:16:36.422 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-27T03:16:36.570 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-27T03:16:36.671 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-27T03:16:36.699 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-27T03:16:36.773 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-27T03:16:36.967 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-27T03:16:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-27T03:16:37.219 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-27T03:16:37.317 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-27T03:16:37.478 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-27T03:16:37.560 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:16:37.685 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-27T03:16:37.762 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-argparse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:37.978 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-27T03:16:38.138 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-27T03:16:38.213 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-27T03:16:38.264 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-27T03:16:38.392 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-27T03:16:38.447 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-27T03:16:38.518 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-27T03:16:38.690 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-27T03:16:38.794 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-27T03:16:38.941 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-27T03:16:39.188 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-27T03:16:39.209 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-27T03:16:39.217 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-27T03:16:39.281 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-27T03:16:39.334 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-27T03:16:39.407 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-27T03:16:39.619 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-27T03:16:39.766 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-27T03:16:39.782 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-27T03:16:39.917 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-27T03:16:40.063 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-27T03:16:40.178 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-27T03:16:40.347 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-27T03:16:40.436 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-27T03:16:40.590 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-27T03:16:40.716 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-27T03:16:40.745 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up zip (3.0-12build2) ... 2024-04-27T03:16:40.842 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-27T03:16:40.871 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-27T03:16:40.931 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-27T03:16:41.193 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-27T03:16:41.319 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-27T03:16:41.510 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-27T03:16:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-27T03:16:41.766 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-27T03:16:41.892 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-27T03:16:41.963 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-27T03:16:42.018 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-27T03:16:42.089 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-27T03:16:42.319 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-27T03:16:42.349 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:16:42.475 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-27T03:16:42.792 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-27T03:16:42.912 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-27T03:16:43.077 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-27T03:16:43.154 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-27T03:16:43.344 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-27T03:16:43.462 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-27T03:16:43.548 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-27T03:16:43.730 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-27T03:16:43.840 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-27T03:16:44.015 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-27T03:16:44.184 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-27T03:16:44.351 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-27T03:16:44.443 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-27T03:16:44.569 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:44.702 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-27T03:16:44.878 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-idna (3.3-1) ... 2024-04-27T03:16:44.995 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-27T03:16:45.179 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-27T03:16:45.315 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-27T03:16:45.441 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librados2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:45.567 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:45.693 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-27T03:16:46.029 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libcephfs2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:46.130 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libradosstriper1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:46.239 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-27T03:16:46.343 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-27T03:16:46.516 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librbd1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:46.595 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:16:46.642 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-modules-core (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:46.704 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-27T03:16:46.759 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:46.956 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-27T03:16:46.956 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-27T03:16:46.963 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-27T03:16:47.114 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-27T03:16:47.240 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-27T03:16:47.460 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libcephfs-dev (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:47.534 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-27T03:16:47.561 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rados (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:47.687 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librgw2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:47.793 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-27T03:16:47.813 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-27T03:16:48.061 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:16:48.212 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-27T03:16:48.538 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-27T03:16:48.789 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-27T03:16:48.950 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rbd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:49.048 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-27T03:16:49.076 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up rbd-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:49.203 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:49.325 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-27T03:16:49.328 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cephfs (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:49.437 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:49.593 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-27T03:16:49.877 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-27T03:16:49.938 INFO:teuthology.orchestra.run.smithi077.stdout:Adding group ceph....done 2024-04-27T03:16:49.995 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-27T03:16:50.262 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-27T03:16:50.298 INFO:teuthology.orchestra.run.smithi077.stdout:Adding system user ceph....done 2024-04-27T03:16:50.363 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-27T03:16:50.466 INFO:teuthology.orchestra.run.smithi077.stdout:Setting system user ceph properties....done 2024-04-27T03:16:50.485 INFO:teuthology.orchestra.run.smithi077.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-27T03:16:50.620 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-27T03:16:50.656 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-27T03:16:50.920 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-27T03:16:50.957 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-27T03:16:51.216 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-27T03:16:51.351 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-27T03:16:51.421 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-test (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:51.477 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-27T03:16:51.548 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up radosgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:51.566 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-27T03:16:51.812 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-27T03:16:51.963 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-27T03:16:52.014 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:16:52.014 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:16:52.356 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-27T03:16:52.499 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-base (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:52.615 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-27T03:16:52.741 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-27T03:16:52.846 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-27T03:16:53.001 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-27T03:16:53.181 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-27T03:16:53.334 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mds (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:53.558 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-27T03:16:53.559 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-27T03:16:53.611 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-27T03:16:53.921 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-27T03:16:54.036 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:54.188 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-27T03:16:54.267 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-27T03:16:54.267 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-27T03:16:54.339 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-27T03:16:54.607 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-27T03:16:54.813 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-osd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:54.884 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-27T03:16:55.070 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-27T03:16:55.070 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-27T03:16:55.237 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-27T03:16:55.613 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-27T03:16:55.876 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:55.915 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-27T03:16:56.002 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:56.178 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mon (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:56.250 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-27T03:16:56.377 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librados2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:56.389 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-27T03:16:56.389 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-27T03:16:56.528 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:56.679 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-27T03:16:56.862 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:56.988 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.081 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.098 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-dashboard (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.215 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libradosstriper1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.249 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-volume (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.341 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-27T03:16:57.602 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-27T03:16:57.618 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librbd1 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.719 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-27T03:16:57.744 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-modules-core (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.845 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:57.871 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-27T03:16:58.041 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-27T03:16:58.041 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-27T03:16:58.522 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs-dev (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:58.681 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rados (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:58.807 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librgw2 (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:16:58.913 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-27T03:16:58.933 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-27T03:16:59.550 INFO:teuthology.orchestra.run.smithi077.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-27T03:16:59.556 DEBUG:teuthology.parallel:result is None 2024-04-27T03:17:00.138 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rbd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:00.247 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up rbd-fuse (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:00.373 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:00.499 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cephfs (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:00.625 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-common (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:01.151 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group ceph....done 2024-04-27T03:17:01.569 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user ceph....done 2024-04-27T03:17:01.729 INFO:teuthology.orchestra.run.smithi110.stdout:Setting system user ceph properties....done 2024-04-27T03:17:01.744 INFO:teuthology.orchestra.run.smithi110.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-27T03:17:01.881 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-27T03:17:02.200 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-27T03:17:02.701 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-test (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:02.827 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up radosgw (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:03.296 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:17:03.296 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-27T03:17:03.737 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-base (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:04.120 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-27T03:17:04.564 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mds (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:04.786 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-27T03:17:04.787 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-27T03:17:05.216 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:05.431 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-27T03:17:05.431 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-27T03:17:05.884 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-osd (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:06.164 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-27T03:17:06.164 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-27T03:17:06.677 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:06.828 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:07.005 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mon (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:07.227 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-27T03:17:07.227 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-27T03:17:07.751 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-cephadm (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:07.995 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:08.200 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-dashboard (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:08.442 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-volume (19.0.0-2447-gb22e2ebd-1jammy) ... 2024-04-27T03:17:09.130 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-27T03:17:09.240 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-27T03:17:09.392 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-27T03:17:10.024 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-27T03:17:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-27T03:17:10.684 DEBUG:teuthology.parallel:result is None 2024-04-27T03:17:10.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T03:17:10.831 DEBUG:teuthology.orchestra.run.smithi077:> dpkg-query -W -f '${Version}' ceph 2024-04-27T03:17:10.850 INFO:teuthology.orchestra.run.smithi077.stdout:19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:17:10.851 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:17:10.851 INFO:teuthology.task.install:The correct ceph version 19.0.0-2447-gb22e2ebd-1jammy is installed. 2024-04-27T03:17:10.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b22e2ebdeb24376882b7bda2a7329c8cccc2276a 2024-04-27T03:17:10.988 DEBUG:teuthology.orchestra.run.smithi110:> dpkg-query -W -f '${Version}' ceph 2024-04-27T03:17:11.008 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:17:11.008 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2447-gb22e2ebd-1jammy 2024-04-27T03:17:11.008 INFO:teuthology.task.install:The correct ceph version 19.0.0-2447-gb22e2ebd-1jammy is installed. 2024-04-27T03:17:11.010 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-27T03:17:11.010 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:11.010 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T03:17:11.025 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:11.025 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-27T03:17:11.065 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-27T03:17:11.066 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:11.066 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T03:17:11.079 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T03:17:11.133 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:11.133 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-04-27T03:17:11.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-27T03:17:11.200 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-27T03:17:11.201 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:11.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T03:17:11.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T03:17:11.269 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:11.269 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-27T03:17:11.282 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-27T03:17:11.337 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-27T03:17:11.337 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:11.337 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T03:17:11.350 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T03:17:11.405 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:11.405 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-04-27T03:17:11.418 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-27T03:17:11.472 INFO:teuthology.run_tasks:Running task ceph... 2024-04-27T03:17:11.557 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-27T03:17:11.557 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2024-04-27T03:17:11.560 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-04-27T03:17:11.572 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-27T03:17:11.572 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-27T03:17:11.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-27T03:17:11.627 INFO:tasks.ceph:Creating extra log directories... 2024-04-27T03:17:11.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-27T03:17:11.671 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-27T03:17:11.693 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-27T03:17:11.693 INFO:tasks.ceph:config {'conf': {'client': {'debug ms': 1, 'rgw gc obj min wait': 15, 'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'osd': {'debug ms': 1, 'debug objclass': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-27T03:17:11.693 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650', 'branch': 'squid', 'description': 'rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7675650', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}}, 'rgw': {'ec-data-pool': True, 'frontend': 'beast'}, 's3tests': {'slow_backend': True}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'osd.0'], ['mgr.x', 'client.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 7340, 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'rgw', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'targets': {'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHwh1HbopmUaqcyPIkoGT2EHqG1tebqFa/9riak5kf46Y1NVmqOVfiFSBvqjmEAg56m6gIJy6c1SqCp4YqJafN8=', 'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC/wIFkg13J1eN7/FIQ6IiAuhaFcolUIVxsW/eOzj9LSwIBwiLSMtFcCQ29aZ4gqASebXvt1eD4xzCFKrd5hjn0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'client': {'debug ms': 1, 'rgw gc obj min wait': 15, 'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'osd': {'debug ms': 1, 'debug objclass': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled'], 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'cluster': 'ceph'}}, {'rgw': {'client.0': None}}, {'radosgw-admin': None}, {'radosgw-admin-rest': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-26_21:40:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-27T03:17:11.694 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-27T03:17:11.727 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-27T03:17:11.733 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2024-04-27T03:17:11.775 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-04-27T03:17:11.789 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:11.789 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T03:17:11.834 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T03:17:11.835 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-04-27T03:17:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T03:17:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:11.883 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-04-27T03:17:11.883 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:11.883 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 03:11:44.907579528 +0000 2024-04-27T03:17:11.883 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 03:11:44.903579564 +0000 2024-04-27T03:17:11.883 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 03:11:44.903579564 +0000 2024-04-27T03:17:11.883 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-04-27T03:17:11.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T03:17:11.937 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T03:17:11.937 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T03:17:11.937 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000442312 s, 1.2 MB/s 2024-04-27T03:17:11.939 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T03:17:11.988 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-04-27T03:17:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T03:17:12.034 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:12.034 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-04-27T03:17:12.034 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:12.034 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 03:11:45.667572739 +0000 2024-04-27T03:17:12.034 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 03:11:45.415574990 +0000 2024-04-27T03:17:12.034 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 03:11:45.415574990 +0000 2024-04-27T03:17:12.035 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-04-27T03:17:12.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T03:17:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T03:17:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T03:17:12.089 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000428397 s, 1.2 MB/s 2024-04-27T03:17:12.091 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T03:17:12.139 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-04-27T03:17:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T03:17:12.186 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:12.186 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-04-27T03:17:12.186 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:12.187 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 03:11:45.899570667 +0000 2024-04-27T03:17:12.187 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 03:11:45.895570702 +0000 2024-04-27T03:17:12.187 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 03:11:45.895570702 +0000 2024-04-27T03:17:12.187 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-04-27T03:17:12.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T03:17:12.240 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T03:17:12.240 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T03:17:12.240 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00034791 s, 1.5 MB/s 2024-04-27T03:17:12.241 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T03:17:12.291 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-04-27T03:17:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T03:17:12.338 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:12.339 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 936 Links: 1 2024-04-27T03:17:12.339 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:12.339 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-04-27 03:11:46.611564306 +0000 2024-04-27T03:17:12.339 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-04-27 03:11:46.391566272 +0000 2024-04-27T03:17:12.339 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-04-27 03:11:46.391566272 +0000 2024-04-27T03:17:12.339 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-04-27T03:17:12.339 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T03:17:12.392 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-04-27T03:17:12.392 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-04-27T03:17:12.392 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000389467 s, 1.3 MB/s 2024-04-27T03:17:12.393 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T03:17:12.443 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1'} 2024-04-27T03:17:12.445 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:12.445 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-04-27T03:17:12.452 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-27T03:17:12.452 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 885 Links: 1 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-27 03:11:44.873202532 +0000 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-27 03:11:44.869202568 +0000 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-27 03:11:44.869202568 +0000 2024-04-27T03:17:12.501 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-27T03:17:12.502 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-27T03:17:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-27T03:17:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-27T03:17:12.555 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000313974 s, 1.6 MB/s 2024-04-27T03:17:12.557 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-27T03:17:12.606 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-04-27T03:17:12.654 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-27T03:17:12.654 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:12.654 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-04-27T03:17:12.654 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:12.654 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-27 03:11:45.581196207 +0000 2024-04-27T03:17:12.655 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-27 03:11:45.341198351 +0000 2024-04-27T03:17:12.655 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-27 03:11:45.341198351 +0000 2024-04-27T03:17:12.655 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-27T03:17:12.655 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-27T03:17:12.706 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-27T03:17:12.707 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-27T03:17:12.707 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000200161 s, 2.6 MB/s 2024-04-27T03:17:12.708 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-27T03:17:12.758 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-04-27T03:17:12.804 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-27T03:17:12.805 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:12.805 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-04-27T03:17:12.805 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:12.805 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-27 03:11:46.081191742 +0000 2024-04-27T03:17:12.805 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-27 03:11:45.821194064 +0000 2024-04-27T03:17:12.805 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-27 03:11:45.821194064 +0000 2024-04-27T03:17:12.805 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-27T03:17:12.805 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-27T03:17:12.859 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-27T03:17:12.859 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-27T03:17:12.860 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000318465 s, 1.6 MB/s 2024-04-27T03:17:12.861 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-27T03:17:12.910 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-04-27T03:17:12.956 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-27T03:17:12.957 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-27T03:17:12.957 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-04-27T03:17:12.957 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-27T03:17:12.957 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-27 03:11:46.569187383 +0000 2024-04-27T03:17:12.957 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-27 03:11:46.305189741 +0000 2024-04-27T03:17:12.957 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-27 03:11:46.305189741 +0000 2024-04-27T03:17:12.957 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2024-04-27T03:17:12.957 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-27T03:17:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-27T03:17:13.010 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-27T03:17:13.011 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000195905 s, 2.6 MB/s 2024-04-27T03:17:13.011 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-27T03:17:13.058 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-27T03:17:13.058 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2024-04-27T03:17:13.058 INFO:tasks.ceph:Generating config... 2024-04-27T03:17:13.059 INFO:tasks.ceph:[client] debug ms = 1 2024-04-27T03:17:13.059 INFO:tasks.ceph:[client] rgw gc obj min wait = 15 2024-04-27T03:17:13.059 INFO:tasks.ceph:[client] debug rgw = 20 2024-04-27T03:17:13.059 INFO:tasks.ceph:[client] setgroup = ceph 2024-04-27T03:17:13.060 INFO:tasks.ceph:[client] setuser = ceph 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] debug objclass = 20 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-27T03:17:13.060 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-27T03:17:13.060 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-27T03:17:13.061 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-27T03:17:13.061 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-27T03:17:13.061 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-27T03:17:13.061 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-27T03:17:13.061 INFO:tasks.ceph:Setting up mon.a... 2024-04-27T03:17:13.061 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-27T03:17:13.095 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/ceph.keyring 2024-04-27T03:17:13.116 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-27T03:17:13.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-27T03:17:13.245 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.77')] 2024-04-27T03:17:13.245 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug objclass': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1, 'rgw gc obj min wait': 15, 'debug rgw': 20, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2024-04-27T03:17:13.246 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:13.246 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-27T03:17:13.294 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:13.371 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:17:13.372 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:17:13.372 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:13.372 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3 2024-04-27T03:17:13.372 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = quincy 2024-04-27T03:17:13.372 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2024-04-27T03:17:13.373 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3 2024-04-27T03:17:13.373 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2024-04-27T03:17:13.368730+0000 2024-04-27T03:17:13.373 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-04-27T03:17:13.368730+0000 2024-04-27T03:17:13.373 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 17 (quincy) 2024-04-27T03:17:13.373 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2024-04-27T03:17:13.373 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-04-27T03:17:13.373 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-27T03:17:13.374 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-27T03:17:13.423 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3... 2024-04-27T03:17:13.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-27T03:17:13.470 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-04-27T03:17:13.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.497 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-04-27T03:17:13.498 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.77 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.499 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.500 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: debug objclass = 20 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore allocator = bitmap 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-04-27T03:17:13.501 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-04-27T03:17:13.502 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.503 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-04-27T03:17:13.504 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout: rgw gc obj min wait = 15 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout: debug rgw = 20 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout: setgroup = ceph 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout: setuser = ceph 2024-04-27T03:17:13.505 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.506 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-04-27T03:17:13.507 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-04-27T03:17:13.508 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = 172.21.15.77 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-04-27T03:17:13.509 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug trim objects = true 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-04-27T03:17:13.510 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: debug objclass = 20 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore allocator = bitmap 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 1/20 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 1/20 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 4/10 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2024-04-27T03:17:13.511 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-04-27T03:17:13.512 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-04-27T03:17:13.513 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: rgw gc obj min wait = 15 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: debug rgw = 20 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: setgroup = ceph 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout: setuser = ceph 2024-04-27T03:17:13.514 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-04-27T03:17:13.516 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-27T03:17:13.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-27T03:17:13.605 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-27T03:17:13.605 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:13.605 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-27T03:17:13.610 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:13.611 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-27T03:17:13.657 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T03:17:13.657 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:13.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-27T03:17:13.658 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-27T03:17:13.724 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:13.724 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:13.770 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:17:13.770 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:13.770 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-27T03:17:13.770 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-27T03:17:13.794 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:13.794 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:13.841 INFO:tasks.ceph:Setting up mon nodes... 2024-04-27T03:17:13.841 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-27T03:17:13.841 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-27T03:17:13.924 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-27T03:17:13.940 INFO:tasks.ceph:Setting up mds nodes... 2024-04-27T03:17:13.940 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-27T03:17:13.941 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-27T03:17:14.019 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-27T03:17:14.030 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-27T03:17:14.031 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'}} 2024-04-27T03:17:14.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-27T03:17:14.045 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1'} 2024-04-27T03:17:14.045 INFO:tasks.ceph:role: osd.0 2024-04-27T03:17:14.045 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T03:17:14.045 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-27T03:17:14.106 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-27T03:17:14.107 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-27T03:17:14.107 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-27T03:17:14.564 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-04-27T03:17:14.566 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-04-27T03:17:14.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-27T03:17:14.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-27T03:17:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:17:14.669 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: /sbin/restorecon: command not found 2024-04-27T03:17:14.669 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:14.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:14.774+0000 7fcc450b77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-27T03:17:14.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:14.778+0000 7fcc450b77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-27T03:17:14.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:14.778+0000 7fcc450b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-27T03:17:14.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:14.778+0000 7fcc450b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-27T03:17:14.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:14.778+0000 7fcc450b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-27T03:17:14.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:14.778+0000 7fcc450b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-27T03:17:17.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-27T03:17:17.783 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-27T03:17:17.796 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-27T03:17:17.796 INFO:tasks.ceph:role: osd.1 2024-04-27T03:17:17.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:17:17.797 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-27T03:17:17.856 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-27T03:17:17.857 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-27T03:17:17.880 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-04-27T03:17:17.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-04-27T03:17:17.882 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-04-27T03:17:17.940 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-27T03:17:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:17:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2024-04-27T03:17:17.994 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-27T03:17:18.048 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-27T03:17:18.048 INFO:tasks.ceph:role: osd.2 2024-04-27T03:17:18.048 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:17:18.048 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-27T03:17:18.104 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-27T03:17:18.104 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-27T03:17:18.105 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-27T03:17:18.128 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-04-27T03:17:18.130 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-04-27T03:17:18.130 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2024-04-27T03:17:18.187 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-27T03:17:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:17:18.237 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2024-04-27T03:17:18.238 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-27T03:17:18.291 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.2': '/dev/vg_nvme/lv_2', 'osd.3': '/dev/vg_nvme/lv_3'} 2024-04-27T03:17:18.291 INFO:tasks.ceph:role: osd.3 2024-04-27T03:17:18.292 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:17:18.292 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-27T03:17:18.347 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-27T03:17:18.347 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-27T03:17:18.348 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-27T03:17:18.374 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-04-27T03:17:18.375 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-04-27T03:17:18.375 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2024-04-27T03:17:18.436 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-27T03:17:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:17:18.490 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2024-04-27T03:17:18.491 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:18.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:18.601+0000 7f5b3da687c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-27T03:17:18.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:18.601+0000 7f5b3da687c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-27T03:17:18.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:18.601+0000 7f5b3da687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-27T03:17:18.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:18.601+0000 7f5b3da687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-27T03:17:18.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:18.601+0000 7f5b3da687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-27T03:17:18.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:18.601+0000 7f5b3da687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-27T03:17:21.577 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-27T03:17:21.590 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:21.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:21.661+0000 7fda087367c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-27T03:17:21.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:21.661+0000 7fda087367c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-27T03:17:21.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:21.661+0000 7fda087367c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-27T03:17:21.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:21.661+0000 7fda087367c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-27T03:17:21.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:21.661+0000 7fda087367c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-27T03:17:21.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:21.661+0000 7fda087367c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-27T03:17:24.641 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-27T03:17:24.700 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:24.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:24.773+0000 7fcdd7ff57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-27T03:17:24.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:24.773+0000 7fcdd7ff57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-27T03:17:24.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:24.773+0000 7fcdd7ff57c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-27T03:17:24.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:24.773+0000 7fcdd7ff57c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-27T03:17:24.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:24.773+0000 7fcdd7ff57c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-27T03:17:24.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:24.773+0000 7fcdd7ff57c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-27T03:17:27.746 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-27T03:17:27.802 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-27T03:17:27.802 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:27.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-27T03:17:27.816 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:27.817 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-27T03:17:27.855 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:27.856 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-27T03:17:27.911 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:27.911 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-27T03:17:27.964 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:27.965 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-27T03:17:28.024 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:28.024 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-27T03:17:28.072 INFO:tasks.ceph:Adding keys to all mons... 2024-04-27T03:17:28.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-27T03:17:28.085 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2024-04-27T03:17:28.085 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQA6bixmIOeSLhAAXs7dkZGAb3fHe8TJOGh6LQ== 2024-04-27T03:17:28.085 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2024-04-27T03:17:28.085 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQA5bixmu2myNhAAm+XvWGfAVn42NCeDhoOV7w== 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQA+bixmKxbxIxAA5uUxU1eyPUkGbDTj6x0hkQ== 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBBbixmeXKIJxAAoKo8QRAQtz6YgCx6sH9+/Q== 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.3] 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBEbixmJ0IyLhAA/Pbxbnpw74BhlCk/idZ5Yw== 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-04-27T03:17:28.086 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQA6bixmfILIABAA7l3ZPyyePBltgezOVKkF4w== 2024-04-27T03:17:28.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-27T03:17:28.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-27T03:17:28.262 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-27T03:17:28.350 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-27T03:17:28.392 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-27T03:17:28.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-27T03:17:28.555 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-27T03:17:28.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-27T03:17:28.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-27T03:17:28.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-27T03:17:28.705 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-27T03:17:28.753 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-27T03:17:28.754 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-27T03:17:28.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-27T03:17:28.798 INFO:tasks.ceph.mon.a:Started 2024-04-27T03:17:28.798 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-27T03:17:28.799 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-27T03:17:28.799 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-27T03:17:28.801 INFO:tasks.ceph.mgr.x:Started 2024-04-27T03:17:28.801 INFO:tasks.ceph:Setting crush tunables to default 2024-04-27T03:17:28.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd crush tunables default 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2eec7640 1 Processor -- start 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2eec7640 1 -- start start 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2eec7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 0x7f3d28133090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2eec7640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d28096e20 con 0x7f3d28089fa0 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2eec7640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f3d2808bd70 con 0x7f3d28132cb0 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2cc63640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 0x7f3d28133090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2cc63640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 0x7f3d28133090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42366/0 (socket says 172.21.15.77:42366) 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2cc63640 1 -- 172.21.15.77:0/2350054999 learned_addr learned my addr 172.21.15.77:0/2350054999 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:28.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2cc63640 1 -- 172.21.15.77:0/2350054999 >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 msgr2=0x7f3d28133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-27T03:17:28.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2cc63640 1 -- 172.21.15.77:0/2350054999 >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 msgr2=0x7f3d28133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-27T03:17:28.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2cc63640 1 --2- 172.21.15.77:0/2350054999 >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 0x7f3d28133090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-27T03:17:28.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.906+0000 7f3d2cc63640 1 --2- 172.21.15.77:0/2350054999 >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 0x7f3d28133090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625447350 0 0) 0x7f3d28096e20 con 0x7f3d28089fa0 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d10003530 con 0x7f3d28089fa0 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4227521524 0 0) 0x7f3d10003530 con 0x7f3d28089fa0 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 msgr2=0x7f3d28133090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 --2- 172.21.15.77:0/2350054999 >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 0x7f3d28133090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2808c080 con 0x7f3d28089fa0 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 205+0+0 (unknown 201508238 0 0) 0x7f3d1c0015b0 con 0x7f3d28089fa0 2024-04-27T03:17:28.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 >> v1:172.21.15.77:6789/0 conn(0x7f3d28089fa0 legacy=0x7f3d2808a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 --2- 172.21.15.77:0/2350054999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3d10004530 con 0x7f3d100039c0 2024-04-27T03:17:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d2cc63640 1 --2- 172.21.15.77:0/2350054999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:28.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.930+0000 7f3d2cc63640 1 -- 172.21.15.77:0/2350054999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d10003530 con 0x7f3d100039c0 2024-04-27T03:17:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2cc63640 1 --2- 172.21.15.77:0/2350054999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d10003dc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009810 tx=0x7f3d1801c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf41f498093d8ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d1800f040 con 0x7f3d100039c0 2024-04-27T03:17:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d180a9530 con 0x7f3d100039c0 2024-04-27T03:17:28.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d277fe640 1 -- 172.21.15.77:0/2350054999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d180a9850 con 0x7f3d100039c0 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 -- 172.21.15.77:0/2350054999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 msgr2=0x7f3d10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 --2- 172.21.15.77:0/2350054999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d10003dc0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009810 tx=0x7f3d1801c6b0 comp rx=0 tx=0).stop 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 -- 172.21.15.77:0/2350054999 shutdown_connections 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 --2- 172.21.15.77:0/2350054999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d10003dc0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 --2- 172.21.15.77:0/2350054999 >> v2:172.21.15.77:3300/0 conn(0x7f3d28132cb0 0x7f3d28133090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 -- 172.21.15.77:0/2350054999 >> 172.21.15.77:0/2350054999 conn(0x7f3d280993d0 msgr2=0x7f3d280997e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 -- 172.21.15.77:0/2350054999 shutdown_connections 2024-04-27T03:17:28.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 -- 172.21.15.77:0/2350054999 wait complete. 2024-04-27T03:17:28.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 Processor -- start 2024-04-27T03:17:28.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.958+0000 7f3d2eec7640 1 -- start start 2024-04-27T03:17:28.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2eec7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d281ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2eec7640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2813b550 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2cc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d281ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2cc63640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d281ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42392/0 (socket says 172.21.15.77:42392) 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2cc63640 1 -- 172.21.15.77:0/872321253 learned_addr learned my addr 172.21.15.77:0/872321253 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2cc63640 1 -- 172.21.15.77:0/872321253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d28137360 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2cc63640 1 --2- 172.21.15.77:0/872321253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d281ca600 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3d180192c0 tx=0x7f3d10005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d257fa640 1 -- 172.21.15.77:0/872321253 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d180c1070 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d257fa640 1 -- 172.21.15.77:0/872321253 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1800f000 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d28136c80 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d257fa640 1 -- 172.21.15.77:0/872321253 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d180bc470 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d281378f0 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d257fa640 1 -- 172.21.15.77:0/872321253 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3d180c4070 con 0x7f3d100039c0 2024-04-27T03:17:28.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d257fa640 1 -- 172.21.15.77:0/872321253 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3d180b8030 con 0x7f3d100039c0 2024-04-27T03:17:28.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.962+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d28062e50 con 0x7f3d100039c0 2024-04-27T03:17:28.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:28.970+0000 7f3d257fa640 1 -- 172.21.15.77:0/872321253 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3d180cb080 con 0x7f3d100039c0 2024-04-27T03:17:29.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.030+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3d28072520 con 0x7f3d100039c0 2024-04-27T03:17:29.041 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d257fa640 1 -- 172.21.15.77:0/872321253 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f3d180ad020 con 0x7f3d100039c0 2024-04-27T03:17:29.041 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-04-27T03:17:29.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 msgr2=0x7f3d281ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d2eec7640 1 --2- 172.21.15.77:0/872321253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d281ca600 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3d180192c0 tx=0x7f3d10005840 comp rx=0 tx=0).stop 2024-04-27T03:17:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 shutdown_connections 2024-04-27T03:17:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d2eec7640 1 --2- 172.21.15.77:0/872321253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d100039c0 0x7f3d281ca600 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 >> 172.21.15.77:0/872321253 conn(0x7f3d280993d0 msgr2=0x7f3d28132450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 shutdown_connections 2024-04-27T03:17:29.043 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:29.034+0000 7f3d2eec7640 1 -- 172.21.15.77:0/872321253 wait complete. 2024-04-27T03:17:29.058 INFO:tasks.ceph:check_enable_crimson: False 2024-04-27T03:17:29.058 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-27T03:17:29.059 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-04-27T03:17:29.059 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-27T03:17:29.069 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:29.069 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-27T03:17:29.083 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:29.083 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-27T03:17:29.134 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-27T03:17:29.135 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-27T03:17:29.187 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7 0 2024-04-27T03:17:29.188 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:29.177+0000 7f0cbb0b7180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-27T03:17:29.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f8364983640 1 Processor -- start 2024-04-27T03:17:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f8364983640 1 -- start start 2024-04-27T03:17:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f8364983640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8360113720 0x7f8360113b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f8364983640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83601140d0 con 0x7f836010ee70 2024-04-27T03:17:29.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f8364983640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f836010fa00 con 0x7f8360113720 2024-04-27T03:17:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f835e59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8360113720 0x7f8360113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f8357fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f836010ee70 0x7f836010f2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:53726/0 (socket says 172.21.15.110:53726) 2024-04-27T03:17:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f8357fff640 1 -- 172.21.15.110:0/2118700556 learned_addr learned my addr 172.21.15.110:0/2118700556 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:17:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f835dd9b640 1 -- 172.21.15.110:0/2118700556 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009770541 0 0) 0x7f83601140d0 con 0x7f836010ee70 2024-04-27T03:17:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.341+0000 7f835dd9b640 1 -- 172.21.15.110:0/2118700556 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f834c003530 con 0x7f836010ee70 2024-04-27T03:17:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f835e59c640 1 -- 172.21.15.110:0/2118700556 >> v1:172.21.15.77:6789/0 conn(0x7f836010ee70 legacy=0x7f836010f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f835e59c640 1 -- 172.21.15.110:0/2118700556 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f836010fbb0 con 0x7f8360113720 2024-04-27T03:17:29.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f835e59c640 1 --2- 172.21.15.110:0/2118700556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8360113720 0x7f8360113b00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8350009740 tx=0x7f835001c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ba0ecadb5d967a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:29.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f835dd9b640 1 -- 172.21.15.110:0/2118700556 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83500b1070 con 0x7f8360113720 2024-04-27T03:17:29.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f835dd9b640 1 -- 172.21.15.110:0/2118700556 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83500ac520 con 0x7f8360113720 2024-04-27T03:17:29.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f835dd9b640 1 -- 172.21.15.110:0/2118700556 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83500ac820 con 0x7f8360113720 2024-04-27T03:17:29.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f8364983640 1 -- 172.21.15.110:0/2118700556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8360113720 msgr2=0x7f8360113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f8364983640 1 --2- 172.21.15.110:0/2118700556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8360113720 0x7f8360113b00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8350009740 tx=0x7f835001c370 comp rx=0 tx=0).stop 2024-04-27T03:17:29.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f8364983640 1 -- 172.21.15.110:0/2118700556 shutdown_connections 2024-04-27T03:17:29.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f8364983640 1 --2- 172.21.15.110:0/2118700556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8360113720 0x7f8360113b00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f8364983640 1 -- 172.21.15.110:0/2118700556 >> 172.21.15.110:0/2118700556 conn(0x7f836006f930 msgr2=0x7f8360071d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:29.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f8364983640 1 -- 172.21.15.110:0/2118700556 shutdown_connections 2024-04-27T03:17:29.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.345+0000 7f8364983640 1 -- 172.21.15.110:0/2118700556 wait complete. 2024-04-27T03:17:29.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8364983640 1 Processor -- start 2024-04-27T03:17:29.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8364983640 1 -- start start 2024-04-27T03:17:29.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8364983640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f836010ee70 0x7f8360070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:29.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8364983640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f836011d200 con 0x7f836010ee70 2024-04-27T03:17:29.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f835e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f836010ee70 0x7f8360070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:29.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f835e59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f836010ee70 0x7f8360070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:55624/0 (socket says 172.21.15.110:55624) 2024-04-27T03:17:29.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f835e59c640 1 -- 172.21.15.110:0/4116932767 learned_addr learned my addr 172.21.15.110:0/4116932767 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:29.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f835e59c640 1 -- 172.21.15.110:0/4116932767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83600706a0 con 0x7f836010ee70 2024-04-27T03:17:29.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f835e59c640 1 --2- 172.21.15.110:0/4116932767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f836010ee70 0x7f8360070160 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f83500af040 tx=0x7f835001abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:29.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8356ffd640 1 -- 172.21.15.110:0/4116932767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83500b1050 con 0x7f836010ee70 2024-04-27T03:17:29.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83600708d0 con 0x7f836010ee70 2024-04-27T03:17:29.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8356ffd640 1 -- 172.21.15.110:0/4116932767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83500b2040 con 0x7f836010ee70 2024-04-27T03:17:29.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8356ffd640 1 -- 172.21.15.110:0/4116932767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83500ab8d0 con 0x7f836010ee70 2024-04-27T03:17:29.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8360070d30 con 0x7f836010ee70 2024-04-27T03:17:29.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8356ffd640 1 -- 172.21.15.110:0/4116932767 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f83500bf030 con 0x7f836010ee70 2024-04-27T03:17:29.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8356ffd640 1 -- 172.21.15.110:0/4116932767 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f83500bd070 con 0x7f836010ee70 2024-04-27T03:17:29.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.349+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8360062e50 con 0x7f836010ee70 2024-04-27T03:17:29.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.357+0000 7f8356ffd640 1 -- 172.21.15.110:0/4116932767 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f83500c6060 con 0x7f836010ee70 2024-04-27T03:17:29.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.425+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7", "id": 0} v 0) -- 0x7f8360071070 con 0x7f836010ee70 2024-04-27T03:17:29.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.425+0000 7f8356ffd640 1 -- 172.21.15.110:0/4116932767 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f8360071070 con 0x7f836010ee70 2024-04-27T03:17:29.436 INFO:teuthology.orchestra.run.smithi110.stdout:0 2024-04-27T03:17:29.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.429+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f836010ee70 msgr2=0x7f8360070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.429+0000 7f8364983640 1 --2- 172.21.15.110:0/4116932767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f836010ee70 0x7f8360070160 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f83500af040 tx=0x7f835001abb0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.429+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 shutdown_connections 2024-04-27T03:17:29.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.429+0000 7f8364983640 1 --2- 172.21.15.110:0/4116932767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f836010ee70 0x7f8360070160 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.429+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 >> 172.21.15.110:0/4116932767 conn(0x7f836006f930 msgr2=0x7f8360072440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:29.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.429+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 shutdown_connections 2024-04-27T03:17:29.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.429+0000 7f8364983640 1 -- 172.21.15.110:0/4116932767 wait complete. 2024-04-27T03:17:29.453 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 74a30006-c87c-4e0e-9e94-9020affcae97 1 2024-04-27T03:17:29.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f1500828640 1 Processor -- start 2024-04-27T03:17:29.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f1500828640 1 -- start start 2024-04-27T03:17:29.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f1500828640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f14fc1147c0 0x7f14fc114ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:29.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f1500828640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14fc115260 con 0x7f14fc10ed70 2024-04-27T03:17:29.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f1500828640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f14fc10f930 con 0x7f14fc1147c0 2024-04-27T03:17:29.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f14fa59c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f14fc1147c0 0x7f14fc114ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:29.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f14f9d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f14fc10ed70 0x7f14fc10f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:53734/0 (socket says 172.21.15.110:53734) 2024-04-27T03:17:29.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f14f9d9b640 1 -- 172.21.15.110:0/507483562 learned_addr learned my addr 172.21.15.110:0/507483562 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:17:29.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f14fa59c640 1 -- 172.21.15.110:0/507483562 >> v1:172.21.15.77:6789/0 conn(0x7f14fc10ed70 legacy=0x7f14fc10f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.557+0000 7f14fa59c640 1 -- 172.21.15.110:0/507483562 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14fc10fae0 con 0x7f14fc1147c0 2024-04-27T03:17:29.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f14fa59c640 1 --2- 172.21.15.110:0/507483562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc1147c0 0x7f14fc114ba0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009740 tx=0x7f14ec01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f714945df3e2ccc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:29.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f14f959a640 1 -- 172.21.15.110:0/507483562 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14ec0b1070 con 0x7f14fc1147c0 2024-04-27T03:17:29.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f14f959a640 1 -- 172.21.15.110:0/507483562 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ec0ac520 con 0x7f14fc1147c0 2024-04-27T03:17:29.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f14f959a640 1 -- 172.21.15.110:0/507483562 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14ec0ac820 con 0x7f14fc1147c0 2024-04-27T03:17:29.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 -- 172.21.15.110:0/507483562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc1147c0 msgr2=0x7f14fc114ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 --2- 172.21.15.110:0/507483562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc1147c0 0x7f14fc114ba0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009740 tx=0x7f14ec01c370 comp rx=0 tx=0).stop 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 -- 172.21.15.110:0/507483562 shutdown_connections 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 --2- 172.21.15.110:0/507483562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc1147c0 0x7f14fc114ba0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 -- 172.21.15.110:0/507483562 >> 172.21.15.110:0/507483562 conn(0x7f14fc070120 msgr2=0x7f14fc072540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 -- 172.21.15.110:0/507483562 shutdown_connections 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 -- 172.21.15.110:0/507483562 wait complete. 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 Processor -- start 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 -- start start 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc10ed70 0x7f14fc1a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f1500828640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f14fc06bea0 con 0x7f14fc10ed70 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.561+0000 7f14fa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc10ed70 0x7f14fc1a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14fa59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc10ed70 0x7f14fc1a4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:55632/0 (socket says 172.21.15.110:55632) 2024-04-27T03:17:29.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14fa59c640 1 -- 172.21.15.110:0/2214388339 learned_addr learned my addr 172.21.15.110:0/2214388339 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14fa59c640 1 -- 172.21.15.110:0/2214388339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14fc1a51c0 con 0x7f14fc10ed70 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14fa59c640 1 --2- 172.21.15.110:0/2214388339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc10ed70 0x7f14fc1a4bf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f14ec01aa20 tx=0x7f14ec01abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14f2ffd640 1 -- 172.21.15.110:0/2214388339 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14ec0b1050 con 0x7f14fc10ed70 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14fc1a5450 con 0x7f14fc10ed70 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14f2ffd640 1 -- 172.21.15.110:0/2214388339 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ec0b2040 con 0x7f14fc10ed70 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14f2ffd640 1 -- 172.21.15.110:0/2214388339 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14ec0aba40 con 0x7f14fc10ed70 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14fc1a94a0 con 0x7f14fc10ed70 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14f2ffd640 1 -- 172.21.15.110:0/2214388339 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f14ec0bf030 con 0x7f14fc10ed70 2024-04-27T03:17:29.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f14f2ffd640 1 -- 172.21.15.110:0/2214388339 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f14ec0bd070 con 0x7f14fc10ed70 2024-04-27T03:17:29.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.565+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14fc062e50 con 0x7f14fc10ed70 2024-04-27T03:17:29.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.573+0000 7f14f2ffd640 1 -- 172.21.15.110:0/2214388339 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f14ec0c6060 con 0x7f14fc10ed70 2024-04-27T03:17:29.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.641+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "74a30006-c87c-4e0e-9e94-9020affcae97", "id": 1} v 0) -- 0x7f14fc1a5620 con 0x7f14fc10ed70 2024-04-27T03:17:29.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f14f2ffd640 1 -- 172.21.15.110:0/2214388339 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "74a30006-c87c-4e0e-9e94-9020affcae97", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f14ec0a9070 con 0x7f14fc10ed70 2024-04-27T03:17:29.653 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-04-27T03:17:29.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc10ed70 msgr2=0x7f14fc1a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f1500828640 1 --2- 172.21.15.110:0/2214388339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc10ed70 0x7f14fc1a4bf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f14ec01aa20 tx=0x7f14ec01abb0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 shutdown_connections 2024-04-27T03:17:29.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f1500828640 1 --2- 172.21.15.110:0/2214388339 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f14fc10ed70 0x7f14fc1a4bf0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 >> 172.21.15.110:0/2214388339 conn(0x7f14fc070120 msgr2=0x7f14fc071970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:29.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 shutdown_connections 2024-04-27T03:17:29.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.645+0000 7f1500828640 1 -- 172.21.15.110:0/2214388339 wait complete. 2024-04-27T03:17:29.671 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 1bd2dbfb-57cb-4673-9684-d9427036427d 2 2024-04-27T03:17:29.704 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:29.697+0000 7f0cbb0b7180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-27T03:17:29.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f0249935640 1 Processor -- start 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f0249935640 1 -- start start 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f0249935640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0244118730 0x7f024411ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f0249935640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f024411b060 con 0x7f02441147e0 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f0249935640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f024411c240 con 0x7f0244118730 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f02427fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0244118730 0x7f024411ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f02427fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0244118730 0x7f024411ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:55634/0 (socket says 172.21.15.110:55634) 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f0242ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f02441147e0 0x7f0244114bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:53738/0 (socket says 172.21.15.110:53738) 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f02427fc640 1 -- 172.21.15.110:0/3447063608 learned_addr learned my addr 172.21.15.110:0/3447063608 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f02427fc640 1 -- 172.21.15.110:0/3447063608 >> v1:172.21.15.77:6789/0 conn(0x7f02441147e0 legacy=0x7f0244114bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.773+0000 7f02427fc640 1 -- 172.21.15.110:0/3447063608 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024411c3c0 con 0x7f0244118730 2024-04-27T03:17:29.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f02427fc640 1 --2- 172.21.15.110:0/3447063608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0244118730 0x7f024411ab20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f02380097f0 tx=0x7f023801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b2fde3dda967d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:29.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0241ffb640 1 -- 172.21.15.110:0/3447063608 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02380ab580 con 0x7f0244118730 2024-04-27T03:17:29.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0241ffb640 1 -- 172.21.15.110:0/3447063608 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02380b0070 con 0x7f0244118730 2024-04-27T03:17:29.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0241ffb640 1 -- 172.21.15.110:0/3447063608 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02380ab870 con 0x7f0244118730 2024-04-27T03:17:29.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 -- 172.21.15.110:0/3447063608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0244118730 msgr2=0x7f024411ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 --2- 172.21.15.110:0/3447063608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0244118730 0x7f024411ab20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f02380097f0 tx=0x7f023801c430 comp rx=0 tx=0).stop 2024-04-27T03:17:29.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 -- 172.21.15.110:0/3447063608 shutdown_connections 2024-04-27T03:17:29.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 --2- 172.21.15.110:0/3447063608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0244118730 0x7f024411ab20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 -- 172.21.15.110:0/3447063608 >> 172.21.15.110:0/3447063608 conn(0x7f0244070050 msgr2=0x7f0244072470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:29.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 -- 172.21.15.110:0/3447063608 shutdown_connections 2024-04-27T03:17:29.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 -- 172.21.15.110:0/3447063608 wait complete. 2024-04-27T03:17:29.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 Processor -- start 2024-04-27T03:17:29.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 -- start start 2024-04-27T03:17:29.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441147e0 0x7f024410ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:29.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0249935640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f024411bf20 con 0x7f02441147e0 2024-04-27T03:17:29.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0242ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441147e0 0x7f024410ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:29.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0242ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441147e0 0x7f024410ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:55650/0 (socket says 172.21.15.110:55650) 2024-04-27T03:17:29.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.777+0000 7f0242ffd640 1 -- 172.21.15.110:0/3946978683 learned_addr learned my addr 172.21.15.110:0/3946978683 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:29.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f0242ffd640 1 -- 172.21.15.110:0/3946978683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024410f380 con 0x7f02441147e0 2024-04-27T03:17:29.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f0242ffd640 1 --2- 172.21.15.110:0/3946978683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441147e0 0x7f024410ee40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f023400c410 tx=0x7f023400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:29.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f022bfff640 1 -- 172.21.15.110:0/3946978683 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234007410 con 0x7f02441147e0 2024-04-27T03:17:29.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024410f5b0 con 0x7f02441147e0 2024-04-27T03:17:29.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f022bfff640 1 -- 172.21.15.110:0/3946978683 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02340075b0 con 0x7f02441147e0 2024-04-27T03:17:29.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f022bfff640 1 -- 172.21.15.110:0/3946978683 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234004920 con 0x7f02441147e0 2024-04-27T03:17:29.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024410fa70 con 0x7f02441147e0 2024-04-27T03:17:29.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f022bfff640 1 -- 172.21.15.110:0/3946978683 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f023401a020 con 0x7f02441147e0 2024-04-27T03:17:29.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f022bfff640 1 -- 172.21.15.110:0/3946978683 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f0234013820 con 0x7f02441147e0 2024-04-27T03:17:29.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.781+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0244062e50 con 0x7f02441147e0 2024-04-27T03:17:29.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.785+0000 7f022bfff640 1 -- 172.21.15.110:0/3946978683 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0234021070 con 0x7f02441147e0 2024-04-27T03:17:29.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.853+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1bd2dbfb-57cb-4673-9684-d9427036427d", "id": 2} v 0) -- 0x7f024410fcf0 con 0x7f02441147e0 2024-04-27T03:17:29.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.857+0000 7f022bfff640 1 -- 172.21.15.110:0/3946978683 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1bd2dbfb-57cb-4673-9684-d9427036427d", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f0234013b00 con 0x7f02441147e0 2024-04-27T03:17:29.867 INFO:teuthology.orchestra.run.smithi110.stdout:2 2024-04-27T03:17:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.861+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441147e0 msgr2=0x7f024410ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:29.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.861+0000 7f0249935640 1 --2- 172.21.15.110:0/3946978683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441147e0 0x7f024410ee40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f023400c410 tx=0x7f023400c8d0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.861+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 shutdown_connections 2024-04-27T03:17:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.861+0000 7f0249935640 1 --2- 172.21.15.110:0/3946978683 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f02441147e0 0x7f024410ee40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.861+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 >> 172.21.15.110:0/3946978683 conn(0x7f0244070050 msgr2=0x7f0244070430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.861+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 shutdown_connections 2024-04-27T03:17:29.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:29.861+0000 7f0249935640 1 -- 172.21.15.110:0/3946978683 wait complete. 2024-04-27T03:17:29.884 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new d73614e8-5d01-4f79-8b12-0e08dabad6b1 3 2024-04-27T03:17:29.918 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:29.909+0000 7f0cbb0b7180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-27T03:17:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72f2328640 1 Processor -- start 2024-04-27T03:17:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72f2328640 1 -- start start 2024-04-27T03:17:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72f2328640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72ec118730 0x7f72ec11ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72f2328640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72ec11b060 con 0x7f72ec1158c0 2024-04-27T03:17:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72f2328640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f72ec11c240 con 0x7f72ec118730 2024-04-27T03:17:30.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72e37fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f72ec118730 0x7f72ec11ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:30.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72ebfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f72ec1158c0 0x7f72ec115ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:53744/0 (socket says 172.21.15.110:53744) 2024-04-27T03:17:30.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72ebfff640 1 -- 172.21.15.110:0/458461471 learned_addr learned my addr 172.21.15.110:0/458461471 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:17:30.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72e37fe640 1 -- 172.21.15.110:0/458461471 >> v1:172.21.15.77:6789/0 conn(0x7f72ec1158c0 legacy=0x7f72ec115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.017+0000 7f72e37fe640 1 -- 172.21.15.110:0/458461471 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ec11c3c0 con 0x7f72ec118730 2024-04-27T03:17:30.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72e37fe640 1 --2- 172.21.15.110:0/458461471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec118730 0x7f72ec11ab20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009740 tx=0x7f72d401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918797e673ef3cf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:30.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72eb7fe640 1 -- 172.21.15.110:0/458461471 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d40b1070 con 0x7f72ec118730 2024-04-27T03:17:30.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72eb7fe640 1 -- 172.21.15.110:0/458461471 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72d40ac520 con 0x7f72ec118730 2024-04-27T03:17:30.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72eb7fe640 1 -- 172.21.15.110:0/458461471 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72d40ac820 con 0x7f72ec118730 2024-04-27T03:17:30.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72f2328640 1 -- 172.21.15.110:0/458461471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec118730 msgr2=0x7f72ec11ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72f2328640 1 --2- 172.21.15.110:0/458461471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec118730 0x7f72ec11ab20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f72d4009740 tx=0x7f72d401c370 comp rx=0 tx=0).stop 2024-04-27T03:17:30.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72f2328640 1 -- 172.21.15.110:0/458461471 shutdown_connections 2024-04-27T03:17:30.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72f2328640 1 --2- 172.21.15.110:0/458461471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec118730 0x7f72ec11ab20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:30.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72f2328640 1 -- 172.21.15.110:0/458461471 >> 172.21.15.110:0/458461471 conn(0x7f72ec0700f0 msgr2=0x7f72ec072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:30.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72f2328640 1 -- 172.21.15.110:0/458461471 shutdown_connections 2024-04-27T03:17:30.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.021+0000 7f72f2328640 1 -- 172.21.15.110:0/458461471 wait complete. 2024-04-27T03:17:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72f2328640 1 Processor -- start 2024-04-27T03:17:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72f2328640 1 -- start start 2024-04-27T03:17:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72f2328640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1158c0 0x7f72ec10ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72f2328640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f72ec11bf20 con 0x7f72ec1158c0 2024-04-27T03:17:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1158c0 0x7f72ec10ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72ebfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1158c0 0x7f72ec10ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:55672/0 (socket says 172.21.15.110:55672) 2024-04-27T03:17:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72ebfff640 1 -- 172.21.15.110:0/4241993066 learned_addr learned my addr 172.21.15.110:0/4241993066 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:30.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72ebfff640 1 -- 172.21.15.110:0/4241993066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ec10f380 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72ebfff640 1 --2- 172.21.15.110:0/4241993066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1158c0 0x7f72ec10ee40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00c480 tx=0x7f72dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72e97fa640 1 -- 172.21.15.110:0/4241993066 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72dc017070 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ec10f610 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72e97fa640 1 -- 172.21.15.110:0/4241993066 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72dc005c20 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72e97fa640 1 -- 172.21.15.110:0/4241993066 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72dc007070 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ec10fb50 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72e97fa640 1 -- 172.21.15.110:0/4241993066 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f72dc01a070 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72e97fa640 1 -- 172.21.15.110:0/4241993066 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f72dc01cc40 con 0x7f72ec1158c0 2024-04-27T03:17:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.025+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72ec062e50 con 0x7f72ec1158c0 2024-04-27T03:17:30.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.033+0000 7f72e97fa640 1 -- 172.21.15.110:0/4241993066 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f72dc01c440 con 0x7f72ec1158c0 2024-04-27T03:17:30.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.097+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d73614e8-5d01-4f79-8b12-0e08dabad6b1", "id": 3} v 0) -- 0x7f72ec112a90 con 0x7f72ec1158c0 2024-04-27T03:17:30.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72e97fa640 1 -- 172.21.15.110:0/4241993066 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d73614e8-5d01-4f79-8b12-0e08dabad6b1", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f72dc01c700 con 0x7f72ec1158c0 2024-04-27T03:17:30.108 INFO:teuthology.orchestra.run.smithi110.stdout:3 2024-04-27T03:17:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1158c0 msgr2=0x7f72ec10ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72f2328640 1 --2- 172.21.15.110:0/4241993066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1158c0 0x7f72ec10ee40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00c480 tx=0x7f72dc00c940 comp rx=0 tx=0).stop 2024-04-27T03:17:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 shutdown_connections 2024-04-27T03:17:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72f2328640 1 --2- 172.21.15.110:0/4241993066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f72ec1158c0 0x7f72ec10ee40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 >> 172.21.15.110:0/4241993066 conn(0x7f72ec0700f0 msgr2=0x7f72ec11a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 shutdown_connections 2024-04-27T03:17:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:30.101+0000 7f72f2328640 1 -- 172.21.15.110:0/4241993066 wait complete. 2024-04-27T03:17:30.124 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-27T03:17:30.124 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-27T03:17:30.127 INFO:tasks.ceph.osd.0:Started 2024-04-27T03:17:30.127 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-27T03:17:30.127 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-27T03:17:30.129 INFO:tasks.ceph.osd.1:Started 2024-04-27T03:17:30.130 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-27T03:17:30.130 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-27T03:17:30.132 INFO:tasks.ceph.osd.2:Started 2024-04-27T03:17:30.132 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-27T03:17:30.132 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-27T03:17:30.134 INFO:tasks.ceph.osd.3:Started 2024-04-27T03:17:30.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T03:17:30.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f7159ccc640 1 Processor -- start 2024-04-27T03:17:30.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f7159ccc640 1 -- start start 2024-04-27T03:17:30.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f7159ccc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7154129290 0x7f7154129670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:30.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f7159ccc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715409bb10 con 0x7f715412c340 2024-04-27T03:17:30.253 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f7159ccc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f715409c020 con 0x7f7154129290 2024-04-27T03:17:30.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f71537fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7154129290 0x7f7154129670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:30.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f7152ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f715412c340 0x7f715412c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48578/0 (socket says 172.21.15.77:48578) 2024-04-27T03:17:30.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.242+0000 7f7152ffd640 1 -- 172.21.15.77:0/218358237 learned_addr learned my addr 172.21.15.77:0/218358237 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:30.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f71537fe640 1 -- 172.21.15.77:0/218358237 >> v1:172.21.15.77:6789/0 conn(0x7f715412c340 legacy=0x7f715412c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f71537fe640 1 -- 172.21.15.77:0/218358237 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715409b410 con 0x7f7154129290 2024-04-27T03:17:30.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f71537fe640 1 --2- 172.21.15.77:0/218358237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154129670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7144009740 tx=0x7f714401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c213c962b9c8f589 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:30.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f71527fc640 1 -- 172.21.15.77:0/218358237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71440b0070 con 0x7f7154129290 2024-04-27T03:17:30.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f71527fc640 1 -- 172.21.15.77:0/218358237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71440ab530 con 0x7f7154129290 2024-04-27T03:17:30.256 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f71527fc640 1 -- 172.21.15.77:0/218358237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71440ab830 con 0x7f7154129290 2024-04-27T03:17:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f7159ccc640 1 -- 172.21.15.77:0/218358237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 msgr2=0x7f7154129670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f7159ccc640 1 --2- 172.21.15.77:0/218358237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154129670 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7144009740 tx=0x7f714401c370 comp rx=0 tx=0).stop 2024-04-27T03:17:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f7159ccc640 1 -- 172.21.15.77:0/218358237 shutdown_connections 2024-04-27T03:17:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f7159ccc640 1 --2- 172.21.15.77:0/218358237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154129670 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f7159ccc640 1 -- 172.21.15.77:0/218358237 >> 172.21.15.77:0/218358237 conn(0x7f715409c620 msgr2=0x7f715409ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f7159ccc640 1 -- 172.21.15.77:0/218358237 shutdown_connections 2024-04-27T03:17:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.246+0000 7f7159ccc640 1 -- 172.21.15.77:0/218358237 wait complete. 2024-04-27T03:17:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f7159ccc640 1 Processor -- start 2024-04-27T03:17:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f7159ccc640 1 -- start start 2024-04-27T03:17:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f7159ccc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154116c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f7159ccc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7154129d30 con 0x7f7154129290 2024-04-27T03:17:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f71537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154116c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f71537fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154116c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42428/0 (socket says 172.21.15.77:42428) 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f71537fe640 1 -- 172.21.15.77:0/1346788449 learned_addr learned my addr 172.21.15.77:0/1346788449 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f71537fe640 1 -- 172.21.15.77:0/1346788449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715412bb40 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f71537fe640 1 --2- 172.21.15.77:0/1346788449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154116c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7144019fa0 tx=0x7f714401abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f713bfff640 1 -- 172.21.15.77:0/1346788449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71440b0050 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f713bfff640 1 -- 172.21.15.77:0/1346788449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71440b1040 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f713bfff640 1 -- 172.21.15.77:0/1346788449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71440b5930 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7154119e30 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71541171c0 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f713bfff640 1 -- 172.21.15.77:0/1346788449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f71440be030 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f713bfff640 1 -- 172.21.15.77:0/1346788449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f71440bc070 con 0x7f7154129290 2024-04-27T03:17:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.250+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7154062e50 con 0x7f7154129290 2024-04-27T03:17:30.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.254+0000 7f713bfff640 1 -- 172.21.15.77:0/1346788449 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f71440c5020 con 0x7f7154129290 2024-04-27T03:17:30.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7154118590 con 0x7f7154129290 2024-04-27T03:17:30.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f713bfff640 1 -- 172.21.15.77:0/1346788449 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) ==== 74+0+4153 (secure 0 0 0) 0x7f7144015ff0 con 0x7f7154129290 2024-04-27T03:17:30.319 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:30.319 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":6,"fsid":"3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3","created":"2024-04-27T03:17:28.925654+0000","modified":"2024-04-27T03:17:30.101677+0000","last_up_change":"0.000000","last_in_change":"2024-04-27T03:17:30.101677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"74a30006-c87c-4e0e-9e94-9020affcae97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1bd2dbfb-57cb-4673-9684-d9427036427d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d73614e8-5d01-4f79-8b12-0e08dabad6b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T03:17:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 msgr2=0x7f7154116c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 --2- 172.21.15.77:0/1346788449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154116c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7144019fa0 tx=0x7f714401abb0 comp rx=0 tx=0).stop 2024-04-27T03:17:30.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 shutdown_connections 2024-04-27T03:17:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 --2- 172.21.15.77:0/1346788449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7154129290 0x7f7154116c80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 >> 172.21.15.77:0/1346788449 conn(0x7f715409c620 msgr2=0x7f7154099190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 shutdown_connections 2024-04-27T03:17:30.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:30.310+0000 7f7159ccc640 1 -- 172.21.15.77:0/1346788449 wait complete. 2024-04-27T03:17:30.336 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-27T03:17:30.336 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-27T03:17:30.487 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.477+0000 7f0cbb0b7180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-27T03:17:30.541 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.533+0000 7f0cbb0b7180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-27T03:17:30.593 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.585+0000 7f0cbb0b7180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-27T03:17:30.669 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.661+0000 7f0cbb0b7180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-27T03:17:30.720 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.713+0000 7f0cbb0b7180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-27T03:17:30.733 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-27T03:17:30.725+0000 7f60f58367c0 -1 Falling back to public interface 2024-04-27T03:17:30.739 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-27T03:17:30.730+0000 7fb5600377c0 -1 Falling back to public interface 2024-04-27T03:17:30.739 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-27T03:17:30.740 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-04-27T03:17:30.733+0000 7f451394c7c0 -1 Falling back to public interface 2024-04-27T03:17:30.764 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-04-27T03:17:30.757+0000 7f03b4daa7c0 -1 Falling back to public interface 2024-04-27T03:17:30.798 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.789+0000 7f0cbb0b7180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-27T03:17:30.850 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.841+0000 7f0cbb0b7180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-27T03:17:30.856 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff372450640 1 Processor -- start 2024-04-27T03:17:30.856 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff372450640 1 -- start start 2024-04-27T03:17:30.857 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff372450640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff36c131610 0x7ff36c133a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:30.857 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff372450640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff36c06ff10 con 0x7ff36c06dc70 2024-04-27T03:17:30.857 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff372450640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ff36c072880 con 0x7ff36c131610 2024-04-27T03:17:30.857 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff36b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff36c131610 0x7ff36c133a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:30.857 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff36bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff36c06dc70 0x7ff36c06e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:48588/0 (socket says 172.21.15.77:48588) 2024-04-27T03:17:30.857 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.846+0000 7ff36bfff640 1 -- 172.21.15.77:0/4027973735 learned_addr learned my addr 172.21.15.77:0/4027973735 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:30.857 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff36b7fe640 1 -- 172.21.15.77:0/4027973735 >> v1:172.21.15.77:6789/0 conn(0x7ff36c06dc70 legacy=0x7ff36c06e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.858 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff36b7fe640 1 -- 172.21.15.77:0/4027973735 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36c06e760 con 0x7ff36c131610 2024-04-27T03:17:30.860 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff36b7fe640 1 --2- 172.21.15.77:0/4027973735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c131610 0x7ff36c133a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff354009740 tx=0x7ff35401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867a63ba006b5740 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:30.860 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff36affd640 1 -- 172.21.15.77:0/4027973735 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3540b0070 con 0x7ff36c131610 2024-04-27T03:17:30.860 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff36affd640 1 -- 172.21.15.77:0/4027973735 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3540ab530 con 0x7ff36c131610 2024-04-27T03:17:30.860 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff36affd640 1 -- 172.21.15.77:0/4027973735 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3540ab830 con 0x7ff36c131610 2024-04-27T03:17:30.861 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff372450640 1 -- 172.21.15.77:0/4027973735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c131610 msgr2=0x7ff36c133a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.861 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff372450640 1 --2- 172.21.15.77:0/4027973735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c131610 0x7ff36c133a00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff354009740 tx=0x7ff35401c370 comp rx=0 tx=0).stop 2024-04-27T03:17:30.861 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff372450640 1 -- 172.21.15.77:0/4027973735 shutdown_connections 2024-04-27T03:17:30.861 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff372450640 1 --2- 172.21.15.77:0/4027973735 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c131610 0x7ff36c133a00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:30.861 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff372450640 1 -- 172.21.15.77:0/4027973735 >> 172.21.15.77:0/4027973735 conn(0x7ff36c09ec60 msgr2=0x7ff36c09f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:30.861 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff372450640 1 -- 172.21.15.77:0/4027973735 shutdown_connections 2024-04-27T03:17:30.861 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.850+0000 7ff372450640 1 -- 172.21.15.77:0/4027973735 wait complete. 2024-04-27T03:17:30.862 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff372450640 1 Processor -- start 2024-04-27T03:17:30.862 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff372450640 1 -- start start 2024-04-27T03:17:30.862 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff372450640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c06dc70 0x7ff36c12c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:30.863 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff372450640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff36c136af0 con 0x7ff36c06dc70 2024-04-27T03:17:30.863 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff36bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c06dc70 0x7ff36c12c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:30.863 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff36bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c06dc70 0x7ff36c12c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42452/0 (socket says 172.21.15.77:42452) 2024-04-27T03:17:30.863 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff36bfff640 1 -- 172.21.15.77:0/1060269710 learned_addr learned my addr 172.21.15.77:0/1060269710 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:30.863 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff36bfff640 1 -- 172.21.15.77:0/1060269710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36c098560 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff36bfff640 1 --2- 172.21.15.77:0/1060269710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c06dc70 0x7ff36c12c100 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff36400cb20 tx=0x7ff36400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff368ff9640 1 -- 172.21.15.77:0/1060269710 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff364017070 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff368ff9640 1 -- 172.21.15.77:0/1060269710 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff36400abf0 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff368ff9640 1 -- 172.21.15.77:0/1060269710 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff364012400 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff36c0976b0 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff36c12c640 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff368ff9640 1 -- 172.21.15.77:0/1060269710 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff36401a070 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff368ff9640 1 -- 172.21.15.77:0/1060269710 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff36401ccb0 con 0x7ff36c06dc70 2024-04-27T03:17:30.864 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.854+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36c0738a0 con 0x7ff36c06dc70 2024-04-27T03:17:30.866 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.858+0000 7ff368ff9640 1 -- 172.21.15.77:0/1060269710 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff36401c7a0 con 0x7ff36c06dc70 2024-04-27T03:17:30.923 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff36c062e50 con 0x7ff36c06dc70 2024-04-27T03:17:30.924 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff368ff9640 1 -- 172.21.15.77:0/1060269710 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) ==== 74+0+4153 (secure 0 0 0) 0x7ff36401c980 con 0x7ff36c06dc70 2024-04-27T03:17:30.924 INFO:teuthology.misc.health.smithi077.stdout: 2024-04-27T03:17:30.924 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":6,"fsid":"3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3","created":"2024-04-27T03:17:28.925654+0000","modified":"2024-04-27T03:17:30.101677+0000","last_up_change":"0.000000","last_in_change":"2024-04-27T03:17:30.101677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"74a30006-c87c-4e0e-9e94-9020affcae97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1bd2dbfb-57cb-4673-9684-d9427036427d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d73614e8-5d01-4f79-8b12-0e08dabad6b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T03:17:30.925 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c06dc70 msgr2=0x7ff36c12c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:30.925 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 --2- 172.21.15.77:0/1060269710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c06dc70 0x7ff36c12c100 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff36400cb20 tx=0x7ff36400a060 comp rx=0 tx=0).stop 2024-04-27T03:17:30.925 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 shutdown_connections 2024-04-27T03:17:30.925 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 --2- 172.21.15.77:0/1060269710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff36c06dc70 0x7ff36c12c100 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:30.925 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 >> 172.21.15.77:0/1060269710 conn(0x7ff36c09ec60 msgr2=0x7ff36c09b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:30.926 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 shutdown_connections 2024-04-27T03:17:30.926 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:30.914+0000 7ff372450640 1 -- 172.21.15.77:0/1060269710 wait complete. 2024-04-27T03:17:30.941 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-27T03:17:30.998 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:30.989+0000 7f0cbb0b7180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-27T03:17:31.059 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:31.049+0000 7f0cbb0b7180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-27T03:17:31.727 INFO:tasks.ceph.mgr.x.smithi110.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-27T03:17:31.727 INFO:tasks.ceph.mgr.x.smithi110.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-27T03:17:31.727 INFO:tasks.ceph.mgr.x.smithi110.stderr: from numpy import show_config as show_numpy_config 2024-04-27T03:17:31.814 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:31.805+0000 7f0cbb0b7180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-27T03:17:31.990 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:31.981+0000 7f0cbb0b7180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-27T03:17:32.141 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:32.133+0000 7f0cbb0b7180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-27T03:17:32.204 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:32.197+0000 7f0cbb0b7180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-27T03:17:32.337 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:32.329+0000 7f0cbb0b7180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-27T03:17:32.936 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:32.929+0000 7f0cbb0b7180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-27T03:17:33.010 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:33.001+0000 7f0cbb0b7180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-27T03:17:33.190 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:33.181+0000 7f0cbb0b7180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-27T03:17:33.262 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:33.253+0000 7f0cbb0b7180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-27T03:17:33.552 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:33.545+0000 7f0cbb0b7180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-27T03:17:33.613 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:33.605+0000 7f0cbb0b7180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-27T03:17:34.455 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-27T03:17:34.446+0000 7fb5600377c0 -1 osd.0 0 log_to_monitors true 2024-04-27T03:17:34.603 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-27T03:17:34.593+0000 7f60f58367c0 -1 osd.1 0 log_to_monitors true 2024-04-27T03:17:34.623 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-04-27T03:17:34.613+0000 7f451394c7c0 -1 osd.2 0 log_to_monitors true 2024-04-27T03:17:34.646 INFO:tasks.ceph.mgr.x.smithi110.stderr:2024-04-27T03:17:34.637+0000 7f0c62401640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-27T03:17:34.651 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-04-27T03:17:34.641+0000 7f03b4daa7c0 -1 osd.3 0 log_to_monitors true 2024-04-27T03:17:36.619 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-27T03:17:36.610+0000 7fb55b7bd640 -1 osd.0 0 waiting for initial osdmap 2024-04-27T03:17:36.646 INFO:tasks.ceph.osd.0.smithi077.stderr:2024-04-27T03:17:36.638+0000 7fb553e04640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:17:36.884 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-04-27T03:17:36.877+0000 7f450f0d2640 -1 osd.2 0 waiting for initial osdmap 2024-04-27T03:17:36.901 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-04-27T03:17:36.893+0000 7f03b1d66640 -1 osd.3 0 waiting for initial osdmap 2024-04-27T03:17:36.912 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-04-27T03:17:36.905+0000 7f4507719640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:17:36.924 INFO:tasks.ceph.osd.3.smithi110.stderr:2024-04-27T03:17:36.917+0000 7f03a8b77640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:17:36.927 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-27T03:17:36.917+0000 7f60f27f2640 -1 osd.1 0 waiting for initial osdmap 2024-04-27T03:17:36.956 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-04-27T03:17:36.949+0000 7f60e9603640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-27T03:17:37.345 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-27T03:17:37.465 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.454+0000 7fc9f8e65640 1 Processor -- start 2024-04-27T03:17:37.465 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.454+0000 7fc9f8e65640 1 -- start start 2024-04-27T03:17:37.466 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f8e65640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc9f4096380 0x7fc9f4096760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:37.466 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f8e65640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f409c5b0 con 0x7fc9f412c060 2024-04-27T03:17:37.466 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f8e65640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f4072470 con 0x7fc9f4096380 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f259c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc9f4096380 0x7fc9f4096760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f1d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc9f412c060 0x7fc9f412c480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39208/0 (socket says 172.21.15.77:39208) 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f1d9b640 1 -- 172.21.15.77:0/750450914 learned_addr learned my addr 172.21.15.77:0/750450914 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f259c640 1 --2- 172.21.15.77:0/750450914 >> v2:172.21.15.77:3300/0 conn(0x7fc9f4096380 0x7fc9f4096760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54848/0 (socket says 172.21.15.77:54848) 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f159a640 1 -- 172.21.15.77:0/750450914 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200989387 0 0) 0x7fc9f409c5b0 con 0x7fc9f412c060 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f159a640 1 -- 172.21.15.77:0/750450914 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e0003530 con 0x7fc9f412c060 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f259c640 1 -- 172.21.15.77:0/750450914 >> v1:172.21.15.77:6789/0 conn(0x7fc9f412c060 legacy=0x7fc9f412c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:37.467 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f259c640 1 -- 172.21.15.77:0/750450914 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f406ed10 con 0x7fc9f4096380 2024-04-27T03:17:37.469 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f259c640 1 --2- 172.21.15.77:0/750450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4096380 0x7fc9f4096760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009740 tx=0x7fc9dc01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5045c6c2ea65d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:37.469 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.458+0000 7fc9f159a640 1 -- 172.21.15.77:0/750450914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc0b0070 con 0x7fc9f4096380 2024-04-27T03:17:37.469 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f159a640 1 -- 172.21.15.77:0/750450914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9dc0ab530 con 0x7fc9f4096380 2024-04-27T03:17:37.469 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f159a640 1 -- 172.21.15.77:0/750450914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc0ab830 con 0x7fc9f4096380 2024-04-27T03:17:37.470 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/750450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4096380 msgr2=0x7fc9f4096760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:37.470 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 --2- 172.21.15.77:0/750450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4096380 0x7fc9f4096760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc009740 tx=0x7fc9dc01c370 comp rx=0 tx=0).stop 2024-04-27T03:17:37.470 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/750450914 shutdown_connections 2024-04-27T03:17:37.470 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 --2- 172.21.15.77:0/750450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f4096380 0x7fc9f4096760 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:37.470 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/750450914 >> 172.21.15.77:0/750450914 conn(0x7fc9f409a850 msgr2=0x7fc9f409ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:37.470 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/750450914 shutdown_connections 2024-04-27T03:17:37.470 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/750450914 wait complete. 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 Processor -- start 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- start start 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f412c060 0x7fc9f406f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f40715b0 con 0x7fc9f412c060 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f412c060 0x7fc9f406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f412c060 0x7fc9f406f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54864/0 (socket says 172.21.15.77:54864) 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f259c640 1 -- 172.21.15.77:0/3001561971 learned_addr learned my addr 172.21.15.77:0/3001561971 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f259c640 1 -- 172.21.15.77:0/3001561971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f4129840 con 0x7fc9f412c060 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f259c640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f412c060 0x7fc9f406f270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0ae040 tx=0x7fc9dc019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:37.472 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9eaffd640 1 -- 172.21.15.77:0/3001561971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc0b8020 con 0x7fc9f412c060 2024-04-27T03:17:37.473 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9eaffd640 1 -- 172.21.15.77:0/3001561971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9dc0b4410 con 0x7fc9f412c060 2024-04-27T03:17:37.473 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f4070be0 con 0x7fc9f412c060 2024-04-27T03:17:37.473 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9eaffd640 1 -- 172.21.15.77:0/3001561971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9dc0b0040 con 0x7fc9f412c060 2024-04-27T03:17:37.473 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f4093260 con 0x7fc9f412c060 2024-04-27T03:17:37.473 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9eaffd640 1 -- 172.21.15.77:0/3001561971 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7fc9dc0bb020 con 0x7fc9f412c060 2024-04-27T03:17:37.473 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f40738a0 con 0x7fc9f412c060 2024-04-27T03:17:37.474 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9eaffd640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc9d4035cd0 0x7fc9d4038190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:37.474 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.462+0000 7fc9eaffd640 1 -- 172.21.15.77:0/3001561971 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1623+0+0 (secure 0 0 0) 0x7fc9dc0a4e00 con 0x7fc9f412c060 2024-04-27T03:17:37.474 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.466+0000 7fc9f1d9b640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc9d4035cd0 0x7fc9d4038190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:37.474 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.466+0000 7fc9f1d9b640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc9d4035cd0 0x7fc9d4038190 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0020b0 tx=0x7fc9ec0016e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:37.478 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.470+0000 7fc9eaffd640 1 -- 172.21.15.77:0/3001561971 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc9dc0c2030 con 0x7fc9f412c060 2024-04-27T03:17:37.639 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.630+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9f4062e50 con 0x7fc9f412c060 2024-04-27T03:17:37.640 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.630+0000 7fc9eaffd640 1 -- 172.21.15.77:0/3001561971 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) ==== 74+0+6104 (secure 0 0 0) 0x7fc9dc015ff0 con 0x7fc9f412c060 2024-04-27T03:17:37.640 INFO:teuthology.misc.health.smithi077.stdout: 2024-04-27T03:17:37.640 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":9,"fsid":"3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3","created":"2024-04-27T03:17:28.925654+0000","modified":"2024-04-27T03:17:37.618506+0000","last_up_change":"2024-04-27T03:17:37.618506+0000","last_in_change":"2024-04-27T03:17:30.101677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6801","nonce":3496324319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6803","nonce":3496324319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6807","nonce":3496324319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6805","nonce":3496324319}]},"public_addr":"172.21.15.77:6801/3496324319","cluster_addr":"172.21.15.77:6803/3496324319","heartbeat_back_addr":"172.21.15.77:6807/3496324319","heartbeat_front_addr":"172.21.15.77:6805/3496324319","state":["exists","up"]},{"osd":1,"uuid":"74a30006-c87c-4e0e-9e94-9020affcae97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6801","nonce":1860462708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6803","nonce":1860462708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6807","nonce":1860462708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6805","nonce":1860462708}]},"public_addr":"172.21.15.110:6801/1860462708","cluster_addr":"172.21.15.110:6803/1860462708","heartbeat_back_addr":"172.21.15.110:6807/1860462708","heartbeat_front_addr":"172.21.15.110:6805/1860462708","state":["exists","up"]},{"osd":2,"uuid":"1bd2dbfb-57cb-4673-9684-d9427036427d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6809","nonce":3575017494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6811","nonce":3575017494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6815","nonce":3575017494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6813","nonce":3575017494}]},"public_addr":"172.21.15.110:6809/3575017494","cluster_addr":"172.21.15.110:6811/3575017494","heartbeat_back_addr":"172.21.15.110:6815/3575017494","heartbeat_front_addr":"172.21.15.110:6813/3575017494","state":["exists","up"]},{"osd":3,"uuid":"d73614e8-5d01-4f79-8b12-0e08dabad6b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6817","nonce":3916647200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6819","nonce":3916647200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6823","nonce":3916647200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6821","nonce":3916647200}]},"public_addr":"172.21.15.110:6817/3916647200","cluster_addr":"172.21.15.110:6819/3916647200","heartbeat_back_addr":"172.21.15.110:6823/3916647200","heartbeat_front_addr":"172.21.15.110:6821/3916647200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc9d4035cd0 msgr2=0x7fc9d4038190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc9d4035cd0 0x7fc9d4038190 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0020b0 tx=0x7fc9ec0016e0 comp rx=0 tx=0).stop 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f412c060 msgr2=0x7fc9f406f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f412c060 0x7fc9f406f270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0ae040 tx=0x7fc9dc019110 comp rx=0 tx=0).stop 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 shutdown_connections 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc9d4035cd0 0x7fc9d4038190 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 --2- 172.21.15.77:0/3001561971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9f412c060 0x7fc9f406f270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 >> 172.21.15.77:0/3001561971 conn(0x7fc9f409a850 msgr2=0x7fc9f412a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 shutdown_connections 2024-04-27T03:17:37.643 INFO:teuthology.misc.health.smithi077.stderr:2024-04-27T03:17:37.634+0000 7fc9f8e65640 1 -- 172.21.15.77:0/3001561971 wait complete. 2024-04-27T03:17:37.658 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-27T03:17:37.658 INFO:tasks.ceph:Creating RBD pool 2024-04-27T03:17:37.659 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-27T03:17:37.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.766+0000 7f994297b640 1 Processor -- start 2024-04-27T03:17:37.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.766+0000 7f994297b640 1 -- start start 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f993c071140 0x7f993c1315d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f993c09baa0 con 0x7f993c06d4d0 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f993c070ad0 con 0x7f993c071140 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f993c071140 0x7f993c1315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f993c06d4d0 0x7f993c06d8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39220/0 (socket says 172.21.15.77:39220) 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993bfff640 1 -- 172.21.15.77:0/2065989022 learned_addr learned my addr 172.21.15.77:0/2065989022 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993b7fe640 1 --2- 172.21.15.77:0/2065989022 >> v2:172.21.15.77:3300/0 conn(0x7f993c071140 0x7f993c1315d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54866/0 (socket says 172.21.15.77:54866) 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993b7fe640 1 -- 172.21.15.77:0/2065989022 >> v1:172.21.15.77:6789/0 conn(0x7f993c06d4d0 legacy=0x7f993c06d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:37.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993b7fe640 1 -- 172.21.15.77:0/2065989022 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c072000 con 0x7f993c071140 2024-04-27T03:17:37.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993b7fe640 1 --2- 172.21.15.77:0/2065989022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c071140 0x7f993c1315d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9924009740 tx=0x7f992401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e5923d2637f84 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:37.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993affd640 1 -- 172.21.15.77:0/2065989022 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99240b0070 con 0x7f993c071140 2024-04-27T03:17:37.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993affd640 1 -- 172.21.15.77:0/2065989022 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99240ab530 con 0x7f993c071140 2024-04-27T03:17:37.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f993affd640 1 -- 172.21.15.77:0/2065989022 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99240ab830 con 0x7f993c071140 2024-04-27T03:17:37.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 -- 172.21.15.77:0/2065989022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c071140 msgr2=0x7f993c1315d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:37.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 --2- 172.21.15.77:0/2065989022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c071140 0x7f993c1315d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9924009740 tx=0x7f992401c310 comp rx=0 tx=0).stop 2024-04-27T03:17:37.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 -- 172.21.15.77:0/2065989022 shutdown_connections 2024-04-27T03:17:37.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 --2- 172.21.15.77:0/2065989022 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c071140 0x7f993c1315d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:37.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 -- 172.21.15.77:0/2065989022 >> 172.21.15.77:0/2065989022 conn(0x7f993c0993d0 msgr2=0x7f993c0997e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:37.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 -- 172.21.15.77:0/2065989022 shutdown_connections 2024-04-27T03:17:37.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.770+0000 7f994297b640 1 -- 172.21.15.77:0/2065989022 wait complete. 2024-04-27T03:17:37.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f994297b640 1 Processor -- start 2024-04-27T03:17:37.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f994297b640 1 -- start start 2024-04-27T03:17:37.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f994297b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c06d4d0 0x7f993c122d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:37.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f994297b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f993c134720 con 0x7f993c06d4d0 2024-04-27T03:17:37.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f993bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c06d4d0 0x7f993c122d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f993bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c06d4d0 0x7f993c122d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54868/0 (socket says 172.21.15.77:54868) 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f993bfff640 1 -- 172.21.15.77:0/757837592 learned_addr learned my addr 172.21.15.77:0/757837592 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f993bfff640 1 -- 172.21.15.77:0/757837592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c126030 con 0x7f993c06d4d0 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f993bfff640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c06d4d0 0x7f993c122d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f993400cb20 tx=0x7f993400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f9938ff9640 1 -- 172.21.15.77:0/757837592 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9934017070 con 0x7f993c06d4d0 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f9938ff9640 1 -- 172.21.15.77:0/757837592 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f993400abf0 con 0x7f993c06d4d0 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993c124fc0 con 0x7f993c06d4d0 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f9938ff9640 1 -- 172.21.15.77:0/757837592 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9934012400 con 0x7f993c06d4d0 2024-04-27T03:17:37.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f993c123420 con 0x7f993c06d4d0 2024-04-27T03:17:37.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f993c062e50 con 0x7f993c06d4d0 2024-04-27T03:17:37.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f9938ff9640 1 -- 172.21.15.77:0/757837592 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f993401a070 con 0x7f993c06d4d0 2024-04-27T03:17:37.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f9938ff9640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f991c035d20 0x7f991c0381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:37.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f9938ff9640 1 -- 172.21.15.77:0/757837592 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 2743+0+0 (secure 0 0 0) 0x7f9934047fd0 con 0x7f993c06d4d0 2024-04-27T03:17:37.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.774+0000 7f993b7fe640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f991c035d20 0x7f991c0381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:37.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.778+0000 7f993b7fe640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f991c035d20 0x7f991c0381e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f99240ad000 tx=0x7f99240a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:37.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.782+0000 7f9938ff9640 1 -- 172.21.15.77:0/757837592 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f99340113a0 con 0x7f993c06d4d0 2024-04-27T03:17:37.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:37.946+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f993c124350 con 0x7f993c06d4d0 2024-04-27T03:17:38.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.614+0000 7f9938ff9640 1 -- 172.21.15.77:0/757837592 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v10) ==== 109+0+0 (secure 0 0 0) 0x7f9934011580 con 0x7f993c06d4d0 2024-04-27T03:17:38.625 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'rbd' created 2024-04-27T03:17:38.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f991c035d20 msgr2=0x7f991c0381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:38.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f991c035d20 0x7f991c0381e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f99240ad000 tx=0x7f99240a6000 comp rx=0 tx=0).stop 2024-04-27T03:17:38.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c06d4d0 msgr2=0x7f993c122d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:38.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c06d4d0 0x7f993c122d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f993400cb20 tx=0x7f993400a060 comp rx=0 tx=0).stop 2024-04-27T03:17:38.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 shutdown_connections 2024-04-27T03:17:38.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f991c035d20 0x7f991c0381e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:38.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 --2- 172.21.15.77:0/757837592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f993c06d4d0 0x7f993c122d10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:38.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 >> 172.21.15.77:0/757837592 conn(0x7f993c0993d0 msgr2=0x7f993c12bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:38.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 shutdown_connections 2024-04-27T03:17:38.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.622+0000 7f994297b640 1 -- 172.21.15.77:0/757837592 wait complete. 2024-04-27T03:17:38.652 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-27T03:17:38.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776f7ac640 1 Processor -- start 2024-04-27T03:17:38.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776f7ac640 1 -- start start 2024-04-27T03:17:38.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776f7ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7768072090 0x7f776812c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:38.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776f7ac640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7768113000 con 0x7f776812c9e0 2024-04-27T03:17:38.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776f7ac640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f776806f8a0 con 0x7f7768072090 2024-04-27T03:17:38.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776d548640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7768072090 0x7f776812c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:38.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776d548640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7768072090 0x7f776812c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54878/0 (socket says 172.21.15.77:54878) 2024-04-27T03:17:38.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776d548640 1 -- 172.21.15.77:0/1458335240 learned_addr learned my addr 172.21.15.77:0/1458335240 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:38.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776d548640 1 -- 172.21.15.77:0/1458335240 >> v1:172.21.15.77:6789/0 conn(0x7f776812c9e0 legacy=0x7f776806e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:38.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.770+0000 7f776d548640 1 -- 172.21.15.77:0/1458335240 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776812c050 con 0x7f7768072090 2024-04-27T03:17:38.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776d548640 1 --2- 172.21.15.77:0/1458335240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f776812c4a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f77640097f0 tx=0x7f776401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5716af0dca7508 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:38.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f775ffff640 1 -- 172.21.15.77:0/1458335240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77640b5070 con 0x7f7768072090 2024-04-27T03:17:38.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f775ffff640 1 -- 172.21.15.77:0/1458335240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77640a88e0 con 0x7f7768072090 2024-04-27T03:17:38.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f775ffff640 1 -- 172.21.15.77:0/1458335240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77640a8be0 con 0x7f7768072090 2024-04-27T03:17:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 -- 172.21.15.77:0/1458335240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 msgr2=0x7f776812c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 --2- 172.21.15.77:0/1458335240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f776812c4a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f77640097f0 tx=0x7f776401ecb0 comp rx=0 tx=0).stop 2024-04-27T03:17:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 -- 172.21.15.77:0/1458335240 shutdown_connections 2024-04-27T03:17:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 --2- 172.21.15.77:0/1458335240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f776812c4a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 -- 172.21.15.77:0/1458335240 >> 172.21.15.77:0/1458335240 conn(0x7f77680992d0 msgr2=0x7f77680996e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 -- 172.21.15.77:0/1458335240 shutdown_connections 2024-04-27T03:17:38.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 -- 172.21.15.77:0/1458335240 wait complete. 2024-04-27T03:17:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 Processor -- start 2024-04-27T03:17:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 -- start start 2024-04-27T03:17:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f77681c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.774+0000 7f776f7ac640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7768113760 con 0x7f7768072090 2024-04-27T03:17:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776d548640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f77681c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776d548640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f77681c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54884/0 (socket says 172.21.15.77:54884) 2024-04-27T03:17:38.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776d548640 1 -- 172.21.15.77:0/714076738 learned_addr learned my addr 172.21.15.77:0/714076738 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:38.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776d548640 1 -- 172.21.15.77:0/714076738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7768119640 con 0x7f7768072090 2024-04-27T03:17:38.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776d548640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f77681c1120 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f776401b8c0 tx=0x7f776401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:38.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f775dffb640 1 -- 172.21.15.77:0/714076738 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77640c0070 con 0x7f7768072090 2024-04-27T03:17:38.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f775dffb640 1 -- 172.21.15.77:0/714076738 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77640b0940 con 0x7f7768072090 2024-04-27T03:17:38.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77681c1660 con 0x7f7768072090 2024-04-27T03:17:38.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f775dffb640 1 -- 172.21.15.77:0/714076738 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77640b5070 con 0x7f7768072090 2024-04-27T03:17:38.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77681c1b20 con 0x7f7768072090 2024-04-27T03:17:38.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7768062e50 con 0x7f7768072090 2024-04-27T03:17:38.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f775dffb640 1 -- 172.21.15.77:0/714076738 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f77640c3070 con 0x7f7768072090 2024-04-27T03:17:38.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f775dffb640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f774c035d20 0x7f774c0381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:38.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f775dffb640 1 -- 172.21.15.77:0/714076738 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 3460+0+0 (secure 0 0 0) 0x7f77640e5eb0 con 0x7f7768072090 2024-04-27T03:17:38.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776cd47640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f774c035d20 0x7f774c0381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:38.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.778+0000 7f776cd47640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f774c035d20 0x7f774c0381e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7758000f70 tx=0x7f7758007a40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.782+0000 7f775dffb640 1 -- 172.21.15.77:0/714076738 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f77640ca020 con 0x7f7768072090 2024-04-27T03:17:38.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:38.946+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f7768118510 con 0x7f7768072090 2024-04-27T03:17:39.626 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.618+0000 7f775dffb640 1 -- 172.21.15.77:0/714076738 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v11) ==== 173+0+0 (secure 0 0 0) 0x7f7764018810 con 0x7f7768072090 2024-04-27T03:17:39.626 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-27T03:17:39.628 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.618+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f774c035d20 msgr2=0x7f774c0381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:39.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.618+0000 7f776f7ac640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f774c035d20 0x7f774c0381e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7758000f70 tx=0x7f7758007a40 comp rx=0 tx=0).stop 2024-04-27T03:17:39.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.618+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 msgr2=0x7f77681c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:39.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.618+0000 7f776f7ac640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f77681c1120 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f776401b8c0 tx=0x7f776401ba50 comp rx=0 tx=0).stop 2024-04-27T03:17:39.629 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.622+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 shutdown_connections 2024-04-27T03:17:39.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.622+0000 7f776f7ac640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f774c035d20 0x7f774c0381e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:39.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.622+0000 7f776f7ac640 1 --2- 172.21.15.77:0/714076738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7768072090 0x7f77681c1120 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:39.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.622+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 >> 172.21.15.77:0/714076738 conn(0x7f77680992d0 msgr2=0x7f776809b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:39.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.622+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 shutdown_connections 2024-04-27T03:17:39.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.622+0000 7f776f7ac640 1 -- 172.21.15.77:0/714076738 wait complete. 2024-04-27T03:17:39.653 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-27T03:17:39.653 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-27T03:17:39.653 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-27T03:17:39.654 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-27T03:17:39.654 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-27T03:17:39.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa14d411640 1 Processor -- start 2024-04-27T03:17:39.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa14d411640 1 -- start start 2024-04-27T03:17:39.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa14d411640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa14806d4d0 0x7fa14806d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:39.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa14d411640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa148098e10 con 0x7fa14806ddf0 2024-04-27T03:17:39.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa14d411640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fa148096d90 con 0x7fa14806d4d0 2024-04-27T03:17:39.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa146ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa14806d4d0 0x7fa14806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:39.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa1467fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa14806ddf0 0x7fa148132920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39230/0 (socket says 172.21.15.77:39230) 2024-04-27T03:17:39.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.766+0000 7fa1467fc640 1 -- 172.21.15.77:0/2071412183 learned_addr learned my addr 172.21.15.77:0/2071412183 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:39.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa146ffd640 1 -- 172.21.15.77:0/2071412183 >> v1:172.21.15.77:6789/0 conn(0x7fa14806ddf0 legacy=0x7fa148132920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:39.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa146ffd640 1 -- 172.21.15.77:0/2071412183 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa148098610 con 0x7fa14806d4d0 2024-04-27T03:17:39.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa146ffd640 1 --2- 172.21.15.77:0/2071412183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa14806d8b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009740 tx=0x7fa13c01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f701e1eb5de9ee server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:39.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa145ffb640 1 -- 172.21.15.77:0/2071412183 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa13c0b0070 con 0x7fa14806d4d0 2024-04-27T03:17:39.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa145ffb640 1 -- 172.21.15.77:0/2071412183 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa13c0ab530 con 0x7fa14806d4d0 2024-04-27T03:17:39.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa145ffb640 1 -- 172.21.15.77:0/2071412183 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa13c0ab830 con 0x7fa14806d4d0 2024-04-27T03:17:39.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa14d411640 1 -- 172.21.15.77:0/2071412183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 msgr2=0x7fa14806d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa14d411640 1 --2- 172.21.15.77:0/2071412183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa14806d8b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009740 tx=0x7fa13c01c370 comp rx=0 tx=0).stop 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa14d411640 1 -- 172.21.15.77:0/2071412183 shutdown_connections 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa14d411640 1 --2- 172.21.15.77:0/2071412183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa14806d8b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa14d411640 1 -- 172.21.15.77:0/2071412183 >> 172.21.15.77:0/2071412183 conn(0x7fa148099440 msgr2=0x7fa148099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa14d411640 1 -- 172.21.15.77:0/2071412183 shutdown_connections 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.770+0000 7fa14d411640 1 -- 172.21.15.77:0/2071412183 wait complete. 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa14d411640 1 Processor -- start 2024-04-27T03:17:39.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa14d411640 1 -- start start 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa14d411640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa148121830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa14d411640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa148136a70 con 0x7fa14806d4d0 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa148121830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa148121830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54916/0 (socket says 172.21.15.77:54916) 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa146ffd640 1 -- 172.21.15.77:0/908861227 learned_addr learned my addr 172.21.15.77:0/908861227 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa146ffd640 1 -- 172.21.15.77:0/908861227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa148095500 con 0x7fa14806d4d0 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa146ffd640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa148121830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0ae040 tx=0x7fa13c019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa12f7fe640 1 -- 172.21.15.77:0/908861227 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa13c0b8020 con 0x7fa14806d4d0 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa12f7fe640 1 -- 172.21.15.77:0/908861227 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa13c0b4410 con 0x7fa14806d4d0 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa148123100 con 0x7fa14806d4d0 2024-04-27T03:17:39.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa12f7fe640 1 -- 172.21.15.77:0/908861227 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa13c0b0040 con 0x7fa14806d4d0 2024-04-27T03:17:39.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa148121d70 con 0x7fa14806d4d0 2024-04-27T03:17:39.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa12f7fe640 1 -- 172.21.15.77:0/908861227 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7fa13c0bb020 con 0x7fa14806d4d0 2024-04-27T03:17:39.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa12f7fe640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa134035d20 0x7fa1340381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:39.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa12f7fe640 1 -- 172.21.15.77:0/908861227 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 3482+0+0 (secure 0 0 0) 0x7fa13c0e21b0 con 0x7fa14806d4d0 2024-04-27T03:17:39.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa148123ab0 con 0x7fa14806d4d0 2024-04-27T03:17:39.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa1467fc640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa134035d20 0x7fa1340381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:39.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.774+0000 7fa1467fc640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa134035d20 0x7fa1340381e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa138001610 tx=0x7fa138007ad0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:39.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.782+0000 7fa12f7fe640 1 -- 172.21.15.77:0/908861227 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa13c0c2030 con 0x7fa14806d4d0 2024-04-27T03:17:39.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.982+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa148062e50 con 0x7fa14806d4d0 2024-04-27T03:17:39.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa12f7fe640 1 -- 172.21.15.77:0/908861227 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+84086 (secure 0 0 0) 0x7fa13c015ff0 con 0x7fa14806d4d0 2024-04-27T03:17:39.996 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa134035d20 msgr2=0x7fa1340381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa134035d20 0x7fa1340381e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa138001610 tx=0x7fa138007ad0 comp rx=0 tx=0).stop 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 msgr2=0x7fa148121830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa148121830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0ae040 tx=0x7fa13c019110 comp rx=0 tx=0).stop 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 shutdown_connections 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa134035d20 0x7fa1340381e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 --2- 172.21.15.77:0/908861227 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa14806d4d0 0x7fa148121830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 >> 172.21.15.77:0/908861227 conn(0x7fa148099440 msgr2=0x7fa14808e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 shutdown_connections 2024-04-27T03:17:39.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:39.986+0000 7fa14d411640 1 -- 172.21.15.77:0/908861227 wait complete. 2024-04-27T03:17:40.013 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":2492415371},{"type":"v1","addr":"172.21.15.110:6825","nonce":2492415371}]},"active_addr":"172.21.15.110:6825/2492415371","active_change":"2024-04-27T03:17:33.629876+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":248178215}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1251638297}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":2196377725}]}]} 2024-04-27T03:17:40.015 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-27T03:17:40.015 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-27T03:17:40.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T03:17:40.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be3086640 1 Processor -- start 2024-04-27T03:17:40.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be3086640 1 -- start start 2024-04-27T03:17:40.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be3086640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9bdc0963b0 0x7f9bdc096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be3086640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bdc098d10 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be3086640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9bdc098610 con 0x7f9bdc0963b0 2024-04-27T03:17:40.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be0e22640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9bdc0963b0 0x7f9bdc096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9bdbfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9bdc06d4d0 0x7f9bdc06d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39246/0 (socket says 172.21.15.77:39246) 2024-04-27T03:17:40.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9bdbfff640 1 -- 172.21.15.77:0/360767652 learned_addr learned my addr 172.21.15.77:0/360767652 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be0e22640 1 -- 172.21.15.77:0/360767652 >> v1:172.21.15.77:6789/0 conn(0x7f9bdc06d4d0 legacy=0x7f9bdc06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.126+0000 7f9be0e22640 1 -- 172.21.15.77:0/360767652 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc071f90 con 0x7f9bdc0963b0 2024-04-27T03:17:40.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be0e22640 1 --2- 172.21.15.77:0/360767652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc0963b0 0x7f9bdc096790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009740 tx=0x7f9bc401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61db74ef1647c6b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9bdb7fe640 1 -- 172.21.15.77:0/360767652 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc40b0070 con 0x7f9bdc0963b0 2024-04-27T03:17:40.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9bdb7fe640 1 -- 172.21.15.77:0/360767652 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc40ab530 con 0x7f9bdc0963b0 2024-04-27T03:17:40.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9bdb7fe640 1 -- 172.21.15.77:0/360767652 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc40ab830 con 0x7f9bdc0963b0 2024-04-27T03:17:40.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 -- 172.21.15.77:0/360767652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc0963b0 msgr2=0x7f9bdc096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 --2- 172.21.15.77:0/360767652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc0963b0 0x7f9bdc096790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009740 tx=0x7f9bc401c400 comp rx=0 tx=0).stop 2024-04-27T03:17:40.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 -- 172.21.15.77:0/360767652 shutdown_connections 2024-04-27T03:17:40.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 --2- 172.21.15.77:0/360767652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc0963b0 0x7f9bdc096790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 -- 172.21.15.77:0/360767652 >> 172.21.15.77:0/360767652 conn(0x7f9bdc099440 msgr2=0x7f9bdc099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 -- 172.21.15.77:0/360767652 shutdown_connections 2024-04-27T03:17:40.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 -- 172.21.15.77:0/360767652 wait complete. 2024-04-27T03:17:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 Processor -- start 2024-04-27T03:17:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 -- start start 2024-04-27T03:17:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc06d4d0 0x7f9bdc121830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be3086640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc134d80 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be0e22640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc06d4d0 0x7f9bdc121830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be0e22640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc06d4d0 0x7f9bdc121830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54936/0 (socket says 172.21.15.77:54936) 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be0e22640 1 -- 172.21.15.77:0/2895948519 learned_addr learned my addr 172.21.15.77:0/2895948519 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.130+0000 7f9be0e22640 1 -- 172.21.15.77:0/2895948519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc131640 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9be0e22640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc06d4d0 0x7f9bdc121830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9bc401a030 tx=0x7f9bc401ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bd97fa640 1 -- 172.21.15.77:0/2895948519 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc40b0050 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bd97fa640 1 -- 172.21.15.77:0/2895948519 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc40b1040 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bdc131e10 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bd97fa640 1 -- 172.21.15.77:0/2895948519 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc40b5610 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bdc123560 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bd97fa640 1 -- 172.21.15.77:0/2895948519 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f9bc40be030 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bdc123cd0 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bd97fa640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bbc035d20 0x7f9bbc0381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bd97fa640 1 -- 172.21.15.77:0/2895948519 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 3482+0+0 (secure 0 0 0) 0x7f9bc40bc070 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bdbfff640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bbc035d20 0x7f9bbc0381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.134+0000 7f9bdbfff640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bbc035d20 0x7f9bbc0381e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc005c10 tx=0x7f9bcc0016a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.138+0000 7f9bd97fa640 1 -- 172.21.15.77:0/2895948519 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9bdc123cd0 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.290+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9bdc062e50 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9bd97fa640 1 -- 172.21.15.77:0/2895948519 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+9780 (secure 0 0 0) 0x7f9bc40c5060 con 0x7f9bdc06d4d0 2024-04-27T03:17:40.302 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:40.302 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"fsid":"3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3","created":"2024-04-27T03:17:28.925654+0000","modified":"2024-04-27T03:17:39.621350+0000","last_up_change":"2024-04-27T03:17:37.618506+0000","last_in_change":"2024-04-27T03:17:30.101677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T03:17:37.706444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-27T03:17:37.988012+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6801","nonce":3496324319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6803","nonce":3496324319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6807","nonce":3496324319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6805","nonce":3496324319}]},"public_addr":"172.21.15.77:6801/3496324319","cluster_addr":"172.21.15.77:6803/3496324319","heartbeat_back_addr":"172.21.15.77:6807/3496324319","heartbeat_front_addr":"172.21.15.77:6805/3496324319","state":["exists","up"]},{"osd":1,"uuid":"74a30006-c87c-4e0e-9e94-9020affcae97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6801","nonce":1860462708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6803","nonce":1860462708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6807","nonce":1860462708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6805","nonce":1860462708}]},"public_addr":"172.21.15.110:6801/1860462708","cluster_addr":"172.21.15.110:6803/1860462708","heartbeat_back_addr":"172.21.15.110:6807/1860462708","heartbeat_front_addr":"172.21.15.110:6805/1860462708","state":["exists","up"]},{"osd":2,"uuid":"1bd2dbfb-57cb-4673-9684-d9427036427d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6809","nonce":3575017494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6811","nonce":3575017494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6815","nonce":3575017494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6813","nonce":3575017494}]},"public_addr":"172.21.15.110:6809/3575017494","cluster_addr":"172.21.15.110:6811/3575017494","heartbeat_back_addr":"172.21.15.110:6815/3575017494","heartbeat_front_addr":"172.21.15.110:6813/3575017494","state":["exists","up"]},{"osd":3,"uuid":"d73614e8-5d01-4f79-8b12-0e08dabad6b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6817","nonce":3916647200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6819","nonce":3916647200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6823","nonce":3916647200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6821","nonce":3916647200}]},"public_addr":"172.21.15.110:6817/3916647200","cluster_addr":"172.21.15.110:6819/3916647200","heartbeat_back_addr":"172.21.15.110:6823/3916647200","heartbeat_front_addr":"172.21.15.110:6821/3916647200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.465777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.568546+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.593450+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.619617+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bbc035d20 msgr2=0x7f9bbc0381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bbc035d20 0x7f9bbc0381e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc005c10 tx=0x7f9bcc0016a0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc06d4d0 msgr2=0x7f9bdc121830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc06d4d0 0x7f9bdc121830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9bc401a030 tx=0x7f9bc401ac40 comp rx=0 tx=0).stop 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 shutdown_connections 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bbc035d20 0x7f9bbc0381e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 --2- 172.21.15.77:0/2895948519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9bdc06d4d0 0x7f9bdc121830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.304 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 >> 172.21.15.77:0/2895948519 conn(0x7f9bdc099440 msgr2=0x7f9bdc09a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 shutdown_connections 2024-04-27T03:17:40.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.294+0000 7f9be3086640 1 -- 172.21.15.77:0/2895948519 wait complete. 2024-04-27T03:17:40.319 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-27T03:17:40.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T03:17:40.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c1d73a640 1 Processor -- start 2024-04-27T03:17:40.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c1d73a640 1 -- start start 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c1d73a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c18093380 0x7f9c18093760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c1d73a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c18096470 con 0x7f9c18095bf0 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c1d73a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9c180731b0 con 0x7f9c18093380 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c16ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c18093380 0x7f9c18093760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c0ffff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9c18095bf0 0x7f9c1812c020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39256/0 (socket says 172.21.15.77:39256) 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c0ffff640 1 -- 172.21.15.77:0/1333296140 learned_addr learned my addr 172.21.15.77:0/1333296140 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c16ffd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c18093380 0x7f9c18093760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54938/0 (socket says 172.21.15.77:54938) 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c16ffd640 1 -- 172.21.15.77:0/1333296140 >> v1:172.21.15.77:6789/0 conn(0x7f9c18095bf0 legacy=0x7f9c1812c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c16ffd640 1 -- 172.21.15.77:0/1333296140 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c18096ab0 con 0x7f9c18093380 2024-04-27T03:17:40.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c16ffd640 1 --2- 172.21.15.77:0/1333296140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18093760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009740 tx=0x7f9c0801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45366a60da2b6efc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c167fc640 1 -- 172.21.15.77:0/1333296140 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c080b0070 con 0x7f9c18093380 2024-04-27T03:17:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c167fc640 1 -- 172.21.15.77:0/1333296140 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c080ab530 con 0x7f9c18093380 2024-04-27T03:17:40.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.430+0000 7f9c167fc640 1 -- 172.21.15.77:0/1333296140 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c080ab830 con 0x7f9c18093380 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/1333296140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 msgr2=0x7f9c18093760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 --2- 172.21.15.77:0/1333296140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18093760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009740 tx=0x7f9c0801c370 comp rx=0 tx=0).stop 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/1333296140 shutdown_connections 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 --2- 172.21.15.77:0/1333296140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18093760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/1333296140 >> 172.21.15.77:0/1333296140 conn(0x7f9c1809d040 msgr2=0x7f9c1809d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/1333296140 shutdown_connections 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/1333296140 wait complete. 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 Processor -- start 2024-04-27T03:17:40.442 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- start start 2024-04-27T03:17:40.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18121d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9c18134730 con 0x7f9c18093380 2024-04-27T03:17:40.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c16ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18121d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c16ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18121d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54942/0 (socket says 172.21.15.77:54942) 2024-04-27T03:17:40.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c16ffd640 1 -- 172.21.15.77:0/3653658858 learned_addr learned my addr 172.21.15.77:0/3653658858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c16ffd640 1 -- 172.21.15.77:0/3653658858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c18131100 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c16ffd640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18121d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9c080ae040 tx=0x7f9c0801abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c0f7fe640 1 -- 172.21.15.77:0/3653658858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c080b0050 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c0f7fe640 1 -- 172.21.15.77:0/3653658858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c080b1040 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c18131970 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c0f7fe640 1 -- 172.21.15.77:0/3653658858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c080b5740 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c18122df0 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c18062e50 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c0f7fe640 1 -- 172.21.15.77:0/3653658858 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f9c080be030 con 0x7f9c18093380 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c0f7fe640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bf8035d20 0x7f9bf80381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c0f7fe640 1 -- 172.21.15.77:0/3653658858 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 3482+0+0 (secure 0 0 0) 0x7f9c080bc070 con 0x7f9c18093380 2024-04-27T03:17:40.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.434+0000 7f9c0ffff640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bf8035d20 0x7f9bf80381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.438+0000 7f9c0ffff640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bf8035d20 0x7f9bf80381e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c00001630 tx=0x7f9c00007a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.442+0000 7f9c0f7fe640 1 -- 172.21.15.77:0/3653658858 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9c080c5020 con 0x7f9c18093380 2024-04-27T03:17:40.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.602+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c18074210 con 0x7f9c18093380 2024-04-27T03:17:40.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.602+0000 7f9c0f7fe640 1 -- 172.21.15.77:0/3653658858 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+9780 (secure 0 0 0) 0x7f9c08015ff0 con 0x7f9c18093380 2024-04-27T03:17:40.612 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:40.612 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":11,"fsid":"3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3","created":"2024-04-27T03:17:28.925654+0000","modified":"2024-04-27T03:17:39.621350+0000","last_up_change":"2024-04-27T03:17:37.618506+0000","last_in_change":"2024-04-27T03:17:30.101677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T03:17:37.706444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-27T03:17:37.988012+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6801","nonce":3496324319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6803","nonce":3496324319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6807","nonce":3496324319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6805","nonce":3496324319}]},"public_addr":"172.21.15.77:6801/3496324319","cluster_addr":"172.21.15.77:6803/3496324319","heartbeat_back_addr":"172.21.15.77:6807/3496324319","heartbeat_front_addr":"172.21.15.77:6805/3496324319","state":["exists","up"]},{"osd":1,"uuid":"74a30006-c87c-4e0e-9e94-9020affcae97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6801","nonce":1860462708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6803","nonce":1860462708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6807","nonce":1860462708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6805","nonce":1860462708}]},"public_addr":"172.21.15.110:6801/1860462708","cluster_addr":"172.21.15.110:6803/1860462708","heartbeat_back_addr":"172.21.15.110:6807/1860462708","heartbeat_front_addr":"172.21.15.110:6805/1860462708","state":["exists","up"]},{"osd":2,"uuid":"1bd2dbfb-57cb-4673-9684-d9427036427d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6809","nonce":3575017494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6811","nonce":3575017494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6815","nonce":3575017494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6813","nonce":3575017494}]},"public_addr":"172.21.15.110:6809/3575017494","cluster_addr":"172.21.15.110:6811/3575017494","heartbeat_back_addr":"172.21.15.110:6815/3575017494","heartbeat_front_addr":"172.21.15.110:6813/3575017494","state":["exists","up"]},{"osd":3,"uuid":"d73614e8-5d01-4f79-8b12-0e08dabad6b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6817","nonce":3916647200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6819","nonce":3916647200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6823","nonce":3916647200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6821","nonce":3916647200}]},"public_addr":"172.21.15.110:6817/3916647200","cluster_addr":"172.21.15.110:6819/3916647200","heartbeat_back_addr":"172.21.15.110:6823/3916647200","heartbeat_front_addr":"172.21.15.110:6821/3916647200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.465777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.568546+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.593450+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.619617+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T03:17:40.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bf8035d20 msgr2=0x7f9bf80381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bf8035d20 0x7f9bf80381e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9c00001630 tx=0x7f9c00007a90 comp rx=0 tx=0).stop 2024-04-27T03:17:40.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 msgr2=0x7f9c18121d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18121d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9c080ae040 tx=0x7f9c0801abb0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 shutdown_connections 2024-04-27T03:17:40.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9bf8035d20 0x7f9bf80381e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 --2- 172.21.15.77:0/3653658858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c18093380 0x7f9c18121d00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 >> 172.21.15.77:0/3653658858 conn(0x7f9c1809d040 msgr2=0x7f9c18071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 shutdown_connections 2024-04-27T03:17:40.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.606+0000 7f9c1d73a640 1 -- 172.21.15.77:0/3653658858 wait complete. 2024-04-27T03:17:40.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-27T03:17:40.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-27T03:17:40.632 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-27T03:17:40.632 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-27T03:17:40.741 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f949813c640 1 Processor -- start 2024-04-27T03:17:40.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f0291725640 1 Processor -- start 2024-04-27T03:17:40.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f0291725640 1 -- start start 2024-04-27T03:17:40.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f949813c640 1 -- start start 2024-04-27T03:17:40.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa54c188640 1 Processor -- start 2024-04-27T03:17:40.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa54c188640 1 -- start start 2024-04-27T03:17:40.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f0291725640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f028c094ac0 0x7f028c133e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f0291725640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f028c097040 con 0x7f028c093b60 2024-04-27T03:17:40.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f0291725640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f028c096af0 con 0x7f028c094ac0 2024-04-27T03:17:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f949813c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f949006fd90 0x7f9490133700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f949813c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9490133c40 con 0x7f949006f2a0 2024-04-27T03:17:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f949813c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9490070cf0 con 0x7f949006fd90 2024-04-27T03:17:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f028a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f028c094ac0 0x7f028c133e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f9495ed8640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f949006f2a0 0x7f949006f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39266/0 (socket says 172.21.15.77:39266) 2024-04-27T03:17:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f9495ed8640 1 -- 172.21.15.77:0/1849347512 learned_addr learned my addr 172.21.15.77:0/1849347512 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f028a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f028c094ac0 0x7f028c133e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54948/0 (socket says 172.21.15.77:54948) 2024-04-27T03:17:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f028a7fc640 1 -- 172.21.15.77:0/1615909098 learned_addr learned my addr 172.21.15.77:0/1615909098 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f94956d7640 1 --2- 172.21.15.77:0/1849347512 >> v2:172.21.15.77:3300/0 conn(0x7f949006fd90 0x7f9490133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa54c188640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fa54406ffe0 0x7fa544130580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa54c188640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54409e2c0 con 0x7fa54406f4f0 2024-04-27T03:17:40.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa54c188640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fa544071e00 con 0x7fa54406ffe0 2024-04-27T03:17:40.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa549f24640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fa54406f4f0 0x7fa54406f8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39272/0 (socket says 172.21.15.77:39272) 2024-04-27T03:17:40.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa549f24640 1 -- 172.21.15.77:0/1334240432 learned_addr learned my addr 172.21.15.77:0/1334240432 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f94956d7640 1 -- 172.21.15.77:0/1849347512 >> v1:172.21.15.77:6789/0 conn(0x7f949006f2a0 legacy=0x7f949006f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f94956d7640 1 -- 172.21.15.77:0/1849347512 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490134e20 con 0x7f949006fd90 2024-04-27T03:17:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f028a7fc640 1 -- 172.21.15.77:0/1615909098 >> v1:172.21.15.77:6789/0 conn(0x7f028c093b60 legacy=0x7f028c0943b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T03:17:40.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7f028a7fc640 1 -- 172.21.15.77:0/1615909098 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c06b470 con 0x7f028c094ac0 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035149203 0 0) 0x7fa54409e2c0 con 0x7fa54406f4f0 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa530005740 con 0x7fa54406f4f0 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa549723640 1 --2- 172.21.15.77:0/1334240432 >> v2:172.21.15.77:3300/0 conn(0x7fa54406ffe0 0x7fa544130580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3531368901 0 0) 0x7fa530005740 con 0x7fa54406f4f0 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 >> v2:172.21.15.77:3300/0 conn(0x7fa54406ffe0 msgr2=0x7fa544130580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 --2- 172.21.15.77:0/1334240432 >> v2:172.21.15.77:3300/0 conn(0x7fa54406ffe0 0x7fa544130580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa544070e30 con 0x7fa54406f4f0 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 205+0+0 (unknown 201508238 0 0) 0x7fa5400015b0 con 0x7fa54406f4f0 2024-04-27T03:17:40.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 >> v1:172.21.15.77:6789/0 conn(0x7fa54406f4f0 legacy=0x7fa54406f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 --2- 172.21.15.77:0/1334240432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa530005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa530006690 con 0x7fa530005bd0 2024-04-27T03:17:40.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa549f24640 1 --2- 172.21.15.77:0/1334240432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa530005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.734+0000 7fa549f24640 1 -- 172.21.15.77:0/1334240432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa530005740 con 0x7fa530005bd0 2024-04-27T03:17:40.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f028a7fc640 1 --2- 172.21.15.77:0/1615909098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c094ac0 0x7f028c133e00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f027c0097f0 tx=0x7f027c01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9909728d125bac47 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.751 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0289ffb640 1 -- 172.21.15.77:0/1615909098 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f027c0b0070 con 0x7f028c094ac0 2024-04-27T03:17:40.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0289ffb640 1 -- 172.21.15.77:0/1615909098 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027c0ab530 con 0x7f028c094ac0 2024-04-27T03:17:40.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0289ffb640 1 -- 172.21.15.77:0/1615909098 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f027c0ab830 con 0x7f028c094ac0 2024-04-27T03:17:40.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f94956d7640 1 --2- 172.21.15.77:0/1849347512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006fd90 0x7f9490133700 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9480009740 tx=0x7f948001c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31efadfa154feec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f9494ed6640 1 -- 172.21.15.77:0/1849347512 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94800b0070 con 0x7f949006fd90 2024-04-27T03:17:40.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f9494ed6640 1 -- 172.21.15.77:0/1849347512 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94800ab530 con 0x7f949006fd90 2024-04-27T03:17:40.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f9494ed6640 1 -- 172.21.15.77:0/1849347512 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94800ab830 con 0x7f949006fd90 2024-04-27T03:17:40.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- 172.21.15.77:0/1615909098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c094ac0 msgr2=0x7f028c133e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 --2- 172.21.15.77:0/1615909098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c094ac0 0x7f028c133e00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f027c0097f0 tx=0x7f027c01c390 comp rx=0 tx=0).stop 2024-04-27T03:17:40.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- 172.21.15.77:0/1615909098 shutdown_connections 2024-04-27T03:17:40.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 --2- 172.21.15.77:0/1615909098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c094ac0 0x7f028c133e00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- 172.21.15.77:0/1615909098 >> 172.21.15.77:0/1615909098 conn(0x7f028c09d040 msgr2=0x7f028c09d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- 172.21.15.77:0/1615909098 shutdown_connections 2024-04-27T03:17:40.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- 172.21.15.77:0/1615909098 wait complete. 2024-04-27T03:17:40.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa549f24640 1 --2- 172.21.15.77:0/1334240432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa530005fd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c190 tx=0x7fa54001cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1205b5a221d2f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa540007cf0 con 0x7fa530005bd0 2024-04-27T03:17:40.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5400a9070 con 0x7fa530005bd0 2024-04-27T03:17:40.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa548f22640 1 -- 172.21.15.77:0/1334240432 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa540005ba0 con 0x7fa530005bd0 2024-04-27T03:17:40.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 -- 172.21.15.77:0/1849347512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006fd90 msgr2=0x7f9490133700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 --2- 172.21.15.77:0/1849347512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006fd90 0x7f9490133700 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9480009740 tx=0x7f948001c370 comp rx=0 tx=0).stop 2024-04-27T03:17:40.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 -- 172.21.15.77:0/1849347512 shutdown_connections 2024-04-27T03:17:40.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 --2- 172.21.15.77:0/1849347512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006fd90 0x7f9490133700 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 -- 172.21.15.77:0/1849347512 >> 172.21.15.77:0/1849347512 conn(0x7f949009e0d0 msgr2=0x7f949006d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 -- 172.21.15.77:0/1849347512 shutdown_connections 2024-04-27T03:17:40.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 -- 172.21.15.77:0/1849347512 wait complete. 2024-04-27T03:17:40.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 Processor -- start 2024-04-27T03:17:40.756 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- start start 2024-04-27T03:17:40.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 -- 172.21.15.77:0/1334240432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 msgr2=0x7fa530005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 --2- 172.21.15.77:0/1334240432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa530005fd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c190 tx=0x7fa54001cce0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c093b60 0x7f028c121ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f028c136ce0 con 0x7f028c093b60 2024-04-27T03:17:40.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 -- 172.21.15.77:0/1334240432 shutdown_connections 2024-04-27T03:17:40.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 --2- 172.21.15.77:0/1334240432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa530005fd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 --2- 172.21.15.77:0/1334240432 >> v2:172.21.15.77:3300/0 conn(0x7fa54406ffe0 0x7fa544130580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 -- 172.21.15.77:0/1334240432 >> 172.21.15.77:0/1334240432 conn(0x7fa54409d610 msgr2=0x7fa54409aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f028affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c093b60 0x7f028c121ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 -- 172.21.15.77:0/1334240432 shutdown_connections 2024-04-27T03:17:40.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7fa54c188640 1 -- 172.21.15.77:0/1334240432 wait complete. 2024-04-27T03:17:40.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f028affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c093b60 0x7f028c121ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54986/0 (socket says 172.21.15.77:54986) 2024-04-27T03:17:40.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f028affd640 1 -- 172.21.15.77:0/2407253746 learned_addr learned my addr 172.21.15.77:0/2407253746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 Processor -- start 2024-04-27T03:17:40.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 -- start start 2024-04-27T03:17:40.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f028affd640 1 -- 172.21.15.77:0/2407253746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c123a20 con 0x7f028c093b60 2024-04-27T03:17:40.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f028affd640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c093b60 0x7f028c121ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f028000c390 tx=0x7f0280001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0280017070 con 0x7f028c093b60 2024-04-27T03:17:40.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0280005430 con 0x7f028c093b60 2024-04-27T03:17:40.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0280003770 con 0x7f028c093b60 2024-04-27T03:17:40.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006f2a0 0x7f949011a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028c124e20 con 0x7f028c093b60 2024-04-27T03:17:40.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f949813c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f94901349f0 con 0x7f949006f2a0 2024-04-27T03:17:40.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028c1246b0 con 0x7f028c093b60 2024-04-27T03:17:40.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.738+0000 7f9495ed8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006f2a0 0x7f949011a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa54c188640 1 Processor -- start 2024-04-27T03:17:40.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f9495ed8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006f2a0 0x7f949011a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54992/0 (socket says 172.21.15.77:54992) 2024-04-27T03:17:40.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f9495ed8640 1 -- 172.21.15.77:0/261372694 learned_addr learned my addr 172.21.15.77:0/261372694 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f9495ed8640 1 -- 172.21.15.77:0/261372694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949011c860 con 0x7f949006f2a0 2024-04-27T03:17:40.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa54c188640 1 -- start start 2024-04-27T03:17:40.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa54c188640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa5441258d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa54c188640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fa54406d960 con 0x7fa530005bd0 2024-04-27T03:17:40.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f028c122fd0 con 0x7f028c093b60 2024-04-27T03:17:40.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa549f24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa5441258d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa549f24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa5441258d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54996/0 (socket says 172.21.15.77:54996) 2024-04-27T03:17:40.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa549f24640 1 -- 172.21.15.77:0/419493277 learned_addr learned my addr 172.21.15.77:0/419493277 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa549f24640 1 -- 172.21.15.77:0/419493277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa544116740 con 0x7fa530005bd0 2024-04-27T03:17:40.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f028001a070 con 0x7f028c093b60 2024-04-27T03:17:40.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f02737fe640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0278035d20 0x7f02780381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f02800480a0 con 0x7f028c093b60 2024-04-27T03:17:40.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f02737fe640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f027803b4a0 0x7f027803d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0280016ec0 con 0x7f027803b4a0 2024-04-27T03:17:40.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f0280048490 con 0x7f028c093b60 2024-04-27T03:17:40.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f9495ed8640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006f2a0 0x7f949011a8b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f948c00c380 tx=0x7f948c00c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f948c017070 con 0x7f949006f2a0 2024-04-27T03:17:40.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948c005c20 con 0x7f949006f2a0 2024-04-27T03:17:40.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949011c620 con 0x7f949006f2a0 2024-04-27T03:17:40.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f948c0070d0 con 0x7f949006f2a0 2024-04-27T03:17:40.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949011d2b0 con 0x7f949006f2a0 2024-04-27T03:17:40.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9478000f90 con 0x7f949006f2a0 2024-04-27T03:17:40.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa549f24640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa5441258d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa54001cd20 tx=0x7fa530007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f028a7fc640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0278035d20 0x7f02780381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f028b7fe640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f027803b4a0 0x7f027803d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa540007cf0 con 0x7fa530005bd0 2024-04-27T03:17:40.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5400b4070 con 0x7fa530005bd0 2024-04-27T03:17:40.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5400047f0 con 0x7fa530005bd0 2024-04-27T03:17:40.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa544116e40 con 0x7fa530005bd0 2024-04-27T03:17:40.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e15aad640 1 Processor -- start 2024-04-27T03:17:40.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f948c01a070 con 0x7f949006f2a0 2024-04-27T03:17:40.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f947403e4e0 0x7f94740409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f948c047870 con 0x7f949006f2a0 2024-04-27T03:17:40.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f9474043c60 0x7f94740460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9474046770 con 0x7f9474043c60 2024-04-27T03:17:40.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f948c04c2a0 con 0x7f949006f2a0 2024-04-27T03:17:40.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f028b7fe640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f027803b4a0 0x7f027803d8e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94956d7640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f947403e4e0 0x7f94740409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94966d9640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f9474043c60 0x7f94740460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa544114290 con 0x7fa530005bd0 2024-04-27T03:17:40.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa544062e50 con 0x7fa530005bd0 2024-04-27T03:17:40.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7fa5400b6070 con 0x7fa530005bd0 2024-04-27T03:17:40.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f94966d9640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f9474043c60 0x7f94740460a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e15aad640 1 -- start start 2024-04-27T03:17:40.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e15aad640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e10071010 0x7f0e10130510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e15aad640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e10071ed0 con 0x7f0e1012af70 2024-04-27T03:17:40.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e15aad640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f0e1009ac80 con 0x7f0e10071010 2024-04-27T03:17:40.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa52803a130 0x7fa52803c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e0effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e10071010 0x7f0e10130510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e0effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0e10071010 0x7f0e10130510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55006/0 (socket says 172.21.15.77:55006) 2024-04-27T03:17:40.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7f0e0effd640 1 -- 172.21.15.77:0/4279338208 learned_addr learned my addr 172.21.15.77:0/4279338208 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7fa5400e1d00 con 0x7fa530005bd0 2024-04-27T03:17:40.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7fa52803f8b0 0x7fa528041cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa540006e50 con 0x7fa52803f8b0 2024-04-27T03:17:40.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.742+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7fa5400e4360 con 0x7fa530005bd0 2024-04-27T03:17:40.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e0effd640 1 -- 172.21.15.77:0/4279338208 >> v1:172.21.15.77:6789/0 conn(0x7f0e1012af70 legacy=0x7f0e1012b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T03:17:40.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e0effd640 1 -- 172.21.15.77:0/4279338208 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e10071bc0 con 0x7f0e10071010 2024-04-27T03:17:40.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f028a7fc640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0278035d20 0x7f02780381e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f027c0ad000 tx=0x7f027c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7fa549723640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa52803a130 0x7fa52803c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7fa54a725640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7fa52803f8b0 0x7fa528041cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9474046770 con 0x7f9474043c60 2024-04-27T03:17:40.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f94956d7640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f947403e4e0 0x7f94740409a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f94800ad000 tx=0x7f94800a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7fa54a725640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7fa52803f8b0 0x7fa528041cf0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7fa549723640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa52803a130 0x7fa52803c5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa534009930 tx=0x7fa534009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e0effd640 1 --2- 172.21.15.77:0/4279338208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10130510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0e080097f0 tx=0x7f0e0801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614b1fab90ed961e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e0e7fc640 1 -- 172.21.15.77:0/4279338208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e080b0070 con 0x7f0e10071010 2024-04-27T03:17:40.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e0e7fc640 1 -- 172.21.15.77:0/4279338208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e080ab530 con 0x7f0e10071010 2024-04-27T03:17:40.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e0e7fc640 1 -- 172.21.15.77:0/4279338208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e080ab830 con 0x7f0e10071010 2024-04-27T03:17:40.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e15aad640 1 -- 172.21.15.77:0/4279338208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 msgr2=0x7f0e10130510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7f0e15aad640 1 --2- 172.21.15.77:0/4279338208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10130510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0e080097f0 tx=0x7f0e0801c3f0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.746+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa540006e50 con 0x7fa52803f8b0 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 -- 172.21.15.77:0/4279338208 shutdown_connections 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 --2- 172.21.15.77:0/4279338208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10130510 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 -- 172.21.15.77:0/4279338208 >> 172.21.15.77:0/4279338208 conn(0x7f0e1009c320 msgr2=0x7f0e1009c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 -- 172.21.15.77:0/4279338208 shutdown_connections 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 -- 172.21.15.77:0/4279338208 wait complete. 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 Processor -- start 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 -- start start 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10128470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f0e15aad640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1006d110 con 0x7f0e10071010 2024-04-27T03:17:40.777 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.750+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0280016ec0 con 0x7f027803b4a0 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10128470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10128470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55010/0 (socket says 172.21.15.77:55010) 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0f7fe640 1 -- 172.21.15.77:0/318082848 learned_addr learned my addr 172.21.15.77:0/318082848 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0f7fe640 1 -- 172.21.15.77:0/318082848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e10119100 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0f7fe640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10128470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c740 tx=0x7f0e0000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e00017070 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e00003bb0 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0e00005070 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e1012ab00 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e10119d60 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e10118480 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f0e0001a070 con 0x7f0e10071010 2024-04-27T03:17:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0df0035d20 0x7f0df00381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f0e00047c20 con 0x7f0e10071010 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f0df003b4a0 0x7f0df003d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0df003dfb0 con 0x7f0df003b4a0 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=12) ==== 24+0+0 (secure 0 0 0) 0x7f0e0004c2a0 con 0x7f0e10071010 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0effd640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0df0035d20 0x7f0df00381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0ffff640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f0df003b4a0 0x7f0df003d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0effd640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0df0035d20 0x7f0df00381e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e080ad000 tx=0x7f0e08008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.754+0000 7f0e0ffff640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f0df003b4a0 0x7f0df003d8e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.781 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.758+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0df003dfb0 con 0x7f0df003b4a0 2024-04-27T03:17:40.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa544116b30 con 0x7fa52803f8b0 2024-04-27T03:17:40.782 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa53a7fc640 1 -- 172.21.15.77:0/419493277 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa544116b30 con 0x7fa52803f8b0 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7fa52803f8b0 msgr2=0x7fa528041cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7fa52803f8b0 0x7fa528041cf0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa52803a130 msgr2=0x7fa52803c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa52803a130 0x7fa52803c5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa534009930 tx=0x7fa534009190 comp rx=0 tx=0).stop 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 msgr2=0x7fa5441258d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa5441258d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa54001cd20 tx=0x7fa530007a50 comp rx=0 tx=0).stop 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 shutdown_connections 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7fa52803f8b0 0x7fa528041cf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa52803a130 0x7fa52803c5f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 --2- 172.21.15.77:0/419493277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa530005bd0 0x7fa5441258d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 >> 172.21.15.77:0/419493277 conn(0x7fa54409d610 msgr2=0x7fa5441281c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 shutdown_connections 2024-04-27T03:17:40.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.762+0000 7fa54c188640 1 -- 172.21.15.77:0/419493277 wait complete. 2024-04-27T03:17:40.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9478002c80 con 0x7f9474043c60 2024-04-27T03:17:40.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f94867fc640 1 -- 172.21.15.77:0/261372694 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9478002c80 con 0x7f9474043c60 2024-04-27T03:17:40.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f9474043c60 msgr2=0x7f94740460a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.785 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f9474043c60 0x7f94740460a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f947403e4e0 msgr2=0x7f94740409a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f947403e4e0 0x7f94740409a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f94800ad000 tx=0x7f94800a6000 comp rx=0 tx=0).stop 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006f2a0 msgr2=0x7f949011a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006f2a0 0x7f949011a8b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f948c00c380 tx=0x7f948c00c840 comp rx=0 tx=0).stop 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 shutdown_connections 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f9474043c60 0x7f94740460a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f947403e4e0 0x7f94740409a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 --2- 172.21.15.77:0/261372694 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f949006f2a0 0x7f949011a8b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 >> 172.21.15.77:0/261372694 conn(0x7f949009e0d0 msgr2=0x7f949012eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 shutdown_connections 2024-04-27T03:17:40.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f949813c640 1 -- 172.21.15.77:0/261372694 wait complete. 2024-04-27T03:17:40.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.770+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e10062e50 con 0x7f0df003b4a0 2024-04-27T03:17:40.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e07fff640 1 -- 172.21.15.77:0/318082848 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0e10062e50 con 0x7f0df003b4a0 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f0df003b4a0 msgr2=0x7f0df003d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f0df003b4a0 0x7f0df003d8e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0df0035d20 msgr2=0x7f0df00381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0df0035d20 0x7f0df00381e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0e080ad000 tx=0x7f0e08008040 comp rx=0 tx=0).stop 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 msgr2=0x7f0e10128470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10128470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000c740 tx=0x7f0e0000cc00 comp rx=0 tx=0).stop 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 shutdown_connections 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f0df003b4a0 0x7f0df003d8e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0df0035d20 0x7f0df00381e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 --2- 172.21.15.77:0/318082848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e10071010 0x7f0e10128470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 >> 172.21.15.77:0/318082848 conn(0x7f0e1009c320 msgr2=0x7f0e10129420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 shutdown_connections 2024-04-27T03:17:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0e15aad640 1 -- 172.21.15.77:0/318082848 wait complete. 2024-04-27T03:17:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.774+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f028c122fd0 con 0x7f027803b4a0 2024-04-27T03:17:40.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f02737fe640 1 -- 172.21.15.77:0/2407253746 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f028c122fd0 con 0x7f027803b4a0 2024-04-27T03:17:40.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f027803b4a0 msgr2=0x7f027803d8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f027803b4a0 0x7f027803d8e0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0278035d20 msgr2=0x7f02780381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0278035d20 0x7f02780381e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f027c0ad000 tx=0x7f027c008040 comp rx=0 tx=0).stop 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c093b60 msgr2=0x7f028c121ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c093b60 0x7f028c121ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f028000c390 tx=0x7f0280001e80 comp rx=0 tx=0).stop 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 shutdown_connections 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0278035d20 0x7f02780381e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f027803b4a0 0x7f027803d8e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 --2- 172.21.15.77:0/2407253746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f028c093b60 0x7f028c121ed0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 >> 172.21.15.77:0/2407253746 conn(0x7f028c09d040 msgr2=0x7f028c071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 shutdown_connections 2024-04-27T03:17:40.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.778+0000 7f0291725640 1 -- 172.21.15.77:0/2407253746 wait complete. 2024-04-27T03:17:40.792 INFO:teuthology.orchestra.run.smithi077.stdout:38654705668 2024-04-27T03:17:40.792 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-27T03:17:40.801 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2024-04-27T03:17:40.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-27T03:17:40.807 INFO:teuthology.orchestra.run.smithi077.stdout:38654705668 2024-04-27T03:17:40.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-27T03:17:40.811 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2024-04-27T03:17:40.811 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-27T03:17:40.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f9218457640 1 Processor -- start 2024-04-27T03:17:40.898 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f9218457640 1 -- start start 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f9218457640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9210071300 0x7f92101302d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f9218457640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9210071cd0 con 0x7f9210070810 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f9218457640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f92100719c0 con 0x7f9210071300 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f92159f2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9210071300 0x7f92101302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f92161f3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9210070810 0x7f9210070bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39284/0 (socket says 172.21.15.77:39284) 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f92159f2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9210071300 0x7f92101302d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55024/0 (socket says 172.21.15.77:55024) 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f92161f3640 1 -- 172.21.15.77:0/3526669298 learned_addr learned my addr 172.21.15.77:0/3526669298 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f92159f2640 1 -- 172.21.15.77:0/3526669298 >> v1:172.21.15.77:6789/0 conn(0x7f9210070810 legacy=0x7f9210070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.899 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.886+0000 7f92159f2640 1 -- 172.21.15.77:0/3526669298 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9210099c40 con 0x7f9210071300 2024-04-27T03:17:40.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f92159f2640 1 --2- 172.21.15.77:0/3526669298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210071300 0x7f92101302d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9200009740 tx=0x7f920001e580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989834d3ff0a2959 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f92151f1640 1 -- 172.21.15.77:0/3526669298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92000b3070 con 0x7f9210071300 2024-04-27T03:17:40.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f92151f1640 1 -- 172.21.15.77:0/3526669298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92000ae530 con 0x7f9210071300 2024-04-27T03:17:40.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f92151f1640 1 -- 172.21.15.77:0/3526669298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92000ae830 con 0x7f9210071300 2024-04-27T03:17:40.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f9218457640 1 -- 172.21.15.77:0/3526669298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210071300 msgr2=0x7f92101302d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f9218457640 1 --2- 172.21.15.77:0/3526669298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210071300 0x7f92101302d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9200009740 tx=0x7f920001e580 comp rx=0 tx=0).stop 2024-04-27T03:17:40.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f9218457640 1 -- 172.21.15.77:0/3526669298 shutdown_connections 2024-04-27T03:17:40.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f9218457640 1 --2- 172.21.15.77:0/3526669298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210071300 0x7f92101302d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f9218457640 1 -- 172.21.15.77:0/3526669298 >> 172.21.15.77:0/3526669298 conn(0x7f921009da80 msgr2=0x7f921009de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f9218457640 1 -- 172.21.15.77:0/3526669298 shutdown_connections 2024-04-27T03:17:40.901 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.890+0000 7f9218457640 1 -- 172.21.15.77:0/3526669298 wait complete. 2024-04-27T03:17:40.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9218457640 1 Processor -- start 2024-04-27T03:17:40.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9218457640 1 -- start start 2024-04-27T03:17:40.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9218457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210070810 0x7f921011ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9218457640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f921006d7a0 con 0x7f9210070810 2024-04-27T03:17:40.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f92161f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210070810 0x7f921011ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f92161f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210070810 0x7f921011ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55026/0 (socket says 172.21.15.77:55026) 2024-04-27T03:17:40.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f92161f3640 1 -- 172.21.15.77:0/1839409548 learned_addr learned my addr 172.21.15.77:0/1839409548 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f92161f3640 1 -- 172.21.15.77:0/1839409548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921011de40 con 0x7f9210070810 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f92161f3640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210070810 0x7f921011ace0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c480 tx=0x7f920c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9206ffd640 1 -- 172.21.15.77:0/1839409548 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f920c017070 con 0x7f9210070810 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9206ffd640 1 -- 172.21.15.77:0/1839409548 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c005c20 con 0x7f9210070810 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921011d910 con 0x7f9210070810 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9206ffd640 1 -- 172.21.15.77:0/1839409548 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f920c007070 con 0x7f9210070810 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921011b3f0 con 0x7f9210070810 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9210062e50 con 0x7f9210070810 2024-04-27T03:17:40.904 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9206ffd640 1 -- 172.21.15.77:0/1839409548 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f920c01a070 con 0x7f9210070810 2024-04-27T03:17:40.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9206ffd640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91f4035d20 0x7f91f40381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.894+0000 7f9206ffd640 1 -- 172.21.15.77:0/1839409548 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f920c047810 con 0x7f9210070810 2024-04-27T03:17:40.905 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.898+0000 7f92159f2640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91f4035d20 0x7f91f40381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.898+0000 7f92159f2640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91f4035d20 0x7f91f40381e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9200009740 tx=0x7f9200008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.902+0000 7f6f93b58640 1 Processor -- start 2024-04-27T03:17:40.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.902+0000 7f6f93b58640 1 -- start start 2024-04-27T03:17:40.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8c06f2a0 0x7f6f8c06f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8c06ffd0 con 0x7f6f8c06fbc0 2024-04-27T03:17:40.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6f8c070cf0 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f918f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8c06f2a0 0x7f6f8c06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f910f3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6f8c06fbc0 0x7f6f8c12e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39292/0 (socket says 172.21.15.77:39292) 2024-04-27T03:17:40.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f918f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6f8c06f2a0 0x7f6f8c06f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55040/0 (socket says 172.21.15.77:55040) 2024-04-27T03:17:40.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f910f3640 1 -- 172.21.15.77:0/1309294282 learned_addr learned my addr 172.21.15.77:0/1309294282 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f9206ffd640 1 -- 172.21.15.77:0/1839409548 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f920c01a760 con 0x7f9210070810 2024-04-27T03:17:40.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f918f4640 1 -- 172.21.15.77:0/1309294282 >> v1:172.21.15.77:6789/0 conn(0x7f6f8c06fbc0 legacy=0x7f6f8c12e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f918f4640 1 -- 172.21.15.77:0/1309294282 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c12eaf0 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f918f4640 1 --2- 172.21.15.77:0/1309294282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c06f680 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009740 tx=0x7f6f8801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fe06b304dc415f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f908f2640 1 -- 172.21.15.77:0/1309294282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f880b0070 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f908f2640 1 -- 172.21.15.77:0/1309294282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f880ab530 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f908f2640 1 -- 172.21.15.77:0/1309294282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f880ab830 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 -- 172.21.15.77:0/1309294282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 msgr2=0x7f6f8c06f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 --2- 172.21.15.77:0/1309294282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c06f680 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009740 tx=0x7f6f8801c370 comp rx=0 tx=0).stop 2024-04-27T03:17:40.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 -- 172.21.15.77:0/1309294282 shutdown_connections 2024-04-27T03:17:40.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 --2- 172.21.15.77:0/1309294282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c06f680 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 -- 172.21.15.77:0/1309294282 >> 172.21.15.77:0/1309294282 conn(0x7f6f8c09e0d0 msgr2=0x7f6f8c06d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 -- 172.21.15.77:0/1309294282 shutdown_connections 2024-04-27T03:17:40.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.906+0000 7f6f93b58640 1 -- 172.21.15.77:0/1309294282 wait complete. 2024-04-27T03:17:40.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f93b58640 1 Processor -- start 2024-04-27T03:17:40.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f93b58640 1 -- start start 2024-04-27T03:17:40.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f93b58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c11ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f93b58640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c070ab0 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f918f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c11ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f918f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c11ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55044/0 (socket says 172.21.15.77:55044) 2024-04-27T03:17:40.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f918f4640 1 -- 172.21.15.77:0/2143783095 learned_addr learned my addr 172.21.15.77:0/2143783095 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f918f4640 1 -- 172.21.15.77:0/2143783095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c11db60 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.919 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f918f4640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c11ada0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6f880ae040 tx=0x7f6f88019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f827fc640 1 -- 172.21.15.77:0/2143783095 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f880b8020 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f827fc640 1 -- 172.21.15.77:0/2143783095 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f880aa800 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f827fc640 1 -- 172.21.15.77:0/2143783095 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f880b0040 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8c11d3c0 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8c11b2e0 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.920 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f8c062e50 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.910+0000 7f6f827fc640 1 -- 172.21.15.77:0/2143783095 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f6f880bb020 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.914+0000 7f6f827fc640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6f70035cb0 0x7f6f70038170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.914+0000 7f6f827fc640 1 -- 172.21.15.77:0/2143783095 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f6f880e2520 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.914+0000 7f6f910f3640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6f70035cb0 0x7f6f70038170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.914+0000 7f6f910f3640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6f70035cb0 0x7f6f70038170 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c001610 tx=0x7f6f7c007ad0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.930 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.922+0000 7f6f827fc640 1 -- 172.21.15.77:0/2143783095 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6f880c2030 con 0x7f6f8c06f2a0 2024-04-27T03:17:40.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7afd781640 1 Processor -- start 2024-04-27T03:17:40.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7afd781640 1 -- start start 2024-04-27T03:17:40.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7afd781640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7af81299f0 0x7f7af80707b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7afd781640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7af8072260 con 0x7f7af8129610 2024-04-27T03:17:40.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7afd781640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f7af80725a0 con 0x7f7af81299f0 2024-04-27T03:17:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7af67fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f7af81299f0 0x7f7af80707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7af6ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f7af8129610 0x7f7af80700a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39294/0 (socket says 172.21.15.77:39294) 2024-04-27T03:17:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7af6ffd640 1 -- 172.21.15.77:0/3316318682 learned_addr learned my addr 172.21.15.77:0/3316318682 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7af67fc640 1 -- 172.21.15.77:0/3316318682 >> v1:172.21.15.77:6789/0 conn(0x7f7af8129610 legacy=0x7f7af80700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.958+0000 7f7af67fc640 1 -- 172.21.15.77:0/3316318682 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af809ba10 con 0x7f7af81299f0 2024-04-27T03:17:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af67fc640 1 --2- 172.21.15.77:0/3316318682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af81299f0 0x7f7af80707b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009740 tx=0x7f7aec01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955139f0ba23775f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af5ffb640 1 -- 172.21.15.77:0/3316318682 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7aec0b0070 con 0x7f7af81299f0 2024-04-27T03:17:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af5ffb640 1 -- 172.21.15.77:0/3316318682 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aec0ab530 con 0x7f7af81299f0 2024-04-27T03:17:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af5ffb640 1 -- 172.21.15.77:0/3316318682 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7aec0ab830 con 0x7f7af81299f0 2024-04-27T03:17:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787ef6a640 1 Processor -- start 2024-04-27T03:17:40.970 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787ef6a640 1 -- start start 2024-04-27T03:17:40.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- 172.21.15.77:0/3316318682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af81299f0 msgr2=0x7f7af80707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 --2- 172.21.15.77:0/3316318682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af81299f0 0x7f7af80707b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009740 tx=0x7f7aec01c370 comp rx=0 tx=0).stop 2024-04-27T03:17:40.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- 172.21.15.77:0/3316318682 shutdown_connections 2024-04-27T03:17:40.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 --2- 172.21.15.77:0/3316318682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af81299f0 0x7f7af80707b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- 172.21.15.77:0/3316318682 >> 172.21.15.77:0/3316318682 conn(0x7f7af809ca00 msgr2=0x7f7af809ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- 172.21.15.77:0/3316318682 shutdown_connections 2024-04-27T03:17:40.971 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- 172.21.15.77:0/3316318682 wait complete. 2024-04-27T03:17:40.972 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787ef6a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f787812ca20 0x7f7878131460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787ef6a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f787809ab20 con 0x7f787812bf30 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787ef6a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f78780718b0 con 0x7f787812ca20 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787cd06640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f787812bf30 0x7f787812c310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39306/0 (socket says 172.21.15.77:39306) 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787cd06640 1 -- 172.21.15.77:0/3675368248 learned_addr learned my addr 172.21.15.77:0/3675368248 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457520252 0 0) 0x7f787809ab20 con 0x7f787812bf30 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7868003530 con 0x7f787812bf30 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1515236130 0 0) 0x7f7868003530 con 0x7f787812bf30 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 >> v2:172.21.15.77:3300/0 conn(0x7f787812ca20 msgr2=0x7f7878131460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 --2- 172.21.15.77:0/3675368248 >> v2:172.21.15.77:3300/0 conn(0x7f787812ca20 0x7f7878131460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 --> v1:172.21.15.77:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7878072020 con 0x7f787812bf30 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 <== mon.0 v1:172.21.15.77:6789/0 3 ==== mon_map magic: 0 ==== 205+0+0 (unknown 201508238 0 0) 0x7f786c0015b0 con 0x7f787812bf30 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 >> v1:172.21.15.77:6789/0 conn(0x7f787812bf30 legacy=0x7f787812c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 --2- 172.21.15.77:0/3675368248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f7868003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7868004610 con 0x7f78680039c0 2024-04-27T03:17:40.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787cd06640 1 --2- 172.21.15.77:0/3675368248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f7868003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 Processor -- start 2024-04-27T03:17:40.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- start start 2024-04-27T03:17:40.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f787cd06640 1 -- 172.21.15.77:0/3675368248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7868003530 con 0x7f78680039c0 2024-04-27T03:17:40.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8129610 0x7f7af806de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7af8070e70 con 0x7f7af8129610 2024-04-27T03:17:40.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8129610 0x7f7af806de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af6ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8129610 0x7f7af806de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55090/0 (socket says 172.21.15.77:55090) 2024-04-27T03:17:40.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af6ffd640 1 -- 172.21.15.77:0/3656750821 learned_addr learned my addr 172.21.15.77:0/3656750821 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af6ffd640 1 -- 172.21.15.77:0/3656750821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af811acc0 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7af6ffd640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8129610 0x7f7af806de90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800c480 tx=0x7f7ae800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7adf7fe640 1 -- 172.21.15.77:0/3656750821 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae8017070 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7adf7fe640 1 -- 172.21.15.77:0/3656750821 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae8005c20 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af8128a70 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7adf7fe640 1 -- 172.21.15.77:0/3656750821 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7ae8007070 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.962+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af811b790 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.966+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af806f970 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.966+0000 7f7adf7fe640 1 -- 172.21.15.77:0/3656750821 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f7ae801a070 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.966+0000 7f7adf7fe640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ae4035d20 0x7f7ae40381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.966+0000 7f7adf7fe640 1 -- 172.21.15.77:0/3656750821 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f7ae8047830 con 0x7f7af8129610 2024-04-27T03:17:40.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.966+0000 7f7af67fc640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ae4035d20 0x7f7ae40381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.966+0000 7f787cd06640 1 --2- 172.21.15.77:0/3675368248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f7868003dc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f786c00c190 tx=0x7f786c01cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45efad3a5cee0f19 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.966+0000 7f7af67fc640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ae4035d20 0x7f7ae40381e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0ad000 tx=0x7f7aec0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f786c007cf0 con 0x7f78680039c0 2024-04-27T03:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786c0a9070 con 0x7f78680039c0 2024-04-27T03:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f78777fe640 1 -- 172.21.15.77:0/3675368248 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f786c005b80 con 0x7f78680039c0 2024-04-27T03:17:40.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f7adf7fe640 1 -- 172.21.15.77:0/3656750821 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7ae8007210 con 0x7f7af8129610 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 -- 172.21.15.77:0/3675368248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 msgr2=0x7f7868003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 --2- 172.21.15.77:0/3675368248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f7868003dc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f786c00c190 tx=0x7f786c01cce0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 -- 172.21.15.77:0/3675368248 shutdown_connections 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 --2- 172.21.15.77:0/3675368248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f7868003dc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 --2- 172.21.15.77:0/3675368248 >> v2:172.21.15.77:3300/0 conn(0x7f787812ca20 0x7f7878131460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 -- 172.21.15.77:0/3675368248 >> 172.21.15.77:0/3675368248 conn(0x7f787809b590 msgr2=0x7f787809b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 -- 172.21.15.77:0/3675368248 shutdown_connections 2024-04-27T03:17:40.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 -- 172.21.15.77:0/3675368248 wait complete. 2024-04-27T03:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 Processor -- start 2024-04-27T03:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 -- start start 2024-04-27T03:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f78781c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.970+0000 7f787ef6a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f7878134550 con 0x7f78680039c0 2024-04-27T03:17:40.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787cd06640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f78781c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787cd06640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f78781c3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55104/0 (socket says 172.21.15.77:55104) 2024-04-27T03:17:40.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787cd06640 1 -- 172.21.15.77:0/1988994061 learned_addr learned my addr 172.21.15.77:0/1988994061 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:40.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787cd06640 1 -- 172.21.15.77:0/1988994061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7878117570 con 0x7f78680039c0 2024-04-27T03:17:40.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787cd06640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f78781c3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f786c0198f0 tx=0x7f7868005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f78757fa640 1 -- 172.21.15.77:0/1988994061 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f786c007cf0 con 0x7f78680039c0 2024-04-27T03:17:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f78757fa640 1 -- 172.21.15.77:0/1988994061 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786c0b4070 con 0x7f78680039c0 2024-04-27T03:17:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7878117330 con 0x7f78680039c0 2024-04-27T03:17:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f78757fa640 1 -- 172.21.15.77:0/1988994061 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f786c0047f0 con 0x7f78680039c0 2024-04-27T03:17:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78781c3b40 con 0x7f78680039c0 2024-04-27T03:17:40.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78781182a0 con 0x7f78680039c0 2024-04-27T03:17:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f78757fa640 1 -- 172.21.15.77:0/1988994061 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f786c0b6070 con 0x7f78680039c0 2024-04-27T03:17:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f78757fa640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f785c035d20 0x7f785c0381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f78757fa640 1 -- 172.21.15.77:0/1988994061 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f786c0e1e60 con 0x7f78680039c0 2024-04-27T03:17:40.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.974+0000 7f7877fff640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f785c035d20 0x7f785c0381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:40.985 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.978+0000 7f7877fff640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f785c035d20 0x7f785c0381e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7864009730 tx=0x7f7864006cb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:40.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:40.986+0000 7f78757fa640 1 -- 172.21.15.77:0/1988994061 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f786c0168d0 con 0x7f78680039c0 2024-04-27T03:17:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.074+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f921011bee0 con 0x7f9210070810 2024-04-27T03:17:41.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9206ffd640 1 -- 172.21.15.77:0/1839409548 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f920c007210 con 0x7f9210070810 2024-04-27T03:17:41.085 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2024-04-27T03:17:41.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91f4035d20 msgr2=0x7f91f40381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91f4035d20 0x7f91f40381e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9200009740 tx=0x7f9200008040 comp rx=0 tx=0).stop 2024-04-27T03:17:41.088 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210070810 msgr2=0x7f921011ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210070810 0x7f921011ace0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f920c00c480 tx=0x7f920c00c940 comp rx=0 tx=0).stop 2024-04-27T03:17:41.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 shutdown_connections 2024-04-27T03:17:41.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91f4035d20 0x7f91f40381e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 --2- 172.21.15.77:0/1839409548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9210070810 0x7f921011ace0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 >> 172.21.15.77:0/1839409548 conn(0x7f921009da80 msgr2=0x7f921006f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:41.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 shutdown_connections 2024-04-27T03:17:41.089 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.078+0000 7f9218457640 1 -- 172.21.15.77:0/1839409548 wait complete. 2024-04-27T03:17:41.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.086+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6f8c11bfa0 con 0x7f6f8c06f2a0 2024-04-27T03:17:41.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.086+0000 7f6f827fc640 1 -- 172.21.15.77:0/2143783095 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6f88015ff0 con 0x7f6f8c06f2a0 2024-04-27T03:17:41.095 INFO:teuthology.orchestra.run.smithi077.stdout:38654705666 2024-04-27T03:17:41.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6f70035cb0 msgr2=0x7f6f70038170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6f70035cb0 0x7f6f70038170 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c001610 tx=0x7f6f7c007ad0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 msgr2=0x7f6f8c11ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c11ada0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6f880ae040 tx=0x7f6f88019110 comp rx=0 tx=0).stop 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 shutdown_connections 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6f70035cb0 0x7f6f70038170 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 --2- 172.21.15.77:0/2143783095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f8c06f2a0 0x7f6f8c11ada0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 >> 172.21.15.77:0/2143783095 conn(0x7f6f8c09e0d0 msgr2=0x7f6f8c1314e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 shutdown_connections 2024-04-27T03:17:41.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.090+0000 7f6f93b58640 1 -- 172.21.15.77:0/2143783095 wait complete. 2024-04-27T03:17:41.104 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.3 2024-04-27T03:17:41.119 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.1 2024-04-27T03:17:41.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7af8062e50 con 0x7f7af8129610 2024-04-27T03:17:41.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7adf7fe640 1 -- 172.21.15.77:0/3656750821 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7ae801b400 con 0x7f7af8129610 2024-04-27T03:17:41.146 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2024-04-27T03:17:41.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ae4035d20 msgr2=0x7f7ae40381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ae4035d20 0x7f7ae40381e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0ad000 tx=0x7f7aec0a6000 comp rx=0 tx=0).stop 2024-04-27T03:17:41.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8129610 msgr2=0x7f7af806de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8129610 0x7f7af806de90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800c480 tx=0x7f7ae800c940 comp rx=0 tx=0).stop 2024-04-27T03:17:41.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 shutdown_connections 2024-04-27T03:17:41.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ae4035d20 0x7f7ae40381e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 --2- 172.21.15.77:0/3656750821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7af8129610 0x7f7af806de90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 >> 172.21.15.77:0/3656750821 conn(0x7f7af809ca00 msgr2=0x7f7af80993a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:41.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 shutdown_connections 2024-04-27T03:17:41.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.138+0000 7f7afd781640 1 -- 172.21.15.77:0/3656750821 wait complete. 2024-04-27T03:17:41.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.146+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7878062e50 con 0x7f78680039c0 2024-04-27T03:17:41.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.146+0000 7f78757fa640 1 -- 172.21.15.77:0/1988994061 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f786c0ad3d0 con 0x7f78680039c0 2024-04-27T03:17:41.157 INFO:teuthology.orchestra.run.smithi077.stdout:38654705666 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f785c035d20 msgr2=0x7f785c0381e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f785c035d20 0x7f785c0381e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7864009730 tx=0x7f7864006cb0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 msgr2=0x7f78781c3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f78781c3600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f786c0198f0 tx=0x7f7868005840 comp rx=0 tx=0).stop 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 shutdown_connections 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f785c035d20 0x7f785c0381e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 --2- 172.21.15.77:0/1988994061 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f78680039c0 0x7f78781c3600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 >> 172.21.15.77:0/1988994061 conn(0x7f787809b590 msgr2=0x7f7878129150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 shutdown_connections 2024-04-27T03:17:41.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:41.150+0000 7f787ef6a640 1 -- 172.21.15.77:0/1988994061 wait complete. 2024-04-27T03:17:41.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.0 2024-04-27T03:17:41.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.2 2024-04-27T03:17:42.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-27T03:17:42.119 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-27T03:17:42.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-27T03:17:42.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f32fde640 1 Processor -- start 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f32fde640 1 -- start start 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f32fde640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f2c12f2a0 0x7f9f2c131690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f32fde640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f2c12bfd0 con 0x7f9f2c12c400 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f32fde640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9f2c12cef0 con 0x7f9f2c12f2a0 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f2bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f2c12f2a0 0x7f9f2c131690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f30d7a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9f2c12c400 0x7f9f2c12c7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39316/0 (socket says 172.21.15.77:39316) 2024-04-27T03:17:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f30d7a640 1 -- 172.21.15.77:0/2588981822 learned_addr learned my addr 172.21.15.77:0/2588981822 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:42.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f2bfff640 1 -- 172.21.15.77:0/2588981822 >> v1:172.21.15.77:6789/0 conn(0x7f9f2c12c400 legacy=0x7f9f2c12c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.222+0000 7f9f2bfff640 1 -- 172.21.15.77:0/2588981822 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2c09ac80 con 0x7f9f2c12f2a0 2024-04-27T03:17:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f2bfff640 1 --2- 172.21.15.77:0/2588981822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12f2a0 0x7f9f2c131690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009740 tx=0x7f9f1801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15713ad9a3d8a704 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f2b7fe640 1 -- 172.21.15.77:0/2588981822 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f180b0070 con 0x7f9f2c12f2a0 2024-04-27T03:17:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f2b7fe640 1 -- 172.21.15.77:0/2588981822 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f180ab530 con 0x7f9f2c12f2a0 2024-04-27T03:17:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f2b7fe640 1 -- 172.21.15.77:0/2588981822 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f180ab830 con 0x7f9f2c12f2a0 2024-04-27T03:17:42.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f32fde640 1 -- 172.21.15.77:0/2588981822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12f2a0 msgr2=0x7f9f2c131690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f32fde640 1 --2- 172.21.15.77:0/2588981822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12f2a0 0x7f9f2c131690 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009740 tx=0x7f9f1801c370 comp rx=0 tx=0).stop 2024-04-27T03:17:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f32fde640 1 -- 172.21.15.77:0/2588981822 shutdown_connections 2024-04-27T03:17:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f32fde640 1 --2- 172.21.15.77:0/2588981822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12f2a0 0x7f9f2c131690 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f32fde640 1 -- 172.21.15.77:0/2588981822 >> 172.21.15.77:0/2588981822 conn(0x7f9f2c09c320 msgr2=0x7f9f2c09c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f32fde640 1 -- 172.21.15.77:0/2588981822 shutdown_connections 2024-04-27T03:17:42.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.226+0000 7f9f32fde640 1 -- 172.21.15.77:0/2588981822 wait complete. 2024-04-27T03:17:42.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f32fde640 1 Processor -- start 2024-04-27T03:17:42.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f32fde640 1 -- start start 2024-04-27T03:17:42.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f32fde640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12c400 0x7f9f2c1278d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f32fde640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c134780 con 0x7f9f2c12c400 2024-04-27T03:17:42.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f30d7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12c400 0x7f9f2c1278d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f30d7a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12c400 0x7f9f2c1278d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55124/0 (socket says 172.21.15.77:55124) 2024-04-27T03:17:42.239 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f30d7a640 1 -- 172.21.15.77:0/506295538 learned_addr learned my addr 172.21.15.77:0/506295538 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f30d7a640 1 -- 172.21.15.77:0/506295538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2c099f30 con 0x7f9f2c12c400 2024-04-27T03:17:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f30d7a640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12c400 0x7f9f2c1278d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c480 tx=0x7f9f2000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f297fa640 1 -- 172.21.15.77:0/506295538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f20017070 con 0x7f9f2c12c400 2024-04-27T03:17:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f297fa640 1 -- 172.21.15.77:0/506295538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f20005c20 con 0x7f9f2c12c400 2024-04-27T03:17:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f297fa640 1 -- 172.21.15.77:0/506295538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f20007070 con 0x7f9f2c12c400 2024-04-27T03:17:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2c1149f0 con 0x7f9f2c12c400 2024-04-27T03:17:42.240 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.230+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f2c127e10 con 0x7f9f2c12c400 2024-04-27T03:17:42.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.234+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f2c115160 con 0x7f9f2c12c400 2024-04-27T03:17:42.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.234+0000 7f9f297fa640 1 -- 172.21.15.77:0/506295538 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9f2001a070 con 0x7f9f2c12c400 2024-04-27T03:17:42.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.234+0000 7f9f297fa640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f10035db0 0x7f9f10038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.241 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.234+0000 7f9f297fa640 1 -- 172.21.15.77:0/506295538 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f9f20047950 con 0x7f9f2c12c400 2024-04-27T03:17:42.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.234+0000 7f9f2bfff640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f10035db0 0x7f9f10038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.242 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.234+0000 7f9f2bfff640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f10035db0 0x7f9f10038270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9f180ad000 tx=0x7f9f180a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.242+0000 7f9f297fa640 1 -- 172.21.15.77:0/506295538 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9f2c115160 con 0x7f9f2c12c400 2024-04-27T03:17:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b7a5be640 1 Processor -- start 2024-04-27T03:17:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b7a5be640 1 -- start start 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b7a5be640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b74094b90 0x7f5b74133f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b7a5be640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b74096200 con 0x7f5b740939d0 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b7a5be640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5b74096510 con 0x7f5b74094b90 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b737fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5b74094b90 0x7f5b74133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b73fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f5b740939d0 0x7f5b74094480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39332/0 (socket says 172.21.15.77:39332) 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b73fff640 1 -- 172.21.15.77:0/2958688606 learned_addr learned my addr 172.21.15.77:0/2958688606 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b737fe640 1 -- 172.21.15.77:0/2958688606 >> v1:172.21.15.77:6789/0 conn(0x7f5b740939d0 legacy=0x7f5b74094480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.258+0000 7f5b737fe640 1 -- 172.21.15.77:0/2958688606 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b74093db0 con 0x7f5b74094b90 2024-04-27T03:17:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b737fe640 1 --2- 172.21.15.77:0/2958688606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b74094b90 0x7f5b74133f10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009740 tx=0x7f5b6801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49815d80f23f5c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b72ffd640 1 -- 172.21.15.77:0/2958688606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b680ab550 con 0x7f5b74094b90 2024-04-27T03:17:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b72ffd640 1 -- 172.21.15.77:0/2958688606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b680b0070 con 0x7f5b74094b90 2024-04-27T03:17:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b72ffd640 1 -- 172.21.15.77:0/2958688606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b680ab840 con 0x7f5b74094b90 2024-04-27T03:17:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 -- 172.21.15.77:0/2958688606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b74094b90 msgr2=0x7f5b74133f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 --2- 172.21.15.77:0/2958688606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b74094b90 0x7f5b74133f10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009740 tx=0x7f5b6801c400 comp rx=0 tx=0).stop 2024-04-27T03:17:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 -- 172.21.15.77:0/2958688606 shutdown_connections 2024-04-27T03:17:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 --2- 172.21.15.77:0/2958688606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b74094b90 0x7f5b74133f10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 -- 172.21.15.77:0/2958688606 >> 172.21.15.77:0/2958688606 conn(0x7f5b7409d150 msgr2=0x7f5b7409d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 -- 172.21.15.77:0/2958688606 shutdown_connections 2024-04-27T03:17:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 -- 172.21.15.77:0/2958688606 wait complete. 2024-04-27T03:17:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 Processor -- start 2024-04-27T03:17:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 -- start start 2024-04-27T03:17:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b740939d0 0x7f5b74121e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b7a5be640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5b74136e80 con 0x7f5b740939d0 2024-04-27T03:17:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b73fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b740939d0 0x7f5b74121e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b73fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b740939d0 0x7f5b74121e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55136/0 (socket says 172.21.15.77:55136) 2024-04-27T03:17:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b73fff640 1 -- 172.21.15.77:0/3984334867 learned_addr learned my addr 172.21.15.77:0/3984334867 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.262+0000 7f5b73fff640 1 -- 172.21.15.77:0/3984334867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b74123bd0 con 0x7f5b740939d0 2024-04-27T03:17:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b73fff640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b740939d0 0x7f5b74121e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400c480 tx=0x7f5b6400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b70ff9640 1 -- 172.21.15.77:0/3984334867 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b64017070 con 0x7f5b740939d0 2024-04-27T03:17:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b70ff9640 1 -- 172.21.15.77:0/3984334867 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b64005c20 con 0x7f5b740939d0 2024-04-27T03:17:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b74072a30 con 0x7f5b740939d0 2024-04-27T03:17:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b74124860 con 0x7f5b740939d0 2024-04-27T03:17:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b70ff9640 1 -- 172.21.15.77:0/3984334867 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b64007070 con 0x7f5b740939d0 2024-04-27T03:17:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b54005090 con 0x7f5b740939d0 2024-04-27T03:17:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b70ff9640 1 -- 172.21.15.77:0/3984334867 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5b6401a070 con 0x7f5b740939d0 2024-04-27T03:17:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b70ff9640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5b60035de0 0x7f5b600382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b70ff9640 1 -- 172.21.15.77:0/3984334867 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f5b640479d0 con 0x7f5b740939d0 2024-04-27T03:17:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b737fe640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5b60035de0 0x7f5b600382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.266+0000 7f5b737fe640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5b60035de0 0x7f5b600382a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5b680af000 tx=0x7f5b680a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.284 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.274+0000 7f5b70ff9640 1 -- 172.21.15.77:0/3984334867 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5b74096200 con 0x7f5b740939d0 2024-04-27T03:17:42.305 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3f19aa640 1 Processor -- start 2024-04-27T03:17:42.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3f19aa640 1 -- start start 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3f19aa640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3ec06fd90 0x7fb3ec133700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3f19aa640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ec133c40 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3f19aa640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ec070cf0 con 0x7fb3ec06fd90 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3ea7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3ec06fd90 0x7fb3ec133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3ea7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fb3ec06fd90 0x7fb3ec133700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55140/0 (socket says 172.21.15.77:55140) 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3eaffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fb3ec06f2a0 0x7fb3ec06f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39340/0 (socket says 172.21.15.77:39340) 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3ea7fc640 1 -- 172.21.15.77:0/186478170 learned_addr learned my addr 172.21.15.77:0/186478170 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3ea7fc640 1 -- 172.21.15.77:0/186478170 >> v1:172.21.15.77:6789/0 conn(0x7fb3ec06f2a0 legacy=0x7fb3ec06f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3ea7fc640 1 -- 172.21.15.77:0/186478170 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec134e20 con 0x7fb3ec06fd90 2024-04-27T03:17:42.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3ea7fc640 1 --2- 172.21.15.77:0/186478170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06fd90 0x7fb3ec133700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00097f0 tx=0x7fb3e001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c09e6f26c0b9a7b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3e9ffb640 1 -- 172.21.15.77:0/186478170 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3e00b0070 con 0x7fb3ec06fd90 2024-04-27T03:17:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3e9ffb640 1 -- 172.21.15.77:0/186478170 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e00ab530 con 0x7fb3ec06fd90 2024-04-27T03:17:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.298+0000 7fb3e9ffb640 1 -- 172.21.15.77:0/186478170 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3e00ab830 con 0x7fb3ec06fd90 2024-04-27T03:17:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/186478170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06fd90 msgr2=0x7fb3ec133700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 --2- 172.21.15.77:0/186478170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06fd90 0x7fb3ec133700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00097f0 tx=0x7fb3e001c430 comp rx=0 tx=0).stop 2024-04-27T03:17:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/186478170 shutdown_connections 2024-04-27T03:17:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 --2- 172.21.15.77:0/186478170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06fd90 0x7fb3ec133700 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3e00097f0 tx=0x7fb3e001c430 comp rx=0 tx=0).stop 2024-04-27T03:17:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/186478170 >> 172.21.15.77:0/186478170 conn(0x7fb3ec09e0d0 msgr2=0x7fb3ec06d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/186478170 shutdown_connections 2024-04-27T03:17:42.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/186478170 wait complete. 2024-04-27T03:17:42.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 Processor -- start 2024-04-27T03:17:42.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- start start 2024-04-27T03:17:42.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06f2a0 0x7fb3ec1c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec1349f0 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3eaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06f2a0 0x7fb3ec1c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3eaffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06f2a0 0x7fb3ec1c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55146/0 (socket says 172.21.15.77:55146) 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3eaffd640 1 -- 172.21.15.77:0/865526920 learned_addr learned my addr 172.21.15.77:0/865526920 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3eaffd640 1 -- 172.21.15.77:0/865526920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec11c720 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3eaffd640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06f2a0 0x7fb3ec1c6010 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c410 tx=0x7fb3dc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3d3fff640 1 -- 172.21.15.77:0/865526920 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3dc007410 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ec11df60 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ec1c6720 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3d3fff640 1 -- 172.21.15.77:0/865526920 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3dc0075b0 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3d3fff640 1 -- 172.21.15.77:0/865526920 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3dc004920 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3cc005090 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3d3fff640 1 -- 172.21.15.77:0/865526920 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb3dc01a020 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.302+0000 7fb3d3fff640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb3d8035d60 0x7fb3d8038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.306+0000 7fb3d3fff640 1 -- 172.21.15.77:0/865526920 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7fb3dc047e70 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.306+0000 7fb3ea7fc640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb3d8035d60 0x7fb3d8038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.306+0000 7fb3ea7fc640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb3d8035d60 0x7fb3d8038220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec06d040 tx=0x7fb3e00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.318 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fb3d3fff640 1 -- 172.21.15.77:0/865526920 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb3dc004d50 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77f4f2640 1 Processor -- start 2024-04-27T03:17:42.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77f4f2640 1 -- start start 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77f4f2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd77812e4d0 0x7fd7781308c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77f4f2640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd778071e90 con 0x7fd77812b1c0 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77ca8d640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fd77812e4d0 0x7fd7781308c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77d28e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fd77812b1c0 0x7fd77812b870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39346/0 (socket says 172.21.15.77:39346) 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77d28e640 1 -- 172.21.15.77:0/387956586 learned_addr learned my addr 172.21.15.77:0/387956586 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77f4f2640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fd778071b50 con 0x7fd77812e4d0 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd76ffff640 1 -- 172.21.15.77:0/387956586 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516781473 0 0) 0x7fd778071e90 con 0x7fd77812b1c0 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd76ffff640 1 -- 172.21.15.77:0/387956586 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd764003530 con 0x7fd77812b1c0 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77ca8d640 1 -- 172.21.15.77:0/387956586 >> v1:172.21.15.77:6789/0 conn(0x7fd77812b1c0 legacy=0x7fd77812b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.310+0000 7fd77ca8d640 1 -- 172.21.15.77:0/387956586 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77812bf80 con 0x7fd77812e4d0 2024-04-27T03:17:42.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77ca8d640 1 --2- 172.21.15.77:0/387956586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812e4d0 0x7fd7781308c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd768009740 tx=0x7fd76801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384b24183931b5a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd76ffff640 1 -- 172.21.15.77:0/387956586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7680b0070 con 0x7fd77812e4d0 2024-04-27T03:17:42.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd76ffff640 1 -- 172.21.15.77:0/387956586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7680ab530 con 0x7fd77812e4d0 2024-04-27T03:17:42.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd76ffff640 1 -- 172.21.15.77:0/387956586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7680ab830 con 0x7fd77812e4d0 2024-04-27T03:17:42.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 -- 172.21.15.77:0/387956586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812e4d0 msgr2=0x7fd7781308c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 --2- 172.21.15.77:0/387956586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812e4d0 0x7fd7781308c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd768009740 tx=0x7fd76801c370 comp rx=0 tx=0).stop 2024-04-27T03:17:42.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 -- 172.21.15.77:0/387956586 shutdown_connections 2024-04-27T03:17:42.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 --2- 172.21.15.77:0/387956586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812e4d0 0x7fd7781308c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.324 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 -- 172.21.15.77:0/387956586 >> 172.21.15.77:0/387956586 conn(0x7fd77809d850 msgr2=0x7fd77809dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 -- 172.21.15.77:0/387956586 shutdown_connections 2024-04-27T03:17:42.325 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 -- 172.21.15.77:0/387956586 wait complete. 2024-04-27T03:17:42.326 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.314+0000 7fd77f4f2640 1 Processor -- start 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77f4f2640 1 -- start start 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77f4f2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812b1c0 0x7fd77812c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77f4f2640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fd778133830 con 0x7fd77812b1c0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77d28e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812b1c0 0x7fd77812c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77d28e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812b1c0 0x7fd77812c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55176/0 (socket says 172.21.15.77:55176) 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77d28e640 1 -- 172.21.15.77:0/3020671547 learned_addr learned my addr 172.21.15.77:0/3020671547 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77d28e640 1 -- 172.21.15.77:0/3020671547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77812de10 con 0x7fd77812b1c0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77d28e640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812b1c0 0x7fd77812c290 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c3a0 tx=0x7fd77400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd76dffb640 1 -- 172.21.15.77:0/3020671547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd774017070 con 0x7fd77812b1c0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd76dffb640 1 -- 172.21.15.77:0/3020671547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd774005c20 con 0x7fd77812b1c0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77811d1b0 con 0x7fd77812b1c0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd76dffb640 1 -- 172.21.15.77:0/3020671547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7740070d0 con 0x7fd77812b1c0 2024-04-27T03:17:42.327 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77812c7d0 con 0x7fd77812b1c0 2024-04-27T03:17:42.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd778062e50 con 0x7fd77812b1c0 2024-04-27T03:17:42.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd76dffb640 1 -- 172.21.15.77:0/3020671547 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd77401a070 con 0x7fd77812b1c0 2024-04-27T03:17:42.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd76dffb640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd75c035de0 0x7fd75c0382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd76dffb640 1 -- 172.21.15.77:0/3020671547 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7fd774047a00 con 0x7fd77812b1c0 2024-04-27T03:17:42.328 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.318+0000 7fd77ca8d640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd75c035de0 0x7fd75c0382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.322+0000 7fd77ca8d640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd75c035de0 0x7fd75c0382a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd7680ad000 tx=0x7fd768008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.326+0000 7fd76dffb640 1 -- 172.21.15.77:0/3020671547 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd774019e10 con 0x7fd77812b1c0 2024-04-27T03:17:42.412 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.402+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9f2c062e50 con 0x7f9f2c12c400 2024-04-27T03:17:42.413 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.402+0000 7f9f297fa640 1 -- 172.21.15.77:0/506295538 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9f20007210 con 0x7f9f2c12c400 2024-04-27T03:17:42.413 INFO:teuthology.orchestra.run.smithi077.stdout:38654705668 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f10035db0 msgr2=0x7f9f10038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f10035db0 0x7f9f10038270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9f180ad000 tx=0x7f9f180a6000 comp rx=0 tx=0).stop 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12c400 msgr2=0x7f9f2c1278d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12c400 0x7f9f2c1278d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c480 tx=0x7f9f2000c940 comp rx=0 tx=0).stop 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 shutdown_connections 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f10035db0 0x7f9f10038270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 --2- 172.21.15.77:0/506295538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f2c12c400 0x7f9f2c1278d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 >> 172.21.15.77:0/506295538 conn(0x7f9f2c09c320 msgr2=0x7f9f2c12aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 shutdown_connections 2024-04-27T03:17:42.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.406+0000 7f9f32fde640 1 -- 172.21.15.77:0/506295538 wait complete. 2024-04-27T03:17:42.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.3 2024-04-27T03:17:42.430 DEBUG:teuthology.parallel:result is None 2024-04-27T03:17:42.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.470+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb3cc005380 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.470+0000 7fb3d3fff640 1 -- 172.21.15.77:0/865526920 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb3dc013920 con 0x7fb3ec06f2a0 2024-04-27T03:17:42.481 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb3d8035d60 msgr2=0x7fb3d8038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb3d8035d60 0x7fb3d8038220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec06d040 tx=0x7fb3e00a6000 comp rx=0 tx=0).stop 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06f2a0 msgr2=0x7fb3ec1c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06f2a0 0x7fb3ec1c6010 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc00c410 tx=0x7fb3dc00c8d0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 shutdown_connections 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb3d8035d60 0x7fb3d8038220 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 --2- 172.21.15.77:0/865526920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb3ec06f2a0 0x7fb3ec1c6010 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 >> 172.21.15.77:0/865526920 conn(0x7fb3ec09e0d0 msgr2=0x7fb3ec12ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 shutdown_connections 2024-04-27T03:17:42.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.474+0000 7fb3f19aa640 1 -- 172.21.15.77:0/865526920 wait complete. 2024-04-27T03:17:42.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.486+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5b54005380 con 0x7f5b740939d0 2024-04-27T03:17:42.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.486+0000 7f5b70ff9640 1 -- 172.21.15.77:0/3984334867 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5b64007210 con 0x7f5b740939d0 2024-04-27T03:17:42.495 INFO:teuthology.orchestra.run.smithi077.stdout:38654705667 2024-04-27T03:17:42.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.486+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5b60035de0 msgr2=0x7f5b600382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.486+0000 7f5b7a5be640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5b60035de0 0x7f5b600382a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5b680af000 tx=0x7f5b680a7000 comp rx=0 tx=0).stop 2024-04-27T03:17:42.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b740939d0 msgr2=0x7f5b74121e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b740939d0 0x7f5b74121e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5b6400c480 tx=0x7f5b6400c940 comp rx=0 tx=0).stop 2024-04-27T03:17:42.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 shutdown_connections 2024-04-27T03:17:42.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5b60035de0 0x7f5b600382a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 --2- 172.21.15.77:0/3984334867 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b740939d0 0x7f5b74121e10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 >> 172.21.15.77:0/3984334867 conn(0x7f5b7409d150 msgr2=0x7f5b740714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 shutdown_connections 2024-04-27T03:17:42.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.490+0000 7f5b7a5be640 1 -- 172.21.15.77:0/3984334867 wait complete. 2024-04-27T03:17:42.504 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2024-04-27T03:17:42.504 DEBUG:teuthology.parallel:result is None 2024-04-27T03:17:42.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.1 2024-04-27T03:17:42.512 DEBUG:teuthology.parallel:result is None 2024-04-27T03:17:42.542 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.534+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd77806c220 con 0x7fd77812b1c0 2024-04-27T03:17:42.543 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.534+0000 7fd76dffb640 1 -- 172.21.15.77:0/3020671547 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd774010e00 con 0x7fd77812b1c0 2024-04-27T03:17:42.543 INFO:teuthology.orchestra.run.smithi077.stdout:38654705668 2024-04-27T03:17:42.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.534+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd75c035de0 msgr2=0x7fd75c0382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.534+0000 7fd77f4f2640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd75c035de0 0x7fd75c0382a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd7680ad000 tx=0x7fd768008040 comp rx=0 tx=0).stop 2024-04-27T03:17:42.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.534+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812b1c0 msgr2=0x7fd77812c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.534+0000 7fd77f4f2640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812b1c0 0x7fd77812c290 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd77400c3a0 tx=0x7fd77400c860 comp rx=0 tx=0).stop 2024-04-27T03:17:42.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.534+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 shutdown_connections 2024-04-27T03:17:42.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.538+0000 7fd77f4f2640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd75c035de0 0x7fd75c0382a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.538+0000 7fd77f4f2640 1 --2- 172.21.15.77:0/3020671547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd77812b1c0 0x7fd77812c290 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.538+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 >> 172.21.15.77:0/3020671547 conn(0x7fd77809d850 msgr2=0x7fd77806e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.538+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 shutdown_connections 2024-04-27T03:17:42.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.538+0000 7fd77f4f2640 1 -- 172.21.15.77:0/3020671547 wait complete. 2024-04-27T03:17:42.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2024-04-27T03:17:42.560 DEBUG:teuthology.parallel:result is None 2024-04-27T03:17:42.560 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-27T03:17:42.560 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:17:42.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e73e3640 1 Processor -- start 2024-04-27T03:17:42.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e73e3640 1 -- start start 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e73e3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30e00963b0 0x7f30e0096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e73e3640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30e0098d10 con 0x7f30e006d4d0 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e73e3640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f30e0098610 con 0x7f30e00963b0 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e517f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30e00963b0 0x7f30e0096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e517f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f30e00963b0 0x7f30e0096790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55182/0 (socket says 172.21.15.77:55182) 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e517f640 1 -- 172.21.15.77:0/3440508292 learned_addr learned my addr 172.21.15.77:0/3440508292 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e497e640 1 --1- 172.21.15.77:0/3440508292 >> v1:172.21.15.77:6789/0 conn(0x7f30e006d4d0 0x7f30e006d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39362/0 (socket says 172.21.15.77:39362) 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e517f640 1 -- 172.21.15.77:0/3440508292 >> v1:172.21.15.77:6789/0 conn(0x7f30e006d4d0 legacy=0x7f30e006d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30e517f640 1 -- 172.21.15.77:0/3440508292 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e0071f90 con 0x7f30e00963b0 2024-04-27T03:17:42.709 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.698+0000 7f30d7fff640 1 -- 172.21.15.77:0/3440508292 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 581364909 0 0) 0x7f30e0098d10 con 0x7f30e006d4d0 2024-04-27T03:17:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e517f640 1 --2- 172.21.15.77:0/3440508292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e00963b0 0x7f30e0096790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f30dc0097f0 tx=0x7f30dc01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31221e68a96cf967 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30d7fff640 1 -- 172.21.15.77:0/3440508292 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30dc0b0070 con 0x7f30e00963b0 2024-04-27T03:17:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30d7fff640 1 -- 172.21.15.77:0/3440508292 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30dc0ab530 con 0x7f30e00963b0 2024-04-27T03:17:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30d7fff640 1 -- 172.21.15.77:0/3440508292 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30dc0ab830 con 0x7f30e00963b0 2024-04-27T03:17:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- 172.21.15.77:0/3440508292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e00963b0 msgr2=0x7f30e0096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 --2- 172.21.15.77:0/3440508292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e00963b0 0x7f30e0096790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f30dc0097f0 tx=0x7f30dc01c4c0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- 172.21.15.77:0/3440508292 shutdown_connections 2024-04-27T03:17:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 --2- 172.21.15.77:0/3440508292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e00963b0 0x7f30e0096790 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- 172.21.15.77:0/3440508292 >> 172.21.15.77:0/3440508292 conn(0x7f30e0099440 msgr2=0x7f30e0099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- 172.21.15.77:0/3440508292 shutdown_connections 2024-04-27T03:17:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- 172.21.15.77:0/3440508292 wait complete. 2024-04-27T03:17:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 Processor -- start 2024-04-27T03:17:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- start start 2024-04-27T03:17:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e006d4d0 0x7f30e01c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f30e0134c00 con 0x7f30e006d4d0 2024-04-27T03:17:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e517f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e006d4d0 0x7f30e01c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e517f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e006d4d0 0x7f30e01c6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55188/0 (socket says 172.21.15.77:55188) 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e517f640 1 -- 172.21.15.77:0/2056669843 learned_addr learned my addr 172.21.15.77:0/2056669843 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e517f640 1 -- 172.21.15.77:0/2056669843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e006f7f0 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e517f640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e006d4d0 0x7f30e01c6150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30dc01a0f0 tx=0x7f30dc01ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30d5ffb640 1 -- 172.21.15.77:0/2056669843 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30dc0b0040 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30d5ffb640 1 -- 172.21.15.77:0/2056669843 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30dc0abe40 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e0126290 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30d5ffb640 1 -- 172.21.15.77:0/2056669843 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30dc004150 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.702+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e008a380 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.706+0000 7f30d5ffb640 1 -- 172.21.15.77:0/2056669843 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f30dc0b3070 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.706+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e0062e50 con 0x7f30e006d4d0 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.706+0000 7f30d5ffb640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f30c4035db0 0x7f30c4038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.714 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.706+0000 7f30d5ffb640 1 -- 172.21.15.77:0/2056669843 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f30dc0e2ad0 con 0x7f30e006d4d0 2024-04-27T03:17:42.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.706+0000 7f30e497e640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f30c4035db0 0x7f30c4038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.715 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.706+0000 7f30e497e640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f30c4035db0 0x7f30c4038270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30d0000c10 tx=0x7f30d0007a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.718 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.710+0000 7f30d5ffb640 1 -- 172.21.15.77:0/2056669843 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f30dc0c3030 con 0x7f30e006d4d0 2024-04-27T03:17:42.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.854+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f30e006c220 con 0x7f30c4035db0 2024-04-27T03:17:42.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30d5ffb640 1 -- 172.21.15.77:0/2056669843 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31882 (secure 0 0 0) 0x7f30e006c220 con 0x7f30c4035db0 2024-04-27T03:17:42.867 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:42.867 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:17:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f30c4035db0 msgr2=0x7f30c4038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f30c4035db0 0x7f30c4038270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f30d0000c10 tx=0x7f30d0007a50 comp rx=0 tx=0).stop 2024-04-27T03:17:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e006d4d0 msgr2=0x7f30e01c6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e006d4d0 0x7f30e01c6150 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30dc01a0f0 tx=0x7f30dc01ad00 comp rx=0 tx=0).stop 2024-04-27T03:17:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 shutdown_connections 2024-04-27T03:17:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f30c4035db0 0x7f30c4038270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 --2- 172.21.15.77:0/2056669843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f30e006d4d0 0x7f30e01c6150 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 >> 172.21.15.77:0/2056669843 conn(0x7f30e0099440 msgr2=0x7f30e009b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 shutdown_connections 2024-04-27T03:17:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.858+0000 7f30e73e3640 1 -- 172.21.15.77:0/2056669843 wait complete. 2024-04-27T03:17:42.883 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-27T03:17:41.634367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":123352,"kb_used_data":1552,"kb_used_omap":6,"kb_used_meta":120505,"kb_avail":377364008,"statfs":{"total":386547056640,"available":386420744192,"internally_reserved":0,"allocated":1589248,"data_stored":1291738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":123397910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,12],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.635303+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:17:39.635303+0000","last_peered":"2024-04-27T03:17:39.635303+0000","last_clean":"2024-04-27T03:17:39.635303+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:17:39.635303+0000","last_undegraded":"2024-04-27T03:17:39.635303+0000","last_fullsized":"2024-04-27T03:17:39.635303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:10:58.472036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.633080+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:17:39.633080+0000","last_peered":"2024-04-27T03:17:39.633080+0000","last_clean":"2024-04-27T03:17:39.633080+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:17:39.633080+0000","last_undegraded":"2024-04-27T03:17:39.633080+0000","last_fullsized":"2024-04-27T03:17:39.633080+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:10:58.472036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.637290+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:17:39.637290+0000","last_peered":"2024-04-27T03:17:39.637290+0000","last_clean":"2024-04-27T03:17:39.637290+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:17:39.637290+0000","last_undegraded":"2024-04-27T03:17:39.637290+0000","last_fullsized":"2024-04-27T03:17:39.637290+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:38.710604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.637578+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:17:39.637578+0000","last_peered":"2024-04-27T03:17:39.637578+0000","last_clean":"2024-04-27T03:17:39.637578+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:17:39.637578+0000","last_undegraded":"2024-04-27T03:17:39.637578+0000","last_fullsized":"2024-04-27T03:17:39.637578+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.635422+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:17:39.635422+0000","last_peered":"2024-04-27T03:17:39.635422+0000","last_clean":"2024-04-27T03:17:39.635422+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:17:39.635422+0000","last_undegraded":"2024-04-27T03:17:39.635422+0000","last_fullsized":"2024-04-27T03:17:39.635422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:08.781891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.637133+0000","last_change":"2024-04-27T03:17:39.633841+0000","last_active":"2024-04-27T03:17:39.637133+0000","last_peered":"2024-04-27T03:17:39.637133+0000","last_clean":"2024-04-27T03:17:39.637133+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:17:39.637133+0000","last_undegraded":"2024-04-27T03:17:39.637133+0000","last_fullsized":"2024-04-27T03:17:39.637133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:08.781891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.638762+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:17:39.638762+0000","last_peered":"2024-04-27T03:17:39.638762+0000","last_clean":"2024-04-27T03:17:39.638762+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:17:39.638762+0000","last_undegraded":"2024-04-27T03:17:39.638762+0000","last_fullsized":"2024-04-27T03:17:39.638762+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:01:48.896188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.636188+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:17:39.636188+0000","last_peered":"2024-04-27T03:17:39.636188+0000","last_clean":"2024-04-27T03:17:39.636188+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:17:39.636188+0000","last_undegraded":"2024-04-27T03:17:39.636188+0000","last_fullsized":"2024-04-27T03:17:39.636188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":159,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:40.437110+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:17:40.437110+0000","last_peered":"2024-04-27T03:17:40.437110+0000","last_clean":"2024-04-27T03:17:40.437110+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:17:40.437110+0000","last_undegraded":"2024-04-27T03:17:40.437110+0000","last_fullsized":"2024-04-27T03:17:40.437110+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:08.781891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":106496,"data_stored":31043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":692224,"data_stored":617021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":692224,"data_stored":617021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,12],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":98304,"data_stored":26653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:17:42.884 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea19064640 1 Processor -- start 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea19064640 1 -- start start 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea19064640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fea140963b0 0x7fea14096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea19064640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea14098d10 con 0x7fea1406d4d0 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea19064640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fea14098610 con 0x7fea140963b0 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea1259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fea1406d4d0 0x7fea1406d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39364/0 (socket says 172.21.15.77:39364) 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea1259c640 1 -- 172.21.15.77:0/158124055 learned_addr learned my addr 172.21.15.77:0/158124055 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea12d9d640 1 --2- 172.21.15.77:0/158124055 >> v2:172.21.15.77:3300/0 conn(0x7fea140963b0 0x7fea14096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea11d9b640 1 -- 172.21.15.77:0/158124055 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 705539439 0 0) 0x7fea14098d10 con 0x7fea1406d4d0 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea11d9b640 1 -- 172.21.15.77:0/158124055 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9fc003530 con 0x7fea1406d4d0 2024-04-27T03:17:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea12d9d640 1 -- 172.21.15.77:0/158124055 >> v1:172.21.15.77:6789/0 conn(0x7fea1406d4d0 legacy=0x7fea1406d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea11d9b640 1 -- 172.21.15.77:0/158124055 <== mon.0 v1:172.21.15.77:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1685816251 0 0) 0x7fe9fc003530 con 0x7fea1406d4d0 2024-04-27T03:17:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.982+0000 7fea12d9d640 1 -- 172.21.15.77:0/158124055 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea14071f90 con 0x7fea140963b0 2024-04-27T03:17:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea12d9d640 1 --2- 172.21.15.77:0/158124055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea140963b0 0x7fea14096790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea08009740 tx=0x7fea0801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a9c2ab49b2713d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea11d9b640 1 -- 172.21.15.77:0/158124055 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea080b0070 con 0x7fea140963b0 2024-04-27T03:17:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea11d9b640 1 -- 172.21.15.77:0/158124055 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea080ab400 con 0x7fea140963b0 2024-04-27T03:17:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea11d9b640 1 -- 172.21.15.77:0/158124055 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea040015b0 con 0x7fea140963b0 2024-04-27T03:17:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea19064640 1 -- 172.21.15.77:0/158124055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea140963b0 msgr2=0x7fea14096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea19064640 1 --2- 172.21.15.77:0/158124055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea140963b0 0x7fea14096790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fea08009740 tx=0x7fea0801c320 comp rx=0 tx=0).stop 2024-04-27T03:17:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea19064640 1 -- 172.21.15.77:0/158124055 shutdown_connections 2024-04-27T03:17:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea19064640 1 --2- 172.21.15.77:0/158124055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea140963b0 0x7fea14096790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea19064640 1 -- 172.21.15.77:0/158124055 >> 172.21.15.77:0/158124055 conn(0x7fea14099440 msgr2=0x7fea14099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea19064640 1 -- 172.21.15.77:0/158124055 shutdown_connections 2024-04-27T03:17:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.986+0000 7fea19064640 1 -- 172.21.15.77:0/158124055 wait complete. 2024-04-27T03:17:42.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea19064640 1 Processor -- start 2024-04-27T03:17:42.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea19064640 1 -- start start 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea19064640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1406d4d0 0x7fea1412ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea19064640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fea14134d80 con 0x7fea1406d4d0 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea12d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1406d4d0 0x7fea1412ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea12d9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1406d4d0 0x7fea1412ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55196/0 (socket says 172.21.15.77:55196) 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea12d9d640 1 -- 172.21.15.77:0/1604242325 learned_addr learned my addr 172.21.15.77:0/1604242325 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea12d9d640 1 -- 172.21.15.77:0/1604242325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea14131550 con 0x7fea1406d4d0 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea12d9d640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1406d4d0 0x7fea1412ee50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea08019f50 tx=0x7fea0801ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fe9fb7fe640 1 -- 172.21.15.77:0/1604242325 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea080b0050 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fe9fb7fe640 1 -- 172.21.15.77:0/1604242325 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea080b1040 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea14096020 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fe9fb7fe640 1 -- 172.21.15.77:0/1604242325 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea080b75a0 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea1412f390 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fe9fb7fe640 1 -- 172.21.15.77:0/1604242325 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fea080be030 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fe9fb7fe640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fea00035db0 0x7fea00038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea141308e0 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fe9fb7fe640 1 -- 172.21.15.77:0/1604242325 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7fea080bc070 con 0x7fea1406d4d0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea1259c640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fea00035db0 0x7fea00038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.990+0000 7fea1259c640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fea00035db0 0x7fea00038270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea04000c10 tx=0x7fea040015a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:42.994+0000 7fe9fb7fe640 1 -- 172.21.15.77:0/1604242325 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fea080c5060 con 0x7fea1406d4d0 2024-04-27T03:17:43.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.138+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea14131970 con 0x7fea00035db0 2024-04-27T03:17:43.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fe9fb7fe640 1 -- 172.21.15.77:0/1604242325 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31882 (secure 0 0 0) 0x7fea04007d90 con 0x7fea00035db0 2024-04-27T03:17:43.151 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:43.151 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fea00035db0 msgr2=0x7fea00038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fea00035db0 0x7fea00038270 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea04000c10 tx=0x7fea040015a0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1406d4d0 msgr2=0x7fea1412ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1406d4d0 0x7fea1412ee50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea08019f50 tx=0x7fea0801ab60 comp rx=0 tx=0).stop 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 shutdown_connections 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fea00035db0 0x7fea00038270 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 --2- 172.21.15.77:0/1604242325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fea1406d4d0 0x7fea1412ee50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.153 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.142+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 >> 172.21.15.77:0/1604242325 conn(0x7fea14099440 msgr2=0x7fea1406f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:43.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.146+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 shutdown_connections 2024-04-27T03:17:43.154 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.146+0000 7fea19064640 1 -- 172.21.15.77:0/1604242325 wait complete. 2024-04-27T03:17:43.168 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-27T03:17:41.634367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":123352,"kb_used_data":1552,"kb_used_omap":6,"kb_used_meta":120505,"kb_avail":377364008,"statfs":{"total":386547056640,"available":386420744192,"internally_reserved":0,"allocated":1589248,"data_stored":1291738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":123397910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,12],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.635303+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:17:39.635303+0000","last_peered":"2024-04-27T03:17:39.635303+0000","last_clean":"2024-04-27T03:17:39.635303+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:17:39.635303+0000","last_undegraded":"2024-04-27T03:17:39.635303+0000","last_fullsized":"2024-04-27T03:17:39.635303+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:10:58.472036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.633080+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:17:39.633080+0000","last_peered":"2024-04-27T03:17:39.633080+0000","last_clean":"2024-04-27T03:17:39.633080+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:17:39.633080+0000","last_undegraded":"2024-04-27T03:17:39.633080+0000","last_fullsized":"2024-04-27T03:17:39.633080+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:10:58.472036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.637290+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:17:39.637290+0000","last_peered":"2024-04-27T03:17:39.637290+0000","last_clean":"2024-04-27T03:17:39.637290+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:17:39.637290+0000","last_undegraded":"2024-04-27T03:17:39.637290+0000","last_fullsized":"2024-04-27T03:17:39.637290+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:38.710604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.637578+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:17:39.637578+0000","last_peered":"2024-04-27T03:17:39.637578+0000","last_clean":"2024-04-27T03:17:39.637578+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:17:39.637578+0000","last_undegraded":"2024-04-27T03:17:39.637578+0000","last_fullsized":"2024-04-27T03:17:39.637578+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.635422+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:17:39.635422+0000","last_peered":"2024-04-27T03:17:39.635422+0000","last_clean":"2024-04-27T03:17:39.635422+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:17:39.635422+0000","last_undegraded":"2024-04-27T03:17:39.635422+0000","last_fullsized":"2024-04-27T03:17:39.635422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:08.781891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.637133+0000","last_change":"2024-04-27T03:17:39.633841+0000","last_active":"2024-04-27T03:17:39.637133+0000","last_peered":"2024-04-27T03:17:39.637133+0000","last_clean":"2024-04-27T03:17:39.637133+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:17:39.637133+0000","last_undegraded":"2024-04-27T03:17:39.637133+0000","last_fullsized":"2024-04-27T03:17:39.637133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:08.781891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.638762+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:17:39.638762+0000","last_peered":"2024-04-27T03:17:39.638762+0000","last_clean":"2024-04-27T03:17:39.638762+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:17:39.638762+0000","last_undegraded":"2024-04-27T03:17:39.638762+0000","last_fullsized":"2024-04-27T03:17:39.638762+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:01:48.896188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:39.636188+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:17:39.636188+0000","last_peered":"2024-04-27T03:17:39.636188+0000","last_clean":"2024-04-27T03:17:39.636188+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:17:39.636188+0000","last_undegraded":"2024-04-27T03:17:39.636188+0000","last_fullsized":"2024-04-27T03:17:39.636188+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":159,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-27T03:17:40.437110+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:17:40.437110+0000","last_peered":"2024-04-27T03:17:40.437110+0000","last_clean":"2024-04-27T03:17:40.437110+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:17:40.437110+0000","last_undegraded":"2024-04-27T03:17:40.437110+0000","last_fullsized":"2024-04-27T03:17:40.437110+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:08.781891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":106496,"data_stored":31043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":692224,"data_stored":617021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":692224,"data_stored":617021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,12],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":96,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":98304,"data_stored":26653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:17:43.169 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-27T03:17:43.169 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-27T03:17:43.169 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-27T03:17:43.170 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-27T03:17:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dff0bc640 1 Processor -- start 2024-04-27T03:17:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dff0bc640 1 -- start start 2024-04-27T03:17:43.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dff0bc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0df806de90 0x7f0df806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dff0bc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0df809ba90 con 0x7f0df806e7b0 2024-04-27T03:17:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dff0bc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f0df806fba0 con 0x7f0df806de90 2024-04-27T03:17:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dfce58640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f0df806de90 0x7f0df806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0df7fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f0df806e7b0 0x7f0df812e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39376/0 (socket says 172.21.15.77:39376) 2024-04-27T03:17:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0df7fff640 1 -- 172.21.15.77:0/1448581398 learned_addr learned my addr 172.21.15.77:0/1448581398 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:17:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dfce58640 1 -- 172.21.15.77:0/1448581398 >> v1:172.21.15.77:6789/0 conn(0x7f0df806e7b0 legacy=0x7f0df812e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.274+0000 7f0dfce58640 1 -- 172.21.15.77:0/1448581398 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df806ffe0 con 0x7f0df806de90 2024-04-27T03:17:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dfce58640 1 --2- 172.21.15.77:0/1448581398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df806e270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009740 tx=0x7f0de801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b824a4effb0770 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0df77fe640 1 -- 172.21.15.77:0/1448581398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0de80b0070 con 0x7f0df806de90 2024-04-27T03:17:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0df77fe640 1 -- 172.21.15.77:0/1448581398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de80ab530 con 0x7f0df806de90 2024-04-27T03:17:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0df77fe640 1 -- 172.21.15.77:0/1448581398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0de80ab830 con 0x7f0df806de90 2024-04-27T03:17:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- 172.21.15.77:0/1448581398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 msgr2=0x7f0df806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 --2- 172.21.15.77:0/1448581398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df806e270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009740 tx=0x7f0de801c400 comp rx=0 tx=0).stop 2024-04-27T03:17:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- 172.21.15.77:0/1448581398 shutdown_connections 2024-04-27T03:17:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 --2- 172.21.15.77:0/1448581398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df806e270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- 172.21.15.77:0/1448581398 >> 172.21.15.77:0/1448581398 conn(0x7f0df809cc30 msgr2=0x7f0df809d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- 172.21.15.77:0/1448581398 shutdown_connections 2024-04-27T03:17:43.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- 172.21.15.77:0/1448581398 wait complete. 2024-04-27T03:17:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 Processor -- start 2024-04-27T03:17:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- start start 2024-04-27T03:17:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df812bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f0df8136d50 con 0x7f0df806de90 2024-04-27T03:17:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dfce58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df812bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dfce58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df812bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55214/0 (socket says 172.21.15.77:55214) 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dfce58640 1 -- 172.21.15.77:0/918886202 learned_addr learned my addr 172.21.15.77:0/918886202 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dfce58640 1 -- 172.21.15.77:0/918886202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df8070dd0 con 0x7f0df806de90 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dfce58640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df812bea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0de80ae040 tx=0x7f0de801ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0df57fa640 1 -- 172.21.15.77:0/918886202 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0de80b0050 con 0x7f0df806de90 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0df57fa640 1 -- 172.21.15.77:0/918886202 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de80b1040 con 0x7f0df806de90 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0df57fa640 1 -- 172.21.15.77:0/918886202 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0de80b5870 con 0x7f0df806de90 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df811eb30 con 0x7f0df806de90 2024-04-27T03:17:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.278+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df812c5b0 con 0x7f0df806de90 2024-04-27T03:17:43.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.282+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df8062e50 con 0x7f0df806de90 2024-04-27T03:17:43.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.282+0000 7f0df57fa640 1 -- 172.21.15.77:0/918886202 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0de80be030 con 0x7f0df806de90 2024-04-27T03:17:43.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.282+0000 7f0df57fa640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0de4035d60 0x7f0de4038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.282+0000 7f0df57fa640 1 -- 172.21.15.77:0/918886202 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7f0de80bc070 con 0x7f0df806de90 2024-04-27T03:17:43.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.282+0000 7f0df7fff640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0de4035d60 0x7f0de4038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.282+0000 7f0df7fff640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0de4035d60 0x7f0de4038220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0dec001630 tx=0x7f0dec007a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.293 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.286+0000 7f0df57fa640 1 -- 172.21.15.77:0/918886202 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0de80c5020 con 0x7f0df806de90 2024-04-27T03:17:43.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.462+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0df80748d0 con 0x7f0df806de90 2024-04-27T03:17:43.472 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.462+0000 7f0df57fa640 1 -- 172.21.15.77:0/918886202 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0de8015ff0 con 0x7f0df806de90 2024-04-27T03:17:43.473 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:17:43.473 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0de4035d60 msgr2=0x7f0de4038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0de4035d60 0x7f0de4038220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0dec001630 tx=0x7f0dec007a90 comp rx=0 tx=0).stop 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 msgr2=0x7f0df812bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df812bea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0de80ae040 tx=0x7f0de801ac40 comp rx=0 tx=0).stop 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 shutdown_connections 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0de4035d60 0x7f0de4038220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 --2- 172.21.15.77:0/918886202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0df806de90 0x7f0df812bea0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 >> 172.21.15.77:0/918886202 conn(0x7f0df809cc30 msgr2=0x7f0df812d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 shutdown_connections 2024-04-27T03:17:43.475 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:17:43.466+0000 7f0dff0bc640 1 -- 172.21.15.77:0/918886202 wait complete. 2024-04-27T03:17:43.490 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-27T03:17:43.490 INFO:teuthology.run_tasks:Running task rgw... 2024-04-27T03:17:43.507 DEBUG:tasks.rgw:config is {'client.0': None} 2024-04-27T03:17:43.507 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2024-04-27T03:17:43.507 INFO:tasks.rgw:Creating data pools 2024-04-27T03:17:43.507 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-04-27T03:17:43.507 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd erasure-code-profile set client.0 k=2 m=1 crush-failure-domain=osd --cluster ceph 2024-04-27T03:17:43.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7fafa28fe640 1 Processor -- start 2024-04-27T03:17:43.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7fafa28fe640 1 -- start start 2024-04-27T03:17:43.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7fafa28fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf9c10f960 0x7faf9c10fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7fafa28fe640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf9c110400 con 0x7faf9c116760 2024-04-27T03:17:43.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7fafa28fe640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7faf9c11b280 con 0x7faf9c10f960 2024-04-27T03:17:43.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7faf9bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf9c10f960 0x7faf9c10fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7faf9bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faf9c10f960 0x7faf9c10fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39428/0 (socket says 172.21.15.110:39428) 2024-04-27T03:17:43.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7faf9bfff640 1 -- 172.21.15.110:0/2036991982 learned_addr learned my addr 172.21.15.110:0/2036991982 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:43.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7faf9b7fe640 1 --1- 172.21.15.110:0/2036991982 >> v1:172.21.15.77:6789/0 conn(0x7faf9c116760 0x7faf9c118b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:58026/0 (socket says 172.21.15.110:58026) 2024-04-27T03:17:43.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7faf9bfff640 1 -- 172.21.15.110:0/2036991982 >> v1:172.21.15.77:6789/0 conn(0x7faf9c116760 legacy=0x7faf9c118b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7faf9affd640 1 -- 172.21.15.110:0/2036991982 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458816521 0 0) 0x7faf9c110400 con 0x7faf9c116760 2024-04-27T03:17:43.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.629+0000 7faf9bfff640 1 -- 172.21.15.110:0/2036991982 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c11b400 con 0x7faf9c10f960 2024-04-27T03:17:43.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7faf9bfff640 1 --2- 172.21.15.110:0/2036991982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c10fd40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0097f0 tx=0x7faf8c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fc55e430e725eb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7faf9affd640 1 -- 172.21.15.110:0/2036991982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf8c0ab580 con 0x7faf9c10f960 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7faf9affd640 1 -- 172.21.15.110:0/2036991982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf8c0b0070 con 0x7faf9c10f960 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7faf9affd640 1 -- 172.21.15.110:0/2036991982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf8c0ab870 con 0x7faf9c10f960 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 -- 172.21.15.110:0/2036991982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 msgr2=0x7faf9c10fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 --2- 172.21.15.110:0/2036991982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c10fd40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0097f0 tx=0x7faf8c01c430 comp rx=0 tx=0).stop 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 -- 172.21.15.110:0/2036991982 shutdown_connections 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 --2- 172.21.15.110:0/2036991982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c10fd40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 -- 172.21.15.110:0/2036991982 >> 172.21.15.110:0/2036991982 conn(0x7faf9c06f440 msgr2=0x7faf9c071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 -- 172.21.15.110:0/2036991982 shutdown_connections 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 -- 172.21.15.110:0/2036991982 wait complete. 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 Processor -- start 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.633+0000 7fafa28fe640 1 -- start start 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7fafa28fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c1ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7fafa28fe640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c11af70 con 0x7faf9c10f960 2024-04-27T03:17:43.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c1ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c1ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39444/0 (socket says 172.21.15.110:39444) 2024-04-27T03:17:43.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf9bfff640 1 -- 172.21.15.110:0/1194324715 learned_addr learned my addr 172.21.15.110:0/1194324715 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:43.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf9bfff640 1 -- 172.21.15.110:0/1194324715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c1aefc0 con 0x7faf9c10f960 2024-04-27T03:17:43.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf9bfff640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c1ae9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faf8c019040 tx=0x7faf8c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf98ff9640 1 -- 172.21.15.110:0/1194324715 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf8c0ba070 con 0x7faf9c10f960 2024-04-27T03:17:43.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf98ff9640 1 -- 172.21.15.110:0/1194324715 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf8c004020 con 0x7faf9c10f960 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9c1af250 con 0x7faf9c10f960 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf98ff9640 1 -- 172.21.15.110:0/1194324715 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faf8c004340 con 0x7faf9c10f960 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9c1a8c70 con 0x7faf9c10f960 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9c062e50 con 0x7faf9c10f960 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf98ff9640 1 -- 172.21.15.110:0/1194324715 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7faf8c003620 con 0x7faf9c10f960 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf98ff9640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faf7c035d90 0x7faf7c038250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf98ff9640 1 -- 172.21.15.110:0/1194324715 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3482+0+0 (secure 0 0 0) 0x7faf8c0e2e60 con 0x7faf9c10f960 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf9b7fe640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faf7c035d90 0x7faf7c038250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:43.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.637+0000 7faf9b7fe640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faf7c035d90 0x7faf7c038250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf84007fd0 tx=0x7faf84007940 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:43.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.649+0000 7faf98ff9640 1 -- 172.21.15.110:0/1194324715 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faf8c0160b0 con 0x7faf9c10f960 2024-04-27T03:17:43.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.821+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "client.0", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]} v 0) -- 0x7faf9c1af420 con 0x7faf9c10f960 2024-04-27T03:17:43.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7faf98ff9640 1 -- 172.21.15.110:0/1194324715 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "client.0", "profile": ["k=2", "m=1", "crush-failure-domain=osd"]}]=0 v13) ==== 151+0+0 (secure 0 0 0) 0x7faf8c0b3540 con 0x7faf9c10f960 2024-04-27T03:17:43.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faf7c035d90 msgr2=0x7faf7c038250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faf7c035d90 0x7faf7c038250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf84007fd0 tx=0x7faf84007940 comp rx=0 tx=0).stop 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 msgr2=0x7faf9c1ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c1ae9f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faf8c019040 tx=0x7faf8c01ac70 comp rx=0 tx=0).stop 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 shutdown_connections 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faf7c035d90 0x7faf7c038250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 --2- 172.21.15.110:0/1194324715 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf9c10f960 0x7faf9c1ae9f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 >> 172.21.15.110:0/1194324715 conn(0x7faf9c06f440 msgr2=0x7faf9c115550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 shutdown_connections 2024-04-27T03:17:43.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:43.949+0000 7fafa28fe640 1 -- 172.21.15.110:0/1194324715 wait complete. 2024-04-27T03:17:43.980 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create default.rgw.buckets.data 64 64 erasure client.0 --cluster ceph 2024-04-27T03:17:44.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.097+0000 7f5c16c10640 1 Processor -- start 2024-04-27T03:17:44.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.097+0000 7f5c16c10640 1 -- start start 2024-04-27T03:17:44.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.097+0000 7f5c16c10640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c1010dfc0 0x7f5c1010e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:44.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c16c10640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1011b090 con 0x7f5c1010e8e0 2024-04-27T03:17:44.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c16c10640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5c1011c270 con 0x7f5c1010dfc0 2024-04-27T03:17:44.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c149ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c1010dfc0 0x7f5c1010e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:44.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c149ac640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5c1010dfc0 0x7f5c1010e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:47930/0 (socket says 172.21.15.110:47930) 2024-04-27T03:17:44.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c149ac640 1 -- 172.21.15.110:0/1221567918 learned_addr learned my addr 172.21.15.110:0/1221567918 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:44.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c149ac640 1 -- 172.21.15.110:0/1221567918 >> v1:172.21.15.77:6789/0 conn(0x7f5c1010e8e0 legacy=0x7f5c10119970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:44.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c149ac640 1 -- 172.21.15.110:0/1221567918 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1011c3f0 con 0x7f5c1010dfc0 2024-04-27T03:17:44.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c149ac640 1 --2- 172.21.15.110:0/1221567918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c1010e3a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c000097f0 tx=0x7f5c0001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9aa5a1e375e21e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:44.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c0f7fe640 1 -- 172.21.15.110:0/1221567918 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c000ab580 con 0x7f5c1010dfc0 2024-04-27T03:17:44.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c0f7fe640 1 -- 172.21.15.110:0/1221567918 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c000b0070 con 0x7f5c1010dfc0 2024-04-27T03:17:44.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.101+0000 7f5c0f7fe640 1 -- 172.21.15.110:0/1221567918 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c000ab870 con 0x7f5c1010dfc0 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/1221567918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 msgr2=0x7f5c1010e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 --2- 172.21.15.110:0/1221567918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c1010e3a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5c000097f0 tx=0x7f5c0001c430 comp rx=0 tx=0).stop 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/1221567918 shutdown_connections 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 --2- 172.21.15.110:0/1221567918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c1010e3a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/1221567918 >> 172.21.15.110:0/1221567918 conn(0x7f5c1006f440 msgr2=0x7f5c10071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/1221567918 shutdown_connections 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/1221567918 wait complete. 2024-04-27T03:17:44.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 Processor -- start 2024-04-27T03:17:44.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- start start 2024-04-27T03:17:44.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c101a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:44.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1011bf30 con 0x7f5c1010dfc0 2024-04-27T03:17:44.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c149ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c101a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:44.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c149ac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c101a6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:47942/0 (socket says 172.21.15.110:47942) 2024-04-27T03:17:44.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c149ac640 1 -- 172.21.15.110:0/3141893208 learned_addr learned my addr 172.21.15.110:0/3141893208 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c149ac640 1 -- 172.21.15.110:0/3141893208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c101a7360 con 0x7f5c1010dfc0 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c149ac640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c101a6e20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001a060 tx=0x7f5c0001ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c0d7fa640 1 -- 172.21.15.110:0/3141893208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c000aa7d0 con 0x7f5c1010dfc0 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c101a75f0 con 0x7f5c1010dfc0 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c0d7fa640 1 -- 172.21.15.110:0/3141893208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c000aa970 con 0x7f5c1010dfc0 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c0d7fa640 1 -- 172.21.15.110:0/3141893208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c000aac70 con 0x7f5c1010dfc0 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c101ab450 con 0x7f5c1010dfc0 2024-04-27T03:17:44.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.105+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c10062e50 con 0x7f5c1010dfc0 2024-04-27T03:17:44.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.109+0000 7f5c0d7fa640 1 -- 172.21.15.110:0/3141893208 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5c000059a0 con 0x7f5c1010dfc0 2024-04-27T03:17:44.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.109+0000 7f5c0d7fa640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5bf0035d60 0x7f5bf0038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:44.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.109+0000 7f5c0d7fa640 1 -- 172.21.15.110:0/3141893208 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3774+0+0 (secure 0 0 0) 0x7f5c000e3ad0 con 0x7f5c1010dfc0 2024-04-27T03:17:44.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.109+0000 7f5c0ffff640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5bf0035d60 0x7f5bf0038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:44.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.109+0000 7f5c0ffff640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5bf0035d60 0x7f5bf0038220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8000f70 tx=0x7f5bf8007a60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:44.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.121+0000 7f5c0d7fa640 1 -- 172.21.15.110:0/3141893208 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5c000160b0 con 0x7f5c1010dfc0 2024-04-27T03:17:44.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:44.285+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "default.rgw.buckets.data", "pg_num": 64, "pgp_num": 64, "pool_type": "erasure", "erasure_code_profile": "client.0"} v 0) -- 0x7f5c1010e8e0 con 0x7f5c1010dfc0 2024-04-27T03:17:45.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c0d7fa640 1 -- 172.21.15.110:0/3141893208 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "default.rgw.buckets.data", "pg_num": 64, "pgp_num": 64, "pool_type": "erasure", "erasure_code_profile": "client.0"}]=0 pool 'default.rgw.buckets.data' created v15) ==== 227+0+0 (secure 0 0 0) 0x7f5c000cd100 con 0x7f5c1010dfc0 2024-04-27T03:17:45.968 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'default.rgw.buckets.data' created 2024-04-27T03:17:45.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5bf0035d60 msgr2=0x7f5bf0038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:45.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5bf0035d60 0x7f5bf0038220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8000f70 tx=0x7f5bf8007a60 comp rx=0 tx=0).stop 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 msgr2=0x7f5c101a6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c101a6e20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001a060 tx=0x7f5c0001ac70 comp rx=0 tx=0).stop 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 shutdown_connections 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5bf0035d60 0x7f5bf0038220 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 --2- 172.21.15.110:0/3141893208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5c1010dfc0 0x7f5c101a6e20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 >> 172.21.15.110:0/3141893208 conn(0x7f5c1006f440 msgr2=0x7f5c101171b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.961+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 shutdown_connections 2024-04-27T03:17:45.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:45.965+0000 7f5c16c10640 1 -- 172.21.15.110:0/3141893208 wait complete. 2024-04-27T03:17:46.001 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2024-04-27T03:17:46.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f26a15fd640 1 Processor -- start 2024-04-27T03:17:46.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f26a15fd640 1 -- start start 2024-04-27T03:17:46.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f26a15fd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f269c10e850 0x7f269c11cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:46.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f26a15fd640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f269c11d340 con 0x7f269c10dd60 2024-04-27T03:17:46.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f26a15fd640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f269c11e520 con 0x7f269c10e850 2024-04-27T03:17:46.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f269a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f269c10e850 0x7f269c11cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:46.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f269a7fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f269c10e850 0x7f269c11cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:47954/0 (socket says 172.21.15.110:47954) 2024-04-27T03:17:46.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f269affd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f269c10dd60 0x7f269c10e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:59470/0 (socket says 172.21.15.110:59470) 2024-04-27T03:17:46.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f269a7fc640 1 -- 172.21.15.110:0/856968210 learned_addr learned my addr 172.21.15.110:0/856968210 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:46.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f269a7fc640 1 -- 172.21.15.110:0/856968210 >> v1:172.21.15.77:6789/0 conn(0x7f269c10dd60 legacy=0x7f269c10e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:46.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.113+0000 7f269a7fc640 1 -- 172.21.15.110:0/856968210 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269c11e6a0 con 0x7f269c10e850 2024-04-27T03:17:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f269a7fc640 1 --2- 172.21.15.110:0/856968210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10e850 0x7f269c11cd70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097f0 tx=0x7f268c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23be2252a0b11b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f2699ffb640 1 -- 172.21.15.110:0/856968210 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f268c0ab580 con 0x7f269c10e850 2024-04-27T03:17:46.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f2699ffb640 1 -- 172.21.15.110:0/856968210 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f268c0b0070 con 0x7f269c10e850 2024-04-27T03:17:46.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f2699ffb640 1 -- 172.21.15.110:0/856968210 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f268c0ab870 con 0x7f269c10e850 2024-04-27T03:17:46.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 -- 172.21.15.110:0/856968210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10e850 msgr2=0x7f269c11cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:46.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 --2- 172.21.15.110:0/856968210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10e850 0x7f269c11cd70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097f0 tx=0x7f268c01c430 comp rx=0 tx=0).stop 2024-04-27T03:17:46.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 -- 172.21.15.110:0/856968210 shutdown_connections 2024-04-27T03:17:46.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 --2- 172.21.15.110:0/856968210 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10e850 0x7f269c11cd70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:46.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 -- 172.21.15.110:0/856968210 >> 172.21.15.110:0/856968210 conn(0x7f269c06f440 msgr2=0x7f269c071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:46.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 -- 172.21.15.110:0/856968210 shutdown_connections 2024-04-27T03:17:46.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 -- 172.21.15.110:0/856968210 wait complete. 2024-04-27T03:17:46.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.117+0000 7f26a15fd640 1 Processor -- start 2024-04-27T03:17:46.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26a15fd640 1 -- start start 2024-04-27T03:17:46.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26a15fd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10dd60 0x7f269c1aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:46.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26a15fd640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f269c11e200 con 0x7f269c10dd60 2024-04-27T03:17:46.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f269affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10dd60 0x7f269c1aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:46.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f269affd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10dd60 0x7f269c1aebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:47970/0 (socket says 172.21.15.110:47970) 2024-04-27T03:17:46.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f269affd640 1 -- 172.21.15.110:0/1872385999 learned_addr learned my addr 172.21.15.110:0/1872385999 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:46.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f269affd640 1 -- 172.21.15.110:0/1872385999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f269c1af1a0 con 0x7f269c10dd60 2024-04-27T03:17:46.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f269affd640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10dd60 0x7f269c1aebd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f269000c3b0 tx=0x7f269000c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:46.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26837fe640 1 -- 172.21.15.110:0/1872385999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2690007410 con 0x7f269c10dd60 2024-04-27T03:17:46.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26837fe640 1 -- 172.21.15.110:0/1872385999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26900075b0 con 0x7f269c10dd60 2024-04-27T03:17:46.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269c1af3d0 con 0x7f269c10dd60 2024-04-27T03:17:46.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26837fe640 1 -- 172.21.15.110:0/1872385999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26900047f0 con 0x7f269c10dd60 2024-04-27T03:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f269c1a8c70 con 0x7f269c10dd60 2024-04-27T03:17:46.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f269c062e50 con 0x7f269c10dd60 2024-04-27T03:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26837fe640 1 -- 172.21.15.110:0/1872385999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f269001a020 con 0x7f269c10dd60 2024-04-27T03:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26837fe640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2688035d60 0x7f2688038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f26837fe640 1 -- 172.21.15.110:0/1872385999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4249+0+0 (secure 0 0 0) 0x7f26900488c0 con 0x7f269c10dd60 2024-04-27T03:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f269a7fc640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2688035d60 0x7f2688038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:46.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.121+0000 7f269a7fc640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2688035d60 0x7f2688038220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f268c0af000 tx=0x7f268c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:46.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.137+0000 7f26837fe640 1 -- 172.21.15.110:0/1872385999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2690021070 con 0x7f269c10dd60 2024-04-27T03:17:46.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.297+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "default.rgw.buckets.data", "app": "rgw"} v 0) -- 0x7f269c10e850 con 0x7f269c10dd60 2024-04-27T03:17:46.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.961+0000 7f26837fe640 1 -- 172.21.15.110:0/1872385999 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "default.rgw.buckets.data", "app": "rgw"}]=0 enabled application 'rgw' on pool 'default.rgw.buckets.data' v16) ==== 185+0+0 (secure 0 0 0) 0x7f269001bc30 con 0x7f269c10dd60 2024-04-27T03:17:46.970 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2688035d60 msgr2=0x7f2688038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2688035d60 0x7f2688038220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f268c0af000 tx=0x7f268c0a7000 comp rx=0 tx=0).stop 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10dd60 msgr2=0x7f269c1aebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10dd60 0x7f269c1aebd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f269000c3b0 tx=0x7f269000c870 comp rx=0 tx=0).stop 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 shutdown_connections 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2688035d60 0x7f2688038220 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 --2- 172.21.15.110:0/1872385999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f269c10dd60 0x7f269c1aebd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 >> 172.21.15.110:0/1872385999 conn(0x7f269c06f440 msgr2=0x7f269c070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 shutdown_connections 2024-04-27T03:17:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:46.965+0000 7f26a15fd640 1 -- 172.21.15.110:0/1872385999 wait complete. 2024-04-27T03:17:47.002 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2024-04-27T03:17:47.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f7cd4f640 1 Processor -- start 2024-04-27T03:17:47.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f7cd4f640 1 -- start start 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f7cd4f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f7810e850 0x7f9f7811cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f7cd4f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f7811d340 con 0x7f9f7810dd60 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f7cd4f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9f7811e520 con 0x7f9f7810e850 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f6ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9f7810e850 0x7f9f7811cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f7659c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9f7810dd60 0x7f9f7810e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:59478/0 (socket says 172.21.15.110:59478) 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f7659c640 1 -- 172.21.15.110:0/137809971 learned_addr learned my addr 172.21.15.110:0/137809971 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f6ffff640 1 -- 172.21.15.110:0/137809971 >> v1:172.21.15.77:6789/0 conn(0x7f9f7810dd60 legacy=0x7f9f7810e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:47.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f6ffff640 1 -- 172.21.15.110:0/137809971 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7811e6a0 con 0x7f9f7810e850 2024-04-27T03:17:47.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.129+0000 7f9f6ffff640 1 --2- 172.21.15.110:0/137809971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810e850 0x7f9f7811cd70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009740 tx=0x7f9f6001c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572687f568b74d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:47.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f75d9b640 1 -- 172.21.15.110:0/137809971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f600b1070 con 0x7f9f7810e850 2024-04-27T03:17:47.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f75d9b640 1 -- 172.21.15.110:0/137809971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f600ac520 con 0x7f9f7810e850 2024-04-27T03:17:47.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f75d9b640 1 -- 172.21.15.110:0/137809971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f600ac820 con 0x7f9f7810e850 2024-04-27T03:17:47.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/137809971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810e850 msgr2=0x7f9f7811cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:47.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 --2- 172.21.15.110:0/137809971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810e850 0x7f9f7811cd70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009740 tx=0x7f9f6001c370 comp rx=0 tx=0).stop 2024-04-27T03:17:47.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/137809971 shutdown_connections 2024-04-27T03:17:47.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 --2- 172.21.15.110:0/137809971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810e850 0x7f9f7811cd70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:47.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/137809971 >> 172.21.15.110:0/137809971 conn(0x7f9f7806f440 msgr2=0x7f9f78071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:47.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/137809971 shutdown_connections 2024-04-27T03:17:47.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/137809971 wait complete. 2024-04-27T03:17:47.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 Processor -- start 2024-04-27T03:17:47.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- start start 2024-04-27T03:17:47.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810dd60 0x7f9f781aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:47.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7811e200 con 0x7f9f7810dd60 2024-04-27T03:17:47.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810dd60 0x7f9f781aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:47.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810dd60 0x7f9f781aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:47984/0 (socket says 172.21.15.110:47984) 2024-04-27T03:17:47.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7659c640 1 -- 172.21.15.110:0/2587853857 learned_addr learned my addr 172.21.15.110:0/2587853857 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:47.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7659c640 1 -- 172.21.15.110:0/2587853857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f781af200 con 0x7f9f7810dd60 2024-04-27T03:17:47.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7659c640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810dd60 0x7f9f781aec30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800c450 tx=0x7f9f6800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:47.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f6effd640 1 -- 172.21.15.110:0/2587853857 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f68017070 con 0x7f9f7810dd60 2024-04-27T03:17:47.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f781af490 con 0x7f9f7810dd60 2024-04-27T03:17:47.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f6effd640 1 -- 172.21.15.110:0/2587853857 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f68005c20 con 0x7f9f7810dd60 2024-04-27T03:17:47.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f6effd640 1 -- 172.21.15.110:0/2587853857 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f680070d0 con 0x7f9f7810dd60 2024-04-27T03:17:47.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.133+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f781a8ea0 con 0x7f9f7810dd60 2024-04-27T03:17:47.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.137+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f78062e50 con 0x7f9f7810dd60 2024-04-27T03:17:47.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.137+0000 7f9f6effd640 1 -- 172.21.15.110:0/2587853857 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9f68007270 con 0x7f9f7810dd60 2024-04-27T03:17:47.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.137+0000 7f9f6effd640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f58035d60 0x7f9f58038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:47.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.137+0000 7f9f6effd640 1 -- 172.21.15.110:0/2587853857 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4260+0+0 (secure 0 0 0) 0x7f9f68048520 con 0x7f9f7810dd60 2024-04-27T03:17:47.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.137+0000 7f9f6ffff640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f58035d60 0x7f9f58038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:47.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.137+0000 7f9f6ffff640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f58035d60 0x7f9f58038220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f600ae000 tx=0x7f9f60008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.149+0000 7f9f6effd640 1 -- 172.21.15.110:0/2587853857 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9f68019df0 con 0x7f9f7810dd60 2024-04-27T03:17:47.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.317+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "default.rgw.buckets.index", "pg_num": 64, "pgp_num": 64} v 0) -- 0x7f9f781af730 con 0x7f9f7810dd60 2024-04-27T03:17:47.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.977+0000 7f9f6effd640 1 -- 172.21.15.110:0/2587853857 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "default.rgw.buckets.index", "pg_num": 64, "pgp_num": 64}]=0 pool 'default.rgw.buckets.index' created v17) ==== 169+0+0 (secure 0 0 0) 0x7f9f781af730 con 0x7f9f7810dd60 2024-04-27T03:17:47.985 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'default.rgw.buckets.index' created 2024-04-27T03:17:47.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f58035d60 msgr2=0x7f9f58038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:47.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f58035d60 0x7f9f58038220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9f600ae000 tx=0x7f9f60008040 comp rx=0 tx=0).stop 2024-04-27T03:17:47.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810dd60 msgr2=0x7f9f781aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:47.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810dd60 0x7f9f781aec30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f6800c450 tx=0x7f9f6800c910 comp rx=0 tx=0).stop 2024-04-27T03:17:47.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 shutdown_connections 2024-04-27T03:17:47.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9f58035d60 0x7f9f58038220 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:47.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 --2- 172.21.15.110:0/2587853857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9f7810dd60 0x7f9f781aec30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:47.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 >> 172.21.15.110:0/2587853857 conn(0x7f9f7806f440 msgr2=0x7f9f78070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:47.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 shutdown_connections 2024-04-27T03:17:47.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:47.981+0000 7f9f7cd4f640 1 -- 172.21.15.110:0/2587853857 wait complete. 2024-04-27T03:17:48.019 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2024-04-27T03:17:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faafbd78640 1 Processor -- start 2024-04-27T03:17:48.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faafbd78640 1 -- start start 2024-04-27T03:17:48.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faafbd78640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faaf410dfc0 0x7faaf410e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:48.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faafbd78640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaf411b090 con 0x7faaf410e8e0 2024-04-27T03:17:48.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faafbd78640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7faaf411c270 con 0x7faaf410dfc0 2024-04-27T03:17:48.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faaf9b14640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faaf410dfc0 0x7faaf410e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:48.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faaf9b14640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7faaf410dfc0 0x7faaf410e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:47996/0 (socket says 172.21.15.110:47996) 2024-04-27T03:17:48.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.133+0000 7faaf9b14640 1 -- 172.21.15.110:0/3700622914 learned_addr learned my addr 172.21.15.110:0/3700622914 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:48.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faaf9b14640 1 -- 172.21.15.110:0/3700622914 >> v1:172.21.15.77:6789/0 conn(0x7faaf410e8e0 legacy=0x7faaf4119970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:48.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faaf9b14640 1 -- 172.21.15.110:0/3700622914 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf411c3f0 con 0x7faaf410dfc0 2024-04-27T03:17:48.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faaf9b14640 1 --2- 172.21.15.110:0/3700622914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf410e3a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faae40097f0 tx=0x7faae401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d289a54d4b6cd31 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:48.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faaf8b12640 1 -- 172.21.15.110:0/3700622914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faae40ab580 con 0x7faaf410dfc0 2024-04-27T03:17:48.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faaf8b12640 1 -- 172.21.15.110:0/3700622914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faae40b0070 con 0x7faaf410dfc0 2024-04-27T03:17:48.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faaf8b12640 1 -- 172.21.15.110:0/3700622914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faae40ab870 con 0x7faaf410dfc0 2024-04-27T03:17:48.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faafbd78640 1 -- 172.21.15.110:0/3700622914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 msgr2=0x7faaf410e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:48.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faafbd78640 1 --2- 172.21.15.110:0/3700622914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf410e3a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faae40097f0 tx=0x7faae401c430 comp rx=0 tx=0).stop 2024-04-27T03:17:48.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faafbd78640 1 -- 172.21.15.110:0/3700622914 shutdown_connections 2024-04-27T03:17:48.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faafbd78640 1 --2- 172.21.15.110:0/3700622914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf410e3a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:48.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faafbd78640 1 -- 172.21.15.110:0/3700622914 >> 172.21.15.110:0/3700622914 conn(0x7faaf406f440 msgr2=0x7faaf4071860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:48.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faafbd78640 1 -- 172.21.15.110:0/3700622914 shutdown_connections 2024-04-27T03:17:48.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.137+0000 7faafbd78640 1 -- 172.21.15.110:0/3700622914 wait complete. 2024-04-27T03:17:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faafbd78640 1 Processor -- start 2024-04-27T03:17:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faafbd78640 1 -- start start 2024-04-27T03:17:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faafbd78640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf41a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faafbd78640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7faaf411bf30 con 0x7faaf410dfc0 2024-04-27T03:17:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaf9b14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf41a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaf9b14640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf41a6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:48012/0 (socket says 172.21.15.110:48012) 2024-04-27T03:17:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaf9b14640 1 -- 172.21.15.110:0/2503123236 learned_addr learned my addr 172.21.15.110:0/2503123236 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:17:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaf9b14640 1 -- 172.21.15.110:0/2503123236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf41a7360 con 0x7faaf410dfc0 2024-04-27T03:17:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaf9b14640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf41a6e20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faae4019040 tx=0x7faae401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaea7fc640 1 -- 172.21.15.110:0/2503123236 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faae40abe10 con 0x7faaf410dfc0 2024-04-27T03:17:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaea7fc640 1 -- 172.21.15.110:0/2503123236 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faae40aa8f0 con 0x7faaf410dfc0 2024-04-27T03:17:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf41a75f0 con 0x7faaf410dfc0 2024-04-27T03:17:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faaea7fc640 1 -- 172.21.15.110:0/2503123236 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faae40aac10 con 0x7faaf410dfc0 2024-04-27T03:17:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf41ab450 con 0x7faaf410dfc0 2024-04-27T03:17:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.141+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf4062e50 con 0x7faaf410dfc0 2024-04-27T03:17:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.145+0000 7faaea7fc640 1 -- 172.21.15.110:0/2503123236 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7faae40059a0 con 0x7faaf410dfc0 2024-04-27T03:17:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.145+0000 7faaea7fc640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faad8035e00 0x7faad80382c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:17:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.145+0000 7faaea7fc640 1 -- 172.21.15.110:0/2503123236 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4624+0+0 (secure 0 0 0) 0x7faae40e3dd0 con 0x7faaf410dfc0 2024-04-27T03:17:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.145+0000 7faaf9313640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faad8035e00 0x7faad80382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:17:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.145+0000 7faaf9313640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faad8035e00 0x7faad80382c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faaf0001610 tx=0x7faaf00079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:17:48.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.157+0000 7faaea7fc640 1 -- 172.21.15.110:0/2503123236 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faae40160b0 con 0x7faaf410dfc0 2024-04-27T03:17:48.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.321+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "default.rgw.buckets.index", "app": "rgw"} v 0) -- 0x7faaf410e8e0 con 0x7faaf410dfc0 2024-04-27T03:17:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.977+0000 7faaea7fc640 1 -- 172.21.15.110:0/2503123236 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "default.rgw.buckets.index", "app": "rgw"}]=0 enabled application 'rgw' on pool 'default.rgw.buckets.index' v18) ==== 187+0+0 (secure 0 0 0) 0x7faae40aadb0 con 0x7faaf410dfc0 2024-04-27T03:17:48.986 INFO:teuthology.orchestra.run.smithi110.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-04-27T03:17:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faad8035e00 msgr2=0x7faad80382c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faad8035e00 0x7faad80382c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7faaf0001610 tx=0x7faaf00079e0 comp rx=0 tx=0).stop 2024-04-27T03:17:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 msgr2=0x7faaf41a6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:17:48.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf41a6e20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faae4019040 tx=0x7faae401ac70 comp rx=0 tx=0).stop 2024-04-27T03:17:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 shutdown_connections 2024-04-27T03:17:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faad8035e00 0x7faad80382c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 --2- 172.21.15.110:0/2503123236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf410dfc0 0x7faaf41a6e20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:17:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 >> 172.21.15.110:0/2503123236 conn(0x7faaf406f440 msgr2=0x7faaf41171b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:17:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 shutdown_connections 2024-04-27T03:17:48.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:17:48.981+0000 7faafbd78640 1 -- 172.21.15.110:0/2503123236 wait complete. 2024-04-27T03:17:49.025 DEBUG:tasks.rgw:Pools created 2024-04-27T03:17:49.025 INFO:tasks.rgw:Starting rgw... 2024-04-27T03:17:49.025 INFO:tasks.rgw:rgw client.0 config is {} 2024-04-27T03:17:49.025 INFO:tasks.rgw:Using beast as radosgw frontend 2024-04-27T03:17:49.026 DEBUG:teuthology.orchestra.run.smithi110:> sudo echo -n http://smithi110.front.sepia.ceph.com:80 | sudo tee /home/ubuntu/cephtest/url_file 2024-04-27T03:17:49.038 INFO:teuthology.orchestra.run.smithi110.stdout:http://smithi110.front.sepia.ceph.com:80 2024-04-27T03:17:49.039 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown ceph /home/ubuntu/cephtest/url_file 2024-04-27T03:17:49.089 INFO:tasks.rgw.client.0:Restarting daemon 2024-04-27T03:17:49.089 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-04-27T03:17:49.133 INFO:tasks.rgw.client.0:Started 2024-04-27T03:17:49.133 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi110.front.sepia.ceph.com:80/ 2024-04-27T03:17:49.133 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:80/ 2024-04-27T03:17:49.152 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-27T03:17:49.152 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2024-04-27T03:17:49.152 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-27T03:17:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-27T03:17:49.153 INFO:teuthology.orchestra.run.smithi110.stderr:curl: (7) Failed to connect to smithi110.front.sepia.ceph.com port 80 after 0 ms: Connection refused 2024-04-27T03:17:50.155 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:80/ 2024-04-27T03:17:50.167 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-27T03:17:50.167 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2024-04-27T03:17:50.167 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-27T03:17:50.167 INFO:teuthology.orchestra.run.smithi110.stderr:curl: (7) Failed to connect to smithi110.front.sepia.ceph.com port 80 after 0 ms: Connection refused 2024-04-27T03:17:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-27T03:17:52.169 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:80/ 2024-04-27T03:17:52.180 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-27T03:17:52.180 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2024-04-27T03:17:52.181 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-27T03:17:52.181 INFO:teuthology.orchestra.run.smithi110.stderr:curl: (7) Failed to connect to smithi110.front.sepia.ceph.com port 80 after 0 ms: Connection refused 2024-04-27T03:17:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-27T03:17:56.183 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:80/ 2024-04-27T03:17:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-04-27T03:17:56.197 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-27T03:17:56.197 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2024-04-27T03:17:56.197 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2024-04-27T03:17:56.198 INFO:teuthology.orchestra.run.smithi110.stderr:curl: (7) Failed to connect to smithi110.front.sepia.ceph.com port 80 after 0 ms: Connection refused 2024-04-27T03:17:58.216 INFO:tasks.rgw.client.0.smithi110.stdout:2024-04-27T03:17:58.209+0000 7f1478215a40 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-27T03:18:04.199 DEBUG:teuthology.orchestra.run.smithi110:> curl http://smithi110.front.sepia.ceph.com:80/ 2024-04-27T03:18:04.211 INFO:teuthology.orchestra.run.smithi110.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-04-27T03:18:04.211 INFO:teuthology.orchestra.run.smithi110.stderr: Dload Upload Total Spent Left Speed 2024-04-27T03:18:04.214 INFO:teuthology.orchestra.run.smithi110.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 187 0 187 0 0 53874 0 --:--:-- --:--:-- --:--:-- 62333 2024-04-27T03:18:04.215 INFO:teuthology.orchestra.run.smithi110.stdout:anonymous 2024-04-27T03:18:04.215 INFO:teuthology.run_tasks:Running task radosgw-admin... 2024-04-27T03:18:04.301 DEBUG:tasks.radosgw_admin:config is: {'client.0': None} 2024-04-27T03:18:04.301 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list'] 2024-04-27T03:18:04.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list'] 2024-04-27T03:18:04.301 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list 2024-04-27T03:18:04.338 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:04.338 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:04.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.329+0000 7f81dcb7d980 1 Processor -- start 2024-04-27T03:18:04.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.329+0000 7f81dcb7d980 1 -- start start 2024-04-27T03:18:04.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81dcb7d980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560f436c2fb0 0x560f43751f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81dcb7d980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x560f4360e0d0 con 0x560f43752490 2024-04-27T03:18:04.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81dcb7d980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x560f43603430 con 0x560f436c2fb0 2024-04-27T03:18:04.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81da698640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560f436c2fb0 0x560f43751f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81d9e97640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x560f43752490 0x560f43754880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34506/0 (socket says 172.21.15.110:34506) 2024-04-27T03:18:04.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81da698640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560f436c2fb0 0x560f43751f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49864/0 (socket says 172.21.15.110:49864) 2024-04-27T03:18:04.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81d9e97640 1 -- 172.21.15.110:0/1594411980 learned_addr learned my addr 172.21.15.110:0/1594411980 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:04.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81da698640 1 -- 172.21.15.110:0/1594411980 >> v1:172.21.15.77:6789/0 conn(0x560f43752490 legacy=0x560f43754880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81da698640 1 -- 172.21.15.110:0/1594411980 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f435fc510 con 0x560f436c2fb0 2024-04-27T03:18:04.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81da698640 1 --2- 172.21.15.110:0/1594411980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f436c2fb0 0x560f43751f50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00a420 tx=0x7f81cc01afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346e129afb576ae2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81d9696640 1 -- 172.21.15.110:0/1594411980 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81cc0af070 con 0x560f436c2fb0 2024-04-27T03:18:04.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81d9696640 1 -- 172.21.15.110:0/1594411980 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc0aa530 con 0x560f436c2fb0 2024-04-27T03:18:04.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.333+0000 7f81d9696640 1 -- 172.21.15.110:0/1594411980 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81cc0aa850 con 0x560f436c2fb0 2024-04-27T03:18:04.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.337+0000 7f81dcb7d980 1 -- 172.21.15.110:0/1594411980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f436c2fb0 msgr2=0x560f43751f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.337+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/1594411980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f436c2fb0 0x560f43751f50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00a420 tx=0x7f81cc01afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.337+0000 7f81dcb7d980 1 -- 172.21.15.110:0/1594411980 shutdown_connections 2024-04-27T03:18:04.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.337+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/1594411980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f436c2fb0 0x560f43751f50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.337+0000 7f81dcb7d980 1 -- 172.21.15.110:0/1594411980 >> 172.21.15.110:0/1594411980 conn(0x560f43749a30 msgr2=0x560f4374be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.337+0000 7f81dcb7d980 1 -- 172.21.15.110:0/1594411980 shutdown_connections 2024-04-27T03:18:04.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.337+0000 7f81dcb7d980 1 -- 172.21.15.110:0/1594411980 wait complete. 2024-04-27T03:18:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 Processor -- start 2024-04-27T03:18:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 -- start start 2024-04-27T03:18:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f4374f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f436522d0 con 0x560f43752490 2024-04-27T03:18:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81da698640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f4374f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81da698640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f4374f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49868/0 (socket says 172.21.15.110:49868) 2024-04-27T03:18:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81da698640 1 -- 172.21.15.110:0/2751026132 learned_addr learned my addr 172.21.15.110:0/2751026132 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81da698640 1 -- 172.21.15.110:0/2751026132 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f435d8ae0 con 0x560f43752490 2024-04-27T03:18:04.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81da698640 1 --2- 172.21.15.110:0/2751026132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f4374f6b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0ad040 tx=0x7f81cc0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfa44b0b18243cb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f80c8608640 1 -- 172.21.15.110:0/2751026132 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81cc0af050 con 0x560f43752490 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f80c8608640 1 -- 172.21.15.110:0/2751026132 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc0b0040 con 0x560f43752490 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f80c8608640 1 -- 172.21.15.110:0/2751026132 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81cc0aaad0 con 0x560f43752490 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2751026132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 msgr2=0x560f4374f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2751026132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f4374f6b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0ad040 tx=0x7f81cc0033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2751026132 shutdown_connections 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2751026132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f4374f6b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2751026132 >> 172.21.15.110:0/2751026132 conn(0x560f43749a30 msgr2=0x560f43836120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2751026132 shutdown_connections 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.353+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2751026132 wait complete. 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 Processor -- start 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 -- start start 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f438437d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f43756ab0 con 0x560f43752490 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81da698640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f438437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81da698640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f438437d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49884/0 (socket says 172.21.15.110:49884) 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81da698640 1 -- 172.21.15.110:0/3751934948 learned_addr learned my addr 172.21.15.110:0/3751934948 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81da698640 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f43843d10 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81da698640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f438437d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f81cc018c00 tx=0x7f81cc017d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f80c6e05640 1 -- 172.21.15.110:0/3751934948 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81cc0b6ac0 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f43843fa0 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f80c6e05640 1 -- 172.21.15.110:0/3751934948 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc0b6c60 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f43846bd0 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f80c6e05640 1 -- 172.21.15.110:0/3751934948 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81cc0cd060 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f80c6e05640 1 -- 172.21.15.110:0/3751934948 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f81cc0c0470 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f80c6e05640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c0034650 0x7f80c0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f80c6e05640 1 -- 172.21.15.110:0/3751934948 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f81cc002040 con 0x560f43752490 2024-04-27T03:18:04.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81d9e97640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c0034650 0x7f80c0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f438486a0 0x560f4384aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.357+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4384afd0 con 0x560f438486a0 2024-04-27T03:18:04.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dae99640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f438486a0 0x560f4384aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81d9e97640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c0034650 0x7f80c0036b10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f81d0000f70 tx=0x7f81d00012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dae99640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f438486a0 0x560f4384aa90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dae99640 1 -- 172.21.15.110:0/3751934948 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81d40083f0 con 0x560f438486a0 2024-04-27T03:18:04.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f4384bd30 0x560f4384e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4384e760 con 0x560f4384bd30 2024-04-27T03:18:04.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81da698640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f4384bd30 0x560f4384e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81da698640 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f4384bd30 0x560f4384e120 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81da698640 1 -- 172.21.15.110:0/3751934948 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f4384bd30 2024-04-27T03:18:04.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43850010 con 0x560f438486a0 2024-04-27T03:18:04.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dae99640 1 -- 172.21.15.110:0/3751934948 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f81d40083f0 con 0x560f438486a0 2024-04-27T03:18:04.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4384f240 con 0x560f4384bd30 2024-04-27T03:18:04.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81da698640 1 -- 172.21.15.110:0/3751934948 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f4384bd30 2024-04-27T03:18:04.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.361+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4384f630 con 0x560f438486a0 2024-04-27T03:18:04.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dae99640 1 -- 172.21.15.110:0/3751934948 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f81d40083f0 con 0x560f438486a0 2024-04-27T03:18:04.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dcb7d980 1 Processor -- start 2024-04-27T03:18:04.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dcb7d980 1 -- start start 2024-04-27T03:18:04.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dcb7d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4385e9a0 0x560f4385edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dcb7d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f43656310 con 0x560f4385e9a0 2024-04-27T03:18:04.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dae99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4385e9a0 0x560f4385edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dae99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4385e9a0 0x560f4385edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49896/0 (socket says 172.21.15.110:49896) 2024-04-27T03:18:04.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dae99640 1 -- 172.21.15.110:0/4137528181 learned_addr learned my addr 172.21.15.110:0/4137528181 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dae99640 1 -- 172.21.15.110:0/4137528181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f4385f300 con 0x560f4385e9a0 2024-04-27T03:18:04.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dae99640 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4385e9a0 0x560f4385edc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x560f438448a0 tx=0x7f81d400af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f80bffff640 1 -- 172.21.15.110:0/4137528181 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4007ce0 con 0x560f4385e9a0 2024-04-27T03:18:04.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f80bffff640 1 -- 172.21.15.110:0/4137528181 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d401d070 con 0x560f4385e9a0 2024-04-27T03:18:04.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f4385fda0 con 0x560f4385e9a0 2024-04-27T03:18:04.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f80bffff640 1 -- 172.21.15.110:0/4137528181 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4018440 con 0x560f4385e9a0 2024-04-27T03:18:04.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f43860270 con 0x560f4385e9a0 2024-04-27T03:18:04.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.365+0000 7f80bffff640 1 -- 172.21.15.110:0/4137528181 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f81d4013070 con 0x560f4385e9a0 2024-04-27T03:18:04.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f80bffff640 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80b4034650 0x7f80b4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f80bffff640 1 -- 172.21.15.110:0/4137528181 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f81d404b480 con 0x560f4385e9a0 2024-04-27T03:18:04.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81da698640 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80b4034650 0x7f80b4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 Processor -- start 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- start start 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81da698640 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80b4034650 0x7f80b4036b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x560f4384ed60 tx=0x7f81cc0e3950 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f4386bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f4374fd40 con 0x560f4386bb20 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dae99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f4386bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dae99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f4386bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49898/0 (socket says 172.21.15.110:49898) 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dae99640 1 -- 172.21.15.110:0/2064708734 learned_addr learned my addr 172.21.15.110:0/2064708734 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dae99640 1 -- 172.21.15.110:0/2064708734 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f4386c510 con 0x560f4386bb20 2024-04-27T03:18:04.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dae99640 1 --2- 172.21.15.110:0/2064708734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f4386bf40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f81d4049890 tx=0x7f81d4017620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d6a1d6a2de45e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f80bdffb640 1 -- 172.21.15.110:0/2064708734 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d403b260 con 0x560f4386bb20 2024-04-27T03:18:04.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f80bdffb640 1 -- 172.21.15.110:0/2064708734 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d4022460 con 0x560f4386bb20 2024-04-27T03:18:04.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f80bdffb640 1 -- 172.21.15.110:0/2064708734 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4020360 con 0x560f4386bb20 2024-04-27T03:18:04.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2064708734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 msgr2=0x560f4386bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2064708734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f4386bf40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f81d4049890 tx=0x7f81d4017620 comp rx=0 tx=0).stop 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2064708734 shutdown_connections 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2064708734 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f4386bf40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2064708734 >> 172.21.15.110:0/2064708734 conn(0x560f43867eb0 msgr2=0x560f43865f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2064708734 shutdown_connections 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2064708734 wait complete. 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 Processor -- start 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- start start 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f43872fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.369+0000 7f81dcb7d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f4386cc60 con 0x560f4386bb20 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dae99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f43872fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dae99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f43872fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49912/0 (socket says 172.21.15.110:49912) 2024-04-27T03:18:04.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 learned_addr learned my addr 172.21.15.110:0/2032131517 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f438734f0 con 0x560f4386bb20 2024-04-27T03:18:04.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dae99640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f43872fb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f81d4017660 tx=0x7f81d4049820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f80abfff640 1 -- 172.21.15.110:0/2032131517 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4031070 con 0x560f4386bb20 2024-04-27T03:18:04.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f80abfff640 1 -- 172.21.15.110:0/2032131517 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d4017a80 con 0x560f4386bb20 2024-04-27T03:18:04.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f43873780 con 0x560f4386bb20 2024-04-27T03:18:04.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f80abfff640 1 -- 172.21.15.110:0/2032131517 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4017d60 con 0x560f4386bb20 2024-04-27T03:18:04.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f43876460 con 0x560f4386bb20 2024-04-27T03:18:04.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f80abfff640 1 -- 172.21.15.110:0/2032131517 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f81d4034070 con 0x560f4386bb20 2024-04-27T03:18:04.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f80abfff640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80a4034650 0x7f80a4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f80abfff640 1 -- 172.21.15.110:0/2032131517 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f81d4037c90 con 0x560f4386bb20 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f4387e610 0x560f43880a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81da698640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80a4034650 0x7f80a4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f438811d0 con 0x560f4387e610 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81d9e97640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f4387e610 0x560f43880a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.373+0000 7f81da698640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80a4034650 0x7f80a4036b10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0bfe90 tx=0x7f81cc0e7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81d9e97640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f4387e610 0x560f43880a80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81d000e5a0 con 0x560f4387e610 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 realm 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4387db70 con 0x560f4387e610 2024-04-27T03:18:04.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81d000e5a0 con 0x560f4387e610 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43881cd0 con 0x560f4387e610 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81d000e5a0 con 0x560f4387e610 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f43882ab0 0x560f43884ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43885520 con 0x560f43882ab0 2024-04-27T03:18:04.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dae99640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f43882ab0 0x560f43884ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dae99640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f43882ab0 0x560f43884ea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:04.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43885dc0 con 0x560f4387e610 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f81d000e5a0 con 0x560f4387e610 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 20 searching for the correct realm 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f43887b30 0x560f43889f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43887b30 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81da698640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f43887b30 0x560f43889f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.377+0000 7f81da698640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f43887b30 0x560f43889f20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81dae99640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x7f81d4036070 2024-04-27T03:18:04.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d402d440 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d000e5a0 con 0x7f81d4036070 2024-04-27T03:18:04.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d000e5a0 con 0x560f4387e610 2024-04-27T03:18:04.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43887b30 2024-04-27T03:18:04.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.381+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x7f81d4036070 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x7f81d4036070 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x7f81d4036070 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f81d0029040 con 0x7f81d4036070 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43887b30 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v20'1 uv1 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.385+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43887b30 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43887b30 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x7f81d4036070 2024-04-27T03:18:04.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x7f81d4036070 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f4387e610 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x7f81d4036070 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x7f81d4036070 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43887b30 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x7f81d4036070 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d0029040 con 0x7f81d4036070 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.389+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43887b30 2024-04-27T03:18:04.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a500 con 0x560f43882ab0 2024-04-27T03:18:04.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:04.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:04.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:04.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:04.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:04.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81d0029040 con 0x560f4387e610 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43886bc0 con 0x560f43882ab0 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43887020 con 0x560f4387e610 2024-04-27T03:18:04.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:04.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 zone default found 2024-04-27T03:18:04.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 4 Realm: () 2024-04-27T03:18:04.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:04.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 zonegroup default 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4388a960 con 0x7f81d4036070 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.393+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4388c7f0 con 0x560f43887b30 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f4388cb90 con 0x560f43887b30 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 20 started sync module instance, tier type = 2024-04-27T03:18:04.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:04.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388bc30 con 0x560f43882ab0 2024-04-27T03:18:04.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94623557531520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388cf30 con 0x560f43882ab0 2024-04-27T03:18:04.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.397+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388d8f0 con 0x560f4387e610 2024-04-27T03:18:04.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94623557538880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388dd10 con 0x560f4387e610 2024-04-27T03:18:04.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388e750 con 0x560f43882ab0 2024-04-27T03:18:04.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94623557531520] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94623557538880] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94623557542432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388d860 con 0x560f43882ab0 2024-04-27T03:18:04.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388eff0 con 0x7f81d4036070 2024-04-27T03:18:04.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94623557544768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388d860 con 0x7f81d4036070 2024-04-27T03:18:04.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.401+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388fab0 con 0x560f43882ab0 2024-04-27T03:18:04.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94623557542432] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94623557544768] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94623557545824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388fa70 con 0x560f43882ab0 2024-04-27T03:18:04.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388fea0 con 0x560f43887b30 2024-04-27T03:18:04.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94623557533728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43890740 con 0x560f43887b30 2024-04-27T03:18:04.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891140 con 0x560f43887b30 2024-04-27T03:18:04.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.405+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94623557545824] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94623557533728] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94623557547392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891530 con 0x560f43887b30 2024-04-27T03:18:04.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891d80 con 0x7f81d4036070 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'3 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94623557555984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f438921b0 con 0x7f81d4036070 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=0 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=1 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=2 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=3 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=4 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=5 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94623557547392] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=6 2024-04-27T03:18:04.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94623557555984] v27'4 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 20 add_watcher() i=7 2024-04-27T03:18:04.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:04.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.409+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f43890c50 con 0x560f4387e610 2024-04-27T03:18:04.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94623557553232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f438926f0 con 0x560f4387e610 2024-04-27T03:18:04.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94623557553232] v27'30 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f8090fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:04.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81dcb7d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:04.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81dcb7d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:04.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81dcb7d980 5 note: GC not initialized 2024-04-27T03:18:04.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f800b7fe640 20 reqs_thread_entry: start 2024-04-27T03:18:04.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81dcb7d980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:04.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81dcb7d980 20 Filter name: none 2024-04-27T03:18:04.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f80097fa640 20 reqs_thread_entry: start 2024-04-27T03:18:04.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.413+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:62 8.0 8:00000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f438b7fd0 con 0x560f43887b30 2024-04-27T03:18:04.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(62 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:63 8.10 8:08000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43891140 con 0x560f43882ab0 2024-04-27T03:18:04.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(63 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.8 8:10000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f4388bc30 con 0x560f4387e610 2024-04-27T03:18:04.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 8.18 8:18000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(65 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.4 8:20000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x7f81d4036070 2024-04-27T03:18:04.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(66 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 8.14 8:28000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43882ab0 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 8.c 8:30000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 8.1c 8:38000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(69 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.417+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 8.2 8:40000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x7f81d4036070 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 8.12 8:48000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43882ab0 2024-04-27T03:18:04.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(71 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 8.a 8:50000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(72 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 8.1a 8:58000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:74 8.6 8:60000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(74 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 8.16 8:68000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(75 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 8.e 8:70000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x7f81d4036070 2024-04-27T03:18:04.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(76 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 8.1e 8:78000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(77 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:78 8.1 8:80000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(78 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.421+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:79 8.11 8:88000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43882ab0 2024-04-27T03:18:04.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(79 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:80 8.9 8:90000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x7f81d4036070 2024-04-27T03:18:04.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(80 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 8.19 8:98000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43882ab0 2024-04-27T03:18:04.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(81 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:82 8.5 8:a0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(82 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:83 8.15 8:a8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(83 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:84 8.d 8:b0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x7f81d4036070 2024-04-27T03:18:04.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(84 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 8.1d 8:b8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(85 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:86 8.3 8:c0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(86 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 8.13 8:c8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 8.b 8:d0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:89 8.1b 8:d8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x7f81d4036070 2024-04-27T03:18:04.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(89 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.425+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 8.7 8:e0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f4387e610 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(90 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 8.17 8:e8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43882ab0 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(91 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:92 8.f 8:f0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x560f43887b30 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(92 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 8.1f 8:f8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f43886740 con 0x7f81d4036070 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(93 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f8090fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:04.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.429+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94623557553232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388bc30 con 0x560f4387e610 2024-04-27T03:18:04.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.433+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(94 data_loggenerations_metadata [watch unwatch cookie 94623557553232] v27'31 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.433+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94623557531520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388bc30 con 0x560f43882ab0 2024-04-27T03:18:04.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.433+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(95 notify.0 [watch unwatch cookie 94623557531520] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.433+0000 7f81dcb7d980 20 remove_watcher() i=0 2024-04-27T03:18:04.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.433+0000 7f81dcb7d980 2 removed watcher, disabling cache 2024-04-27T03:18:04.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.433+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94623557538880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891100 con 0x560f4387e610 2024-04-27T03:18:04.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.437+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(96 notify.1 [watch unwatch cookie 94623557538880] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x560f4387e610 2024-04-27T03:18:04.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.437+0000 7f81dcb7d980 20 remove_watcher() i=1 2024-04-27T03:18:04.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.437+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94623557542432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891100 con 0x560f43882ab0 2024-04-27T03:18:04.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.437+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(97 notify.2 [watch unwatch cookie 94623557542432] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.437+0000 7f81dcb7d980 20 remove_watcher() i=2 2024-04-27T03:18:04.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.437+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94623557544768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891100 con 0x7f81d4036070 2024-04-27T03:18:04.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(98 notify.3 [watch unwatch cookie 94623557544768] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81dcb7d980 20 remove_watcher() i=3 2024-04-27T03:18:04.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94623557545824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891100 con 0x560f43882ab0 2024-04-27T03:18:04.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81dae99640 1 -- 172.21.15.110:0/2032131517 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(99 notify.4 [watch unwatch cookie 94623557545824] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560f43882ab0 2024-04-27T03:18:04.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81dcb7d980 20 remove_watcher() i=4 2024-04-27T03:18:04.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94623557533728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f43891100 con 0x560f43887b30 2024-04-27T03:18:04.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(100 notify.5 [watch unwatch cookie 94623557533728] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81dcb7d980 20 remove_watcher() i=5 2024-04-27T03:18:04.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.441+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94623557547392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388d440 con 0x560f43887b30 2024-04-27T03:18:04.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81da698640 1 -- 172.21.15.110:0/2032131517 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(101 notify.6 [watch unwatch cookie 94623557547392] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc0b2070 con 0x560f43887b30 2024-04-27T03:18:04.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 20 remove_watcher() i=6 2024-04-27T03:18:04.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94623557555984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f4388c020 con 0x7f81d4036070 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(102 notify.7 [watch unwatch cookie 94623557555984] v27'5 uv3 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d003c1c0 con 0x7f81d4036070 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 20 remove_watcher() i=7 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f43882ab0 msgr2=0x560f43884ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f43882ab0 0x560f43884ea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f4387e610 msgr2=0x560f43880a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f4387e610 0x560f43880a80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 msgr2=0x7f81d402d440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d402d440 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f43887b30 msgr2=0x560f43889f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f43887b30 0x560f43889f20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80a4034650 msgr2=0x7f80a4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80a4034650 0x7f80a4036b10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81cc0bfe90 tx=0x7f81cc0e7000 comp rx=0 tx=0).stop 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 msgr2=0x560f43872fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81d9e97640 1 -- 172.21.15.110:0/2032131517 reap_dead start 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f43872fb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f81d4017660 tx=0x7f81d4049820 comp rx=0 tx=0).stop 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 shutdown_connections 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/2032131517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4386bb20 0x560f43872fb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 >> 172.21.15.110:0/2032131517 conn(0x560f43867eb0 msgr2=0x560f4386d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 shutdown_connections 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/2032131517 wait complete. 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80b4034650 msgr2=0x7f80b4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80b4034650 0x7f80b4036b10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x560f4384ed60 tx=0x7f81cc0e3950 comp rx=0 tx=0).stop 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4385e9a0 msgr2=0x560f4385edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4385e9a0 0x560f4385edc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x560f438448a0 tx=0x7f81d400af40 comp rx=0 tx=0).stop 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 shutdown_connections 2024-04-27T03:18:04.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80b4034650 0x7f80b4036b10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/4137528181 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f4385e9a0 0x560f4385edc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 >> 172.21.15.110:0/4137528181 conn(0x560f438537b0 msgr2=0x560f438542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 shutdown_connections 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.445+0000 7f81dcb7d980 1 -- 172.21.15.110:0/4137528181 wait complete. 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f4384bd30 msgr2=0x560f4384e120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f4384bd30 0x560f4384e120 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f438486a0 msgr2=0x560f4384aa90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f438486a0 0x560f4384aa90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c0034650 msgr2=0x7f80c0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c0034650 0x7f80c0036b10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f81d0000f70 tx=0x7f81d00012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 msgr2=0x560f438437d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f438437d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f81cc018c00 tx=0x7f81cc017d70 comp rx=0 tx=0).stop 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 shutdown_connections 2024-04-27T03:18:04.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f4384bd30 0x560f4384e120 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f438486a0 0x560f4384aa90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c0034650 0x7f80c0036b10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 --2- 172.21.15.110:0/3751934948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f43752490 0x560f438437d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 >> 172.21.15.110:0/3751934948 conn(0x560f43749a30 msgr2=0x560f4374e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 shutdown_connections 2024-04-27T03:18:04.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.449+0000 7f81dcb7d980 1 -- 172.21.15.110:0/3751934948 wait complete. 2024-04-27T03:18:04.466 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2024-04-27T03:18:04.467 DEBUG:tasks.util.rgw: json result: [] 2024-04-27T03:18:04.467 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'bud'] 2024-04-27T03:18:04.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'bud'] 2024-04-27T03:18:04.467 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid bud 2024-04-27T03:18:04.544 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:04.544 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:04.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b1bfc2980 1 Processor -- start 2024-04-27T03:18:04.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b1bfc2980 1 -- start start 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b1bfc2980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5624c7916890 0x5624c79a7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b1bfc2980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5624c78610d0 con 0x5624c79a7840 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b1bfc2980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5624c7856430 con 0x5624c7916890 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b19add640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5624c7916890 0x5624c79a7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b19add640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5624c7916890 0x5624c79a7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49920/0 (socket says 172.21.15.110:49920) 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b192dc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5624c79a7840 0x5624c79a9c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34520/0 (socket says 172.21.15.110:34520) 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b19add640 1 -- 172.21.15.110:0/3723261699 learned_addr learned my addr 172.21.15.110:0/3723261699 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b19add640 1 -- 172.21.15.110:0/3723261699 >> v1:172.21.15.77:6789/0 conn(0x5624c79a7840 legacy=0x5624c79a9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.537+0000 7f8b19add640 1 -- 172.21.15.110:0/3723261699 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c784f510 con 0x5624c7916890 2024-04-27T03:18:04.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b19add640 1 --2- 172.21.15.110:0/3723261699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c79a7300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00a5c0 tx=0x7f8b0c01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fcaec1ae77c6fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b18adb640 1 -- 172.21.15.110:0/3723261699 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0a6070 con 0x5624c7916890 2024-04-27T03:18:04.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b18adb640 1 -- 172.21.15.110:0/3723261699 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b0c0af070 con 0x5624c7916890 2024-04-27T03:18:04.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b18adb640 1 -- 172.21.15.110:0/3723261699 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0aa5c0 con 0x5624c7916890 2024-04-27T03:18:04.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3723261699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 msgr2=0x5624c79a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/3723261699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c79a7300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00a5c0 tx=0x7f8b0c01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:04.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3723261699 shutdown_connections 2024-04-27T03:18:04.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/3723261699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c79a7300 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3723261699 >> 172.21.15.110:0/3723261699 conn(0x5624c799ca20 msgr2=0x5624c799ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3723261699 shutdown_connections 2024-04-27T03:18:04.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.541+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3723261699 wait complete. 2024-04-27T03:18:04.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b1bfc2980 1 Processor -- start 2024-04-27T03:18:04.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b1bfc2980 1 -- start start 2024-04-27T03:18:04.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b1bfc2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a8f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b1bfc2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624c78a52d0 con 0x5624c7916890 2024-04-27T03:18:04.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b19add640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a8f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b19add640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a8f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49926/0 (socket says 172.21.15.110:49926) 2024-04-27T03:18:04.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b19add640 1 -- 172.21.15.110:0/1384008645 learned_addr learned my addr 172.21.15.110:0/1384008645 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b19add640 1 -- 172.21.15.110:0/1384008645 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c782bae0 con 0x5624c7916890 2024-04-27T03:18:04.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8b19add640 1 --2- 172.21.15.110:0/1384008645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a8f630 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00a5c0 tx=0x7f8b0c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32a67805d3b1f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8a0760e640 1 -- 172.21.15.110:0/1384008645 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0aa9b0 con 0x5624c7916890 2024-04-27T03:18:04.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8a0760e640 1 -- 172.21.15.110:0/1384008645 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b0c0aab50 con 0x5624c7916890 2024-04-27T03:18:04.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.557+0000 7f8a0760e640 1 -- 172.21.15.110:0/1384008645 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0aae50 con 0x5624c7916890 2024-04-27T03:18:04.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1384008645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 msgr2=0x5624c7a8f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1384008645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a8f630 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00a5c0 tx=0x7f8b0c003030 comp rx=0 tx=0).stop 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1384008645 shutdown_connections 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1384008645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a8f630 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1384008645 >> 172.21.15.110:0/1384008645 conn(0x5624c799ca20 msgr2=0x5624c7a8b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1384008645 shutdown_connections 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1384008645 wait complete. 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 Processor -- start 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- start start 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a96610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624c79ade10 con 0x5624c7916890 2024-04-27T03:18:04.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b19add640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a96610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b19add640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a96610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49930/0 (socket says 172.21.15.110:49930) 2024-04-27T03:18:04.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b19add640 1 -- 172.21.15.110:0/2894614155 learned_addr learned my addr 172.21.15.110:0/2894614155 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b19add640 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c7a96b50 con 0x5624c7916890 2024-04-27T03:18:04.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b19add640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a96610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c01b380 tx=0x7f8b0c0180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8a05e0b640 1 -- 172.21.15.110:0/2894614155 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0c6750 con 0x5624c7916890 2024-04-27T03:18:04.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5624c7a96de0 con 0x5624c7916890 2024-04-27T03:18:04.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8a05e0b640 1 -- 172.21.15.110:0/2894614155 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b0c0cc070 con 0x5624c7916890 2024-04-27T03:18:04.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8a05e0b640 1 -- 172.21.15.110:0/2894614155 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0c6a20 con 0x5624c7916890 2024-04-27T03:18:04.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.561+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c7a99a10 con 0x5624c7916890 2024-04-27T03:18:04.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8a05e0b640 1 -- 172.21.15.110:0/2894614155 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8b0c0c6bc0 con 0x5624c7916890 2024-04-27T03:18:04.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8a05e0b640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8a000346a0 0x7f8a00036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8a05e0b640 1 -- 172.21.15.110:0/2894614155 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f8b0c0e5740 con 0x5624c7916890 2024-04-27T03:18:04.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b192dc640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8a000346a0 0x7f8a00036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c79a7840 0x5624c7a9d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7a9da30 con 0x5624c79a7840 2024-04-27T03:18:04.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c79a7840 0x5624c7a9d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b192dc640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8a000346a0 0x7f8a00036b60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b08000f70 tx=0x7f8b08001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c79a7840 0x5624c7a9d4f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1a2de640 1 -- 172.21.15.110:0/2894614155 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8b1400a4d0 con 0x5624c79a7840 2024-04-27T03:18:04.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7a9e7e0 0x5624c7aa0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7aa1210 con 0x5624c7a9e7e0 2024-04-27T03:18:04.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b19add640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7a9e7e0 0x5624c7aa0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b19add640 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7a9e7e0 0x5624c7aa0bd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b19add640 1 -- 172.21.15.110:0/2894614155 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7a9e7e0 2024-04-27T03:18:04.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7aa2ac0 con 0x5624c79a7840 2024-04-27T03:18:04.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1a2de640 1 -- 172.21.15.110:0/2894614155 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f8b1400a4d0 con 0x5624c79a7840 2024-04-27T03:18:04.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.565+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7aa1cc0 con 0x5624c7a9e7e0 2024-04-27T03:18:04.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b19add640 1 -- 172.21.15.110:0/2894614155 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7a9e7e0 2024-04-27T03:18:04.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7aa20b0 con 0x5624c79a7840 2024-04-27T03:18:04.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1a2de640 1 -- 172.21.15.110:0/2894614155 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f8b1400a4d0 con 0x5624c79a7840 2024-04-27T03:18:04.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1bfc2980 1 Processor -- start 2024-04-27T03:18:04.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1bfc2980 1 -- start start 2024-04-27T03:18:04.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1bfc2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7ab1430 0x5624c7ab1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1bfc2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624c78a9310 con 0x5624c7ab1430 2024-04-27T03:18:04.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1a2de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7ab1430 0x5624c7ab1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1a2de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7ab1430 0x5624c7ab1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49934/0 (socket says 172.21.15.110:49934) 2024-04-27T03:18:04.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1471986048 learned_addr learned my addr 172.21.15.110:0/1471986048 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1471986048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c7ab1d90 con 0x5624c7ab1430 2024-04-27T03:18:04.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7ab1430 0x5624c7ab1850 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b14017680 tx=0x7f8b14017a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f89feffd640 1 -- 172.21.15.110:0/1471986048 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b140157d0 con 0x5624c7ab1430 2024-04-27T03:18:04.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5624c7ab2830 con 0x5624c7ab1430 2024-04-27T03:18:04.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f89feffd640 1 -- 172.21.15.110:0/1471986048 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b14015970 con 0x5624c7ab1430 2024-04-27T03:18:04.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f89feffd640 1 -- 172.21.15.110:0/1471986048 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b14015c50 con 0x5624c7ab1430 2024-04-27T03:18:04.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.569+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c7ab2d00 con 0x5624c7ab1430 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f89feffd640 1 -- 172.21.15.110:0/1471986048 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8b14029440 con 0x5624c7ab1430 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f89feffd640 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89f00346a0 0x7f89f0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f89feffd640 1 -- 172.21.15.110:0/1471986048 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f8b1400e040 con 0x5624c7ab1430 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b19add640 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89f00346a0 0x7f89f0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 Processor -- start 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 -- start start 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7abea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624c7a8fcc0 con 0x5624c7abe630 2024-04-27T03:18:04.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b19add640 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89f00346a0 0x7f89f0036b60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x5624c7aa17e0 tx=0x7f8b0c0e3bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1a2de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7abea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1a2de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7abea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49936/0 (socket says 172.21.15.110:49936) 2024-04-27T03:18:04.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1a2de640 1 -- 172.21.15.110:0/3862931565 learned_addr learned my addr 172.21.15.110:0/3862931565 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1a2de640 1 -- 172.21.15.110:0/3862931565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c7abf000 con 0x5624c7abe630 2024-04-27T03:18:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/3862931565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7abea30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8b1401e8f0 tx=0x7f8b14012500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b64658dcaff6ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f89fcff9640 1 -- 172.21.15.110:0/3862931565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b140284d0 con 0x5624c7abe630 2024-04-27T03:18:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f89fcff9640 1 -- 172.21.15.110:0/3862931565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1404c0d0 con 0x5624c7abe630 2024-04-27T03:18:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f89fcff9640 1 -- 172.21.15.110:0/3862931565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b1401eb90 con 0x5624c7abe630 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3862931565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 msgr2=0x5624c7abea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/3862931565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7abea30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8b1401e8f0 tx=0x7f8b14012500 comp rx=0 tx=0).stop 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3862931565 shutdown_connections 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/3862931565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7abea30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3862931565 >> 172.21.15.110:0/3862931565 conn(0x5624c7aba970 msgr2=0x5624c7ab89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3862931565 shutdown_connections 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.573+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/3862931565 wait complete. 2024-04-27T03:18:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 Processor -- start 2024-04-27T03:18:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 -- start start 2024-04-27T03:18:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7ac5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624c7abf7a0 con 0x5624c7abe630 2024-04-27T03:18:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1a2de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7ac5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1a2de640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7ac5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49950/0 (socket says 172.21.15.110:49950) 2024-04-27T03:18:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 learned_addr learned my addr 172.21.15.110:0/1502172952 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624c7ac5fe0 con 0x5624c7abe630 2024-04-27T03:18:04.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7ac5aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8b1401eb20 tx=0x7f8b1404a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f89eeffd640 1 -- 172.21.15.110:0/1502172952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b1403a070 con 0x5624c7abe630 2024-04-27T03:18:04.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f89eeffd640 1 -- 172.21.15.110:0/1502172952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b14049300 con 0x5624c7abe630 2024-04-27T03:18:04.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5624c7ac6270 con 0x5624c7abe630 2024-04-27T03:18:04.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f89eeffd640 1 -- 172.21.15.110:0/1502172952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b140495e0 con 0x5624c7abe630 2024-04-27T03:18:04.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624c7ac8f50 con 0x5624c7abe630 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f89eeffd640 1 -- 172.21.15.110:0/1502172952 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8b14044040 con 0x5624c7abe630 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f89eeffd640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89e40347f0 0x7f89e4036cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f89eeffd640 1 -- 172.21.15.110:0/1502172952 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f8b14014410 con 0x5624c7abe630 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b19add640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89e40347f0 0x7f89e4036cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c7ad1160 0x5624c7ad3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad3d50 con 0x5624c7ad1160 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.577+0000 7f8b192dc640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c7ad1160 0x5624c7ad3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b19add640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89e40347f0 0x7f89e4036cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0c5fd0 tx=0x7f8b0c0e8000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b192dc640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c7ad1160 0x5624c7ad3600 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 realm 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad0630 con 0x5624c7ad1160 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad4850 con 0x5624c7ad1160 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7ad5630 0x5624c7ad7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad80a0 con 0x5624c7ad5630 2024-04-27T03:18:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7ad5630 0x5624c7ad7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7ad5630 0x5624c7ad7a20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad88f0 con 0x5624c7ad1160 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 20 searching for the correct realm 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624c7ada660 0x5624c7adca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ada660 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b19add640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624c7ada660 0x5624c7adca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.581+0000 7f8b19add640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624c7ada660 0x5624c7adca50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad5630 2024-04-27T03:18:04.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b1a2de640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8b14057040 0x7f8b1401a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x7f8b14057040 2024-04-27T03:18:04.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8b14057040 0x7f8b1401a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8b14057040 0x7f8b1401a9b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ada660 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad5630 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.585+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad5630 2024-04-27T03:18:04.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x7f8b14057040 2024-04-27T03:18:04.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x7f8b14057040 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad5630 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ada660 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v20'1 uv1 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad5630 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.589+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ada660 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ada660 2024-04-27T03:18:04.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x7f8b14057040 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x5624c7ad1160 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5624c7add080 con 0x7f8b14057040 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f8b0803e040 con 0x5624c7ada660 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f8b0803e040 con 0x7f8b14057040 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f8b0803e040 con 0x5624c7ad5630 2024-04-27T03:18:04.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.593+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f8b0803e040 con 0x5624c7ad5630 2024-04-27T03:18:04.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f8b0803e040 con 0x5624c7ada660 2024-04-27T03:18:04.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f8b0803e040 con 0x5624c7ad5630 2024-04-27T03:18:04.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f8b0803e040 con 0x5624c7ad5630 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:04.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad9270 con 0x5624c7ad1160 2024-04-27T03:18:04.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad8eb0 con 0x5624c7ad5630 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.597+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad8ec0 con 0x5624c7ad1160 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 zone default found 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 4 Realm: () 2024-04-27T03:18:04.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 zonegroup default 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ad9da0 con 0x7f8b14057040 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7adf8e0 con 0x5624c7ada660 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7adfc80 con 0x5624c7ada660 2024-04-27T03:18:04.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 started sync module instance, tier type = 2024-04-27T03:18:04.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:04.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7aded20 con 0x5624c7ad5630 2024-04-27T03:18:04.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94715968877120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7aded20 con 0x5624c7ad5630 2024-04-27T03:18:04.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.601+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae0640 con 0x5624c7ad1160 2024-04-27T03:18:04.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94715968881040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae0ee0 con 0x5624c7ad1160 2024-04-27T03:18:04.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae18e0 con 0x5624c7ad5630 2024-04-27T03:18:04.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94715968877120] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94715968881040] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94715968874608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae0a30 con 0x5624c7ad5630 2024-04-27T03:18:04.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae0a30 con 0x7f8b14057040 2024-04-27T03:18:04.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.605+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.609+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94715968888016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae0a30 con 0x7f8b14057040 2024-04-27T03:18:04.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.609+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae2c20 con 0x5624c7ad5630 2024-04-27T03:18:04.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.609+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94715968874608] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.609+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94715968888016] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.609+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.609+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94715968889072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae2be0 con 0x5624c7ad5630 2024-04-27T03:18:04.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.609+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae3010 con 0x5624c7ada660 2024-04-27T03:18:04.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94715968885744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae38b0 con 0x5624c7ada660 2024-04-27T03:18:04.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae42b0 con 0x5624c7ada660 2024-04-27T03:18:04.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94715968889072] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94715968885744] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94715968890640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae46a0 con 0x5624c7ada660 2024-04-27T03:18:04.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae4f30 con 0x7f8b14057040 2024-04-27T03:18:04.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94715968890640] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'6 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94715968896448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae4b50 con 0x7f8b14057040 2024-04-27T03:18:04.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 20 add_watcher() i=0 2024-04-27T03:18:04.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 20 add_watcher() i=1 2024-04-27T03:18:04.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 20 add_watcher() i=2 2024-04-27T03:18:04.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 20 add_watcher() i=3 2024-04-27T03:18:04.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 20 add_watcher() i=4 2024-04-27T03:18:04.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 20 add_watcher() i=5 2024-04-27T03:18:04.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.613+0000 7f8b1bfc2980 20 add_watcher() i=6 2024-04-27T03:18:04.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94715968896448] v27'7 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b1bfc2980 20 add_watcher() i=7 2024-04-27T03:18:04.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b1bfc2980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:04.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae83e0 con 0x5624c7ad1160 2024-04-27T03:18:04.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94715968899184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae58a0 con 0x5624c7ad1160 2024-04-27T03:18:04.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94715968899184] v27'32 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f89cf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:04.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b1bfc2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:04.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.617+0000 7f8b1bfc2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:04.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.621+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae8210 con 0x5624c7ad5630 2024-04-27T03:18:04.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.621+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'40 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.621+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae8340 con 0x7f8b14057040 2024-04-27T03:18:04.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.621+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'16 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.621+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af16c0 con 0x5624c7ada660 2024-04-27T03:18:04.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.625+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'16 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.625+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af1b70 con 0x5624c7ad1160 2024-04-27T03:18:04.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.625+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'33 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.625+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af0ac0 con 0x5624c7ad5630 2024-04-27T03:18:04.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.629+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'28 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.629+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af0ac0 con 0x5624c7ada660 2024-04-27T03:18:04.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.629+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'25 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.629+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af0ac0 con 0x5624c7ad5630 2024-04-27T03:18:04.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.633+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'18 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.633+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af0c90 con 0x5624c7ad1160 2024-04-27T03:18:04.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.633+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'20 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.633+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af26d0 con 0x5624c7ad5630 2024-04-27T03:18:04.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.637+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'10 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.637+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af26d0 con 0x5624c7ad5630 2024-04-27T03:18:04.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.637+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'16 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.637+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af26d0 con 0x5624c7ad5630 2024-04-27T03:18:04.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.641+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'29 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.641+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af26d0 con 0x5624c7ada660 2024-04-27T03:18:04.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.641+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'12 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.641+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2760 con 0x5624c7ad5630 2024-04-27T03:18:04.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.645+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'16 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.645+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2760 con 0x5624c7ad1160 2024-04-27T03:18:04.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.645+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'31 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.645+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af27f0 con 0x5624c7ad1160 2024-04-27T03:18:04.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.649+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'34 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.649+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af29c0 con 0x5624c7ad5630 2024-04-27T03:18:04.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.649+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'20 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.649+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af29c0 con 0x5624c7ad5630 2024-04-27T03:18:04.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.653+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'30 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.653+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2860 con 0x5624c7ada660 2024-04-27T03:18:04.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.653+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'22 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.653+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad5630 2024-04-27T03:18:04.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.657+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'18 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.657+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad1160 2024-04-27T03:18:04.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.657+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'35 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.657+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af0440 con 0x5624c7ad5630 2024-04-27T03:18:04.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.661+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'21 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.661+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad5630 2024-04-27T03:18:04.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.661+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'26 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.661+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ada660 2024-04-27T03:18:04.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.665+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'18 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.665+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad5630 2024-04-27T03:18:04.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.665+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'28 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.665+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ada660 2024-04-27T03:18:04.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.669+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'12 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.669+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ada660 2024-04-27T03:18:04.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.669+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'26 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.669+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ada660 2024-04-27T03:18:04.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.673+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'18 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.673+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad5630 2024-04-27T03:18:04.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.677+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'22 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.677+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad1160 2024-04-27T03:18:04.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.677+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'21 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.677+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad5630 2024-04-27T03:18:04.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.681+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'22 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.681+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x5624c7ad1160 2024-04-27T03:18:04.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.681+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'32 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.681+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7af2e20 con 0x7f8b14057040 2024-04-27T03:18:04.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'16 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f89ce7e4640 20 reqs_thread_entry: start 2024-04-27T03:18:04.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:04.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 20 Filter name: none 2024-04-27T03:18:04.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f893ffff640 20 reqs_thread_entry: start 2024-04-27T03:18:04.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 10 cache get: name=default.rgw.meta+users.uid+bud : miss 2024-04-27T03:18:04.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.4 8:267425a8:users.uid::bud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5624c7b0de50 con 0x7f8b14057040 2024-04-27T03:18:04.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(94 bud [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x0 2024-04-27T03:18:04.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2024-04-27T03:18:04.694 INFO:teuthology.orchestra.run.smithi110.stderr:could not remove user: unable to remove user, user does not exist 2024-04-27T03:18:04.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f89cf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:04.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.685+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94715968899184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7aee740 con 0x5624c7ad1160 2024-04-27T03:18:04.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.689+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(95 data_loggenerations_metadata [watch unwatch cookie 94715968899184] v27'36 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.689+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94715968877120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae4840 con 0x5624c7ad5630 2024-04-27T03:18:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(96 notify.0 [watch unwatch cookie 94715968877120] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1bfc2980 20 remove_watcher() i=0 2024-04-27T03:18:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1bfc2980 2 removed watcher, disabling cache 2024-04-27T03:18:04.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:97 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94715968881040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7adf240 con 0x5624c7ad1160 2024-04-27T03:18:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(97 notify.1 [watch unwatch cookie 94715968881040] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x5624c7ad1160 2024-04-27T03:18:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1bfc2980 20 remove_watcher() i=1 2024-04-27T03:18:04.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94715968874608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7adf240 con 0x5624c7ad5630 2024-04-27T03:18:04.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(98 notify.2 [watch unwatch cookie 94715968874608] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1bfc2980 20 remove_watcher() i=2 2024-04-27T03:18:04.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.693+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:99 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94715968888016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7adf240 con 0x7f8b14057040 2024-04-27T03:18:04.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.697+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(99 notify.3 [watch unwatch cookie 94715968888016] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.697+0000 7f8b1bfc2980 20 remove_watcher() i=3 2024-04-27T03:18:04.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.697+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94715968889072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae1cd0 con 0x5624c7ad5630 2024-04-27T03:18:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.697+0000 7f8b1a2de640 1 -- 172.21.15.110:0/1502172952 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(100 notify.4 [watch unwatch cookie 94715968889072] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b14026020 con 0x5624c7ad5630 2024-04-27T03:18:04.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.697+0000 7f8b1bfc2980 20 remove_watcher() i=4 2024-04-27T03:18:04.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.697+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94715968885744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae2130 con 0x5624c7ada660 2024-04-27T03:18:04.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(101 notify.5 [watch unwatch cookie 94715968885744] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 20 remove_watcher() i=5 2024-04-27T03:18:04.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94715968890640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae2130 con 0x5624c7ada660 2024-04-27T03:18:04.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b19add640 1 -- 172.21.15.110:0/1502172952 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(102 notify.6 [watch unwatch cookie 94715968890640] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0c0c5d00 con 0x5624c7ada660 2024-04-27T03:18:04.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 20 remove_watcher() i=6 2024-04-27T03:18:04.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:103 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94715968896448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5624c7ae2130 con 0x7f8b14057040 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(103 notify.7 [watch unwatch cookie 94715968896448] v27'8 uv6 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8b0800f620 con 0x7f8b14057040 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 20 remove_watcher() i=7 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7ad5630 msgr2=0x5624c7ad7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7ad5630 0x5624c7ad7a20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c7ad1160 msgr2=0x5624c7ad3600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c7ad1160 0x5624c7ad3600 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8b14057040 msgr2=0x7f8b1401a9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8b14057040 0x7f8b1401a9b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624c7ada660 msgr2=0x5624c7adca50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624c7ada660 0x5624c7adca50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89e40347f0 msgr2=0x7f89e4036cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89e40347f0 0x7f89e4036cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0c5fd0 tx=0x7f8b0c0e8000 comp rx=0 tx=0).stop 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 msgr2=0x5624c7ac5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1502172952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7abe630 0x5624c7ac5aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8b1401eb20 tx=0x7f8b1404a870 comp rx=0 tx=0).stop 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 reap_dead start 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b192dc640 1 -- 172.21.15.110:0/1502172952 reap_dead start 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.701+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 shutdown_connections 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 >> 172.21.15.110:0/1502172952 conn(0x5624c7aba970 msgr2=0x5624c7ab7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 shutdown_connections 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1502172952 wait complete. 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89f00346a0 msgr2=0x7f89f0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89f00346a0 0x7f89f0036b60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x5624c7aa17e0 tx=0x7f8b0c0e3bf0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7ab1430 msgr2=0x5624c7ab1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7ab1430 0x5624c7ab1850 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b14017680 tx=0x7f8b14017a80 comp rx=0 tx=0).stop 2024-04-27T03:18:04.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 shutdown_connections 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f89f00346a0 0x7f89f0036b60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/1471986048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7ab1430 0x5624c7ab1850 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 >> 172.21.15.110:0/1471986048 conn(0x5624c7aa6240 msgr2=0x5624c7aa6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 shutdown_connections 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/1471986048 wait complete. 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7a9e7e0 msgr2=0x5624c7aa0bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7a9e7e0 0x5624c7aa0bd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c79a7840 msgr2=0x5624c7a9d4f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c79a7840 0x5624c7a9d4f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8a000346a0 msgr2=0x7f8a00036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8a000346a0 0x7f8a00036b60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8b08000f70 tx=0x7f8b08001380 comp rx=0 tx=0).stop 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 msgr2=0x5624c7a96610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a96610 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c01b380 tx=0x7f8b0c0180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 shutdown_connections 2024-04-27T03:18:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624c7a9e7e0 0x5624c7aa0bd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624c79a7840 0x5624c7a9d4f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8a000346a0 0x7f8a00036b60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 --2- 172.21.15.110:0/2894614155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624c7916890 0x5624c7a96610 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 >> 172.21.15.110:0/2894614155 conn(0x5624c799ca20 msgr2=0x5624c7a89cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 shutdown_connections 2024-04-27T03:18:04.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.705+0000 7f8b1bfc2980 1 -- 172.21.15.110:0/2894614155 wait complete. 2024-04-27T03:18:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-27T03:18:04.724 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'bar'] 2024-04-27T03:18:04.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'bar'] 2024-04-27T03:18:04.725 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid bar 2024-04-27T03:18:04.801 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f9026f85980 1 Processor -- start 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f9026f85980 1 -- start start 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f9026f85980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a3ca3f79c0 0x55a3ca3f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f9026f85980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a3ca2b00d0 con 0x55a3ca3f34b0 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f9026f85980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a3ca2a5430 con 0x55a3ca3f79c0 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f902429f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a3ca3f79c0 0x55a3ca3f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f902429f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a3ca3f79c0 0x55a3ca3f9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49952/0 (socket says 172.21.15.110:49952) 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f9024aa0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55a3ca3f34b0 0x55a3ca3f3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34528/0 (socket says 172.21.15.110:34528) 2024-04-27T03:18:04.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f902429f640 1 -- 172.21.15.110:0/3053396519 learned_addr learned my addr 172.21.15.110:0/3053396519 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f902429f640 1 -- 172.21.15.110:0/3053396519 >> v1:172.21.15.77:6789/0 conn(0x55a3ca3f34b0 legacy=0x55a3ca3f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.793+0000 7f902429f640 1 -- 172.21.15.110:0/3053396519 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3ca29e510 con 0x55a3ca3f79c0 2024-04-27T03:18:04.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f902429f640 1 --2- 172.21.15.110:0/3053396519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca3f9db0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f901800a5c0 tx=0x7f901801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420d63bc71e952b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9023a9e640 1 -- 172.21.15.110:0/3053396519 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90180a6070 con 0x55a3ca3f79c0 2024-04-27T03:18:04.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9023a9e640 1 -- 172.21.15.110:0/3053396519 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90180af070 con 0x55a3ca3f79c0 2024-04-27T03:18:04.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9023a9e640 1 -- 172.21.15.110:0/3053396519 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90180aa5c0 con 0x55a3ca3f79c0 2024-04-27T03:18:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9026f85980 1 -- 172.21.15.110:0/3053396519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 msgr2=0x55a3ca3f9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9026f85980 1 --2- 172.21.15.110:0/3053396519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca3f9db0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f901800a5c0 tx=0x7f901801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9026f85980 1 -- 172.21.15.110:0/3053396519 shutdown_connections 2024-04-27T03:18:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9026f85980 1 --2- 172.21.15.110:0/3053396519 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca3f9db0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9026f85980 1 -- 172.21.15.110:0/3053396519 >> 172.21.15.110:0/3053396519 conn(0x55a3ca366b90 msgr2=0x55a3ca366fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9026f85980 1 -- 172.21.15.110:0/3053396519 shutdown_connections 2024-04-27T03:18:04.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.797+0000 7f9026f85980 1 -- 172.21.15.110:0/3053396519 wait complete. 2024-04-27T03:18:04.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9026f85980 1 Processor -- start 2024-04-27T03:18:04.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9026f85980 1 -- start start 2024-04-27T03:18:04.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9026f85980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f34b0 0x55a3ca4de5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9026f85980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a3ca2f42d0 con 0x55a3ca3f34b0 2024-04-27T03:18:04.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9024aa0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f34b0 0x55a3ca4de5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9024aa0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f34b0 0x55a3ca4de5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49960/0 (socket says 172.21.15.110:49960) 2024-04-27T03:18:04.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9024aa0640 1 -- 172.21.15.110:0/2660626241 learned_addr learned my addr 172.21.15.110:0/2660626241 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9024aa0640 1 -- 172.21.15.110:0/2660626241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3ca27aae0 con 0x55a3ca3f34b0 2024-04-27T03:18:04.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f9024aa0640 1 --2- 172.21.15.110:0/2660626241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f34b0 0x55a3ca4de5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9014001690 tx=0x7f9014001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2356a5278d93744 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f8f12e0d640 1 -- 172.21.15.110:0/2660626241 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9014003460 con 0x55a3ca3f34b0 2024-04-27T03:18:04.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f8f12e0d640 1 -- 172.21.15.110:0/2660626241 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9014003600 con 0x55a3ca3f34b0 2024-04-27T03:18:04.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.813+0000 7f8f12e0d640 1 -- 172.21.15.110:0/2660626241 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9014003920 con 0x55a3ca3f34b0 2024-04-27T03:18:04.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- 172.21.15.110:0/2660626241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f34b0 msgr2=0x55a3ca4de5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 --2- 172.21.15.110:0/2660626241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f34b0 0x55a3ca4de5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9014001690 tx=0x7f9014001b50 comp rx=0 tx=0).stop 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- 172.21.15.110:0/2660626241 shutdown_connections 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 --2- 172.21.15.110:0/2660626241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f34b0 0x55a3ca4de5d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- 172.21.15.110:0/2660626241 >> 172.21.15.110:0/2660626241 conn(0x55a3ca366b90 msgr2=0x55a3ca4da160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- 172.21.15.110:0/2660626241 shutdown_connections 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- 172.21.15.110:0/2660626241 wait complete. 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 Processor -- start 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- start start 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca4e5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a3ca3fcde0 con 0x55a3ca3f79c0 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9024aa0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca4e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9024aa0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca4e5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49962/0 (socket says 172.21.15.110:49962) 2024-04-27T03:18:04.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9024aa0640 1 -- 172.21.15.110:0/1727756174 learned_addr learned my addr 172.21.15.110:0/1727756174 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9024aa0640 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3ca4e5d70 con 0x55a3ca3f79c0 2024-04-27T03:18:04.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9024aa0640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca4e5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9014000f90 tx=0x7f9014001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f8f1160a640 1 -- 172.21.15.110:0/1727756174 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9014021020 con 0x55a3ca3f79c0 2024-04-27T03:18:04.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a3ca4e6000 con 0x55a3ca3f79c0 2024-04-27T03:18:04.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f8f1160a640 1 -- 172.21.15.110:0/1727756174 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901401d680 con 0x55a3ca3f79c0 2024-04-27T03:18:04.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f8f1160a640 1 -- 172.21.15.110:0/1727756174 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901401d8b0 con 0x55a3ca3f79c0 2024-04-27T03:18:04.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.817+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3ca4e8c80 con 0x55a3ca3f79c0 2024-04-27T03:18:04.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f8f1160a640 1 -- 172.21.15.110:0/1727756174 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9014024020 con 0x55a3ca3f79c0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f8f1160a640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f0c034650 0x7f8f0c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f8f1160a640 1 -- 172.21.15.110:0/1727756174 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9014018040 con 0x55a3ca3f79c0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f902429f640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f0c034650 0x7f8f0c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca4ea8c0 0x55a3ca4eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca4ed1f0 con 0x55a3ca4ea8c0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f90252a1640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca4ea8c0 0x55a3ca4eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f902429f640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f0c034650 0x7f8f0c036b10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90180ae000 tx=0x7f90180a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f90252a1640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca4ea8c0 0x55a3ca4eccb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f90252a1640 1 -- 172.21.15.110:0/1727756174 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f901c0083f0 con 0x55a3ca4ea8c0 2024-04-27T03:18:04.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca4edf50 0x55a3ca4f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca4f0930 con 0x55a3ca4edf50 2024-04-27T03:18:04.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9024aa0640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca4edf50 0x55a3ca4f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9024aa0640 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca4edf50 0x55a3ca4f0340 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9024aa0640 1 -- 172.21.15.110:0/1727756174 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca4edf50 2024-04-27T03:18:04.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca4f21e0 con 0x55a3ca4ea8c0 2024-04-27T03:18:04.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f90252a1640 1 -- 172.21.15.110:0/1727756174 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f901c0083f0 con 0x55a3ca4ea8c0 2024-04-27T03:18:04.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.821+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca4f1410 con 0x55a3ca4edf50 2024-04-27T03:18:04.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9024aa0640 1 -- 172.21.15.110:0/1727756174 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca4edf50 2024-04-27T03:18:04.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca4f1800 con 0x55a3ca4ea8c0 2024-04-27T03:18:04.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f90252a1640 1 -- 172.21.15.110:0/1727756174 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f901c0083f0 con 0x55a3ca4ea8c0 2024-04-27T03:18:04.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9026f85980 1 Processor -- start 2024-04-27T03:18:04.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9026f85980 1 -- start start 2024-04-27T03:18:04.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9026f85980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca500b20 0x55a3ca500f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9026f85980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a3ca2f8310 con 0x55a3ca500b20 2024-04-27T03:18:04.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f90252a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca500b20 0x55a3ca500f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f90252a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca500b20 0x55a3ca500f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49964/0 (socket says 172.21.15.110:49964) 2024-04-27T03:18:04.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f90252a1640 1 -- 172.21.15.110:0/420151009 learned_addr learned my addr 172.21.15.110:0/420151009 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f90252a1640 1 -- 172.21.15.110:0/420151009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3ca501480 con 0x55a3ca500b20 2024-04-27T03:18:04.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f90252a1640 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca500b20 0x55a3ca500f40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f901c00aab0 tx=0x7f901c00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f8f0a7fc640 1 -- 172.21.15.110:0/420151009 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c007d30 con 0x55a3ca500b20 2024-04-27T03:18:04.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a3ca501f20 con 0x55a3ca500b20 2024-04-27T03:18:04.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f8f0a7fc640 1 -- 172.21.15.110:0/420151009 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c01d070 con 0x55a3ca500b20 2024-04-27T03:18:04.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f8f0a7fc640 1 -- 172.21.15.110:0/420151009 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c018440 con 0x55a3ca500b20 2024-04-27T03:18:04.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.825+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3ca5023f0 con 0x55a3ca500b20 2024-04-27T03:18:04.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f8f0a7fc640 1 -- 172.21.15.110:0/420151009 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f901c013070 con 0x55a3ca500b20 2024-04-27T03:18:04.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f8f0a7fc640 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8efc034650 0x7f8efc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f8f0a7fc640 1 -- 172.21.15.110:0/420151009 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f901c04b5e0 con 0x55a3ca500b20 2024-04-27T03:18:04.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9024aa0640 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8efc034650 0x7f8efc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 Processor -- start 2024-04-27T03:18:04.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- start start 2024-04-27T03:18:04.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca50e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a3ca4decc0 con 0x55a3ca50dd00 2024-04-27T03:18:04.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9024aa0640 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8efc034650 0x7f8efc036b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x55a3ca4f0f30 tx=0x7f9014047ed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f90252a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca50e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f90252a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca50e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49970/0 (socket says 172.21.15.110:49970) 2024-04-27T03:18:04.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f90252a1640 1 -- 172.21.15.110:0/3644342600 learned_addr learned my addr 172.21.15.110:0/3644342600 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f90252a1640 1 -- 172.21.15.110:0/3644342600 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3ca50e6f0 con 0x55a3ca50dd00 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f90252a1640 1 --2- 172.21.15.110:0/3644342600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca50e120 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f901c049930 tx=0x7f901c049b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427b553ad444dd33 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f8efbfff640 1 -- 172.21.15.110:0/3644342600 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c047070 con 0x55a3ca50dd00 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f8efbfff640 1 -- 172.21.15.110:0/3644342600 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c022460 con 0x55a3ca50dd00 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f8efbfff640 1 -- 172.21.15.110:0/3644342600 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c018bc0 con 0x55a3ca50dd00 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- 172.21.15.110:0/3644342600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 msgr2=0x55a3ca50e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 --2- 172.21.15.110:0/3644342600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca50e120 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f901c049930 tx=0x7f901c049b30 comp rx=0 tx=0).stop 2024-04-27T03:18:04.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- 172.21.15.110:0/3644342600 shutdown_connections 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 --2- 172.21.15.110:0/3644342600 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca50e120 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- 172.21.15.110:0/3644342600 >> 172.21.15.110:0/3644342600 conn(0x55a3ca50a040 msgr2=0x55a3ca5080a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- 172.21.15.110:0/3644342600 shutdown_connections 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- 172.21.15.110:0/3644342600 wait complete. 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 Processor -- start 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- start start 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca5151c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.829+0000 7f9026f85980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a3ca50ee10 con 0x55a3ca50dd00 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f90252a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca5151c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f90252a1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca5151c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:49986/0 (socket says 172.21.15.110:49986) 2024-04-27T03:18:04.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 learned_addr learned my addr 172.21.15.110:0/799722789 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:04.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3ca515700 con 0x55a3ca50dd00 2024-04-27T03:18:04.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f90252a1640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca5151c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f901c049930 tx=0x7f901c0498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f8efa7fc640 1 -- 172.21.15.110:0/799722789 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c031070 con 0x55a3ca50dd00 2024-04-27T03:18:04.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f8efa7fc640 1 -- 172.21.15.110:0/799722789 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c016d60 con 0x55a3ca50dd00 2024-04-27T03:18:04.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f8efa7fc640 1 -- 172.21.15.110:0/799722789 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f901c020820 con 0x55a3ca50dd00 2024-04-27T03:18:04.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a3ca515990 con 0x55a3ca50dd00 2024-04-27T03:18:04.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3ca518670 con 0x55a3ca50dd00 2024-04-27T03:18:04.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f8efa7fc640 1 -- 172.21.15.110:0/799722789 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f901c034070 con 0x55a3ca50dd00 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f8efa7fc640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ef0034650 0x7f8ef0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f8efa7fc640 1 -- 172.21.15.110:0/799722789 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f901c037c90 con 0x55a3ca50dd00 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f9024aa0640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ef0034650 0x7f8ef0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca520870 0x55a3ca522d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca523480 con 0x55a3ca520870 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f902429f640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca520870 0x55a3ca522d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f9024aa0640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ef0034650 0x7f8ef0036b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f901401bfd0 tx=0x7f901403d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.833+0000 7f902429f640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca520870 0x55a3ca522d30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 realm 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5238b0 con 0x55a3ca520870 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca524180 con 0x55a3ca520870 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca524ba0 0x55a3ca526fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca527740 con 0x55a3ca524ba0 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f90252a1640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca524ba0 0x55a3ca526fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f90252a1640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca524ba0 0x55a3ca526fd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca527f90 con 0x55a3ca520870 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 20 searching for the correct realm 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a3ca529d00 0x55a3ca52c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca529d00 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9024aa0640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a3ca529d00 0x55a3ca52c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.837+0000 7f9024aa0640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a3ca529d00 0x55a3ca52c0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f90252a1640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f901c036070 0x7f901c0218a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x7f901c036070 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f901c036070 0x7f901c0218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f901c036070 0x7f901c0218a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca529d00 2024-04-27T03:18:04.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.841+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x7f901c036070 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x7f901c036070 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca529d00 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v20'1 uv1 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.845+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca529d00 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca529d00 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x7f901c036070 2024-04-27T03:18:04.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca520870 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x7f901c036070 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca529d00 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x7f901c036070 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.849+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca529d00 2024-04-27T03:18:04.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52c770 con 0x55a3ca524ba0 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca528910 con 0x55a3ca520870 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca528520 con 0x55a3ca524ba0 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca529200 con 0x55a3ca520870 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 zone default found 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 4 Realm: () 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 zonegroup default 2024-04-27T03:18:04.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52d080 con 0x7f901c036070 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52f2b0 con 0x55a3ca529d00 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.853+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52f650 con 0x55a3ca529d00 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 20 started sync module instance, tier type = 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52e6f0 con 0x55a3ca524ba0 2024-04-27T03:18:04.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94161962461968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52e350 con 0x55a3ca524ba0 2024-04-27T03:18:04.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5304c0 con 0x55a3ca520870 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94161962467344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5308e0 con 0x55a3ca520870 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.857+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca531300 con 0x55a3ca524ba0 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94161962461968] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94161962467344] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94161962470896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca530430 con 0x55a3ca524ba0 2024-04-27T03:18:04.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca531ba0 con 0x7f901c036070 2024-04-27T03:18:04.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94161962473200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca530430 con 0x7f901c036070 2024-04-27T03:18:04.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca532620 con 0x55a3ca524ba0 2024-04-27T03:18:04.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.861+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94161962470896] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94161962473200] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94161962474256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca532130 con 0x55a3ca524ba0 2024-04-27T03:18:04.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca532ef0 con 0x55a3ca529d00 2024-04-27T03:18:04.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94161962478144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca532130 con 0x55a3ca529d00 2024-04-27T03:18:04.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca533c30 con 0x55a3ca529d00 2024-04-27T03:18:04.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94161962474256] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94161962478144] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94161962480272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca534050 con 0x55a3ca529d00 2024-04-27T03:18:04.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.865+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5348a0 con 0x7f901c036070 2024-04-27T03:18:04.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94161962480272] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'9 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94161962483792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca534c90 con 0x7f901c036070 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=0 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=1 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=2 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=3 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=4 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=5 2024-04-27T03:18:04.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=6 2024-04-27T03:18:04.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94161962483792] v27'10 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 20 add_watcher() i=7 2024-04-27T03:18:04.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:04.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca537cc0 con 0x55a3ca520870 2024-04-27T03:18:04.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.869+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94161962485360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5351a0 con 0x55a3ca520870 2024-04-27T03:18:04.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.873+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94161962485360] v27'37 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.873+0000 7f8edafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:04.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.873+0000 7f9026f85980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:04.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.873+0000 7f9026f85980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:04.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.873+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca537b10 con 0x55a3ca524ba0 2024-04-27T03:18:04.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.873+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'41 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.873+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca537b60 con 0x7f901c036070 2024-04-27T03:18:04.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.877+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'17 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.877+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca540f70 con 0x55a3ca529d00 2024-04-27T03:18:04.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.877+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'17 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.877+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5414e0 con 0x55a3ca520870 2024-04-27T03:18:04.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.881+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'38 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.881+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca541990 con 0x55a3ca524ba0 2024-04-27T03:18:04.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.881+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'31 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.881+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca541990 con 0x55a3ca529d00 2024-04-27T03:18:04.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.885+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'27 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.885+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5403d0 con 0x55a3ca524ba0 2024-04-27T03:18:04.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.885+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'19 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.885+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5403d0 con 0x55a3ca520870 2024-04-27T03:18:04.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.889+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'22 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.889+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5403d0 con 0x55a3ca524ba0 2024-04-27T03:18:04.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.889+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'11 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.889+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.893+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'17 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.893+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.897+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'32 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.897+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca529d00 2024-04-27T03:18:04.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.897+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'13 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.897+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.897+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'17 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.897+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca520870 2024-04-27T03:18:04.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.901+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'33 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.901+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca520870 2024-04-27T03:18:04.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.901+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'39 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.901+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.905+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'23 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.905+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.905+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'33 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.905+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca529d00 2024-04-27T03:18:04.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.909+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'23 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.909+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.909+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'19 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.909+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca520870 2024-04-27T03:18:04.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.913+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'40 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.913+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.917+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'24 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.917+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.917+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'27 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.917+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca529d00 2024-04-27T03:18:04.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.917+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'19 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.917+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.921+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'29 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.921+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca529d00 2024-04-27T03:18:04.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.921+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'13 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.925+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca529d00 2024-04-27T03:18:04.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.925+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'28 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.925+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca529d00 2024-04-27T03:18:04.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.925+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'19 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.925+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.929+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'23 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.929+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca520870 2024-04-27T03:18:04.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.933+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'23 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.933+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca524ba0 2024-04-27T03:18:04.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.933+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'25 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.933+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x55a3ca520870 2024-04-27T03:18:04.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'34 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca542300 con 0x7f901c036070 2024-04-27T03:18:04.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'17 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f8ed9fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 20 Filter name: none 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f8e4b7fe640 20 reqs_thread_entry: start 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 10 cache get: name=default.rgw.meta+users.uid+bar : miss 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 20 rados->read ofs=0 len=0 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.e 8:718f6a54:users.uid::bar:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca53fea0 con 0x7f901c036070 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(94 bar [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:04.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x0 2024-04-27T03:18:04.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f9026f85980 10 adding default.rgw.meta+users.uid+bar to cache LRU end 2024-04-27T03:18:04.947 INFO:teuthology.orchestra.run.smithi110.stderr:could not remove user: unable to remove user, user does not exist 2024-04-27T03:18:04.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.937+0000 7f8edafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:04.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.941+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94161962485360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca53dda0 con 0x55a3ca520870 2024-04-27T03:18:04.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.941+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(95 data_loggenerations_metadata [watch unwatch cookie 94161962485360] v27'41 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.941+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94161962461968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca534640 con 0x55a3ca524ba0 2024-04-27T03:18:04.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.945+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(96 notify.0 [watch unwatch cookie 94161962461968] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.945+0000 7f9026f85980 20 remove_watcher() i=0 2024-04-27T03:18:04.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.945+0000 7f9026f85980 2 removed watcher, disabling cache 2024-04-27T03:18:04.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.945+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:97 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94161962467344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca52eb10 con 0x55a3ca520870 2024-04-27T03:18:04.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.945+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(97 notify.1 [watch unwatch cookie 94161962467344] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x55a3ca520870 2024-04-27T03:18:04.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.945+0000 7f9026f85980 20 remove_watcher() i=1 2024-04-27T03:18:04.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.945+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94161962470896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca530010 con 0x55a3ca524ba0 2024-04-27T03:18:04.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(98 notify.2 [watch unwatch cookie 94161962470896] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f9026f85980 20 remove_watcher() i=2 2024-04-27T03:18:04.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:99 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94161962473200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca530010 con 0x7f901c036070 2024-04-27T03:18:04.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(99 notify.3 [watch unwatch cookie 94161962473200] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f9026f85980 20 remove_watcher() i=3 2024-04-27T03:18:04.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94161962474256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca5316f0 con 0x55a3ca524ba0 2024-04-27T03:18:04.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f90252a1640 1 -- 172.21.15.110:0/799722789 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(100 notify.4 [watch unwatch cookie 94161962474256] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901c033030 con 0x55a3ca524ba0 2024-04-27T03:18:04.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f9026f85980 20 remove_watcher() i=4 2024-04-27T03:18:04.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.949+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94161962478144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca530010 con 0x55a3ca529d00 2024-04-27T03:18:04.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(101 notify.5 [watch unwatch cookie 94161962478144] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 20 remove_watcher() i=5 2024-04-27T03:18:04.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94161962480272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca530010 con 0x55a3ca529d00 2024-04-27T03:18:04.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9024aa0640 1 -- 172.21.15.110:0/799722789 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(102 notify.6 [watch unwatch cookie 94161962480272] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f901400f070 con 0x55a3ca529d00 2024-04-27T03:18:04.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 20 remove_watcher() i=6 2024-04-27T03:18:04.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:103 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94161962483792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a3ca533290 con 0x7f901c036070 2024-04-27T03:18:04.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(103 notify.7 [watch unwatch cookie 94161962483792] v27'11 uv9 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90180aaa10 con 0x7f901c036070 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 20 remove_watcher() i=7 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca524ba0 msgr2=0x55a3ca526fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca524ba0 0x55a3ca526fd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca520870 msgr2=0x55a3ca522d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca520870 0x55a3ca522d30 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f901c036070 msgr2=0x7f901c0218a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f901c036070 0x7f901c0218a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a3ca529d00 msgr2=0x55a3ca52c0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a3ca529d00 0x55a3ca52c0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ef0034650 msgr2=0x7f8ef0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ef0034650 0x7f8ef0036b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f901401bfd0 tx=0x7f901403d000 comp rx=0 tx=0).stop 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 msgr2=0x55a3ca5151c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 --2- 172.21.15.110:0/799722789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca50dd00 0x55a3ca5151c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f901c049930 tx=0x7f901c0498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 reap_dead start 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f902429f640 1 -- 172.21.15.110:0/799722789 reap_dead start 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 shutdown_connections 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.953+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 >> 172.21.15.110:0/799722789 conn(0x55a3ca50a040 msgr2=0x55a3ca508070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 shutdown_connections 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/799722789 wait complete. 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8efc034650 msgr2=0x7f8efc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8efc034650 0x7f8efc036b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x55a3ca4f0f30 tx=0x7f9014047ed0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca500b20 msgr2=0x55a3ca500f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca500b20 0x55a3ca500f40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f901c00aab0 tx=0x7f901c00af70 comp rx=0 tx=0).stop 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 shutdown_connections 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8efc034650 0x7f8efc036b10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/420151009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca500b20 0x55a3ca500f40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 >> 172.21.15.110:0/420151009 conn(0x55a3ca4f5930 msgr2=0x55a3ca4f6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 shutdown_connections 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/420151009 wait complete. 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca4edf50 msgr2=0x55a3ca4f0340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca4edf50 0x55a3ca4f0340 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca4ea8c0 msgr2=0x55a3ca4eccb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca4ea8c0 0x55a3ca4eccb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f0c034650 msgr2=0x7f8f0c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f0c034650 0x7f8f0c036b10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90180ae000 tx=0x7f90180a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 msgr2=0x55a3ca4e5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca4e5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9014000f90 tx=0x7f9014001260 comp rx=0 tx=0).stop 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 shutdown_connections 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a3ca4edf50 0x55a3ca4f0340 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a3ca4ea8c0 0x55a3ca4eccb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f0c034650 0x7f8f0c036b10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 --2- 172.21.15.110:0/1727756174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a3ca3f79c0 0x55a3ca4e5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:04.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 >> 172.21.15.110:0/1727756174 conn(0x55a3ca366b90 msgr2=0x55a3ca4d8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:04.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 shutdown_connections 2024-04-27T03:18:04.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:04.957+0000 7f9026f85980 1 -- 172.21.15.110:0/1727756174 wait complete. 2024-04-27T03:18:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-27T03:18:04.976 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud'] 2024-04-27T03:18:04.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud'] 2024-04-27T03:18:04.976 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud 2024-04-27T03:18:05.054 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:05.054 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:05.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.045+0000 7f167b351980 1 Processor -- start 2024-04-27T03:18:05.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.045+0000 7f167b351980 1 -- start start 2024-04-27T03:18:05.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f167b351980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ae376c5850 0x55ae37756340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f167b351980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55ae376100d0 con 0x55ae37756880 2024-04-27T03:18:05.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f167b351980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55ae37605430 con 0x55ae376c5850 2024-04-27T03:18:05.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f1678e6c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ae376c5850 0x55ae37756340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f1673fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55ae37756880 0x55ae37758c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34544/0 (socket says 172.21.15.110:34544) 2024-04-27T03:18:05.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f1678e6c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ae376c5850 0x55ae37756340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50000/0 (socket says 172.21.15.110:50000) 2024-04-27T03:18:05.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f1673fff640 1 -- 172.21.15.110:0/860515382 learned_addr learned my addr 172.21.15.110:0/860515382 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:05.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f1678e6c640 1 -- 172.21.15.110:0/860515382 >> v1:172.21.15.77:6789/0 conn(0x55ae37756880 legacy=0x55ae37758c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f1678e6c640 1 -- 172.21.15.110:0/860515382 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae375fe510 con 0x55ae376c5850 2024-04-27T03:18:05.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f1678e6c640 1 --2- 172.21.15.110:0/860515382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37756340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f166c00a480 tx=0x7f166c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a2e00221b925e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f16737fe640 1 -- 172.21.15.110:0/860515382 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c0b0070 con 0x55ae376c5850 2024-04-27T03:18:05.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f16737fe640 1 -- 172.21.15.110:0/860515382 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f166c0ab540 con 0x55ae376c5850 2024-04-27T03:18:05.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.049+0000 7f16737fe640 1 -- 172.21.15.110:0/860515382 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c0ab840 con 0x55ae376c5850 2024-04-27T03:18:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.053+0000 7f167b351980 1 -- 172.21.15.110:0/860515382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 msgr2=0x55ae37756340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.053+0000 7f167b351980 1 --2- 172.21.15.110:0/860515382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37756340 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f166c00a480 tx=0x7f166c01b280 comp rx=0 tx=0).stop 2024-04-27T03:18:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.053+0000 7f167b351980 1 -- 172.21.15.110:0/860515382 shutdown_connections 2024-04-27T03:18:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.053+0000 7f167b351980 1 --2- 172.21.15.110:0/860515382 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37756340 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.053+0000 7f167b351980 1 -- 172.21.15.110:0/860515382 >> 172.21.15.110:0/860515382 conn(0x55ae3774ba20 msgr2=0x55ae3774de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.053+0000 7f167b351980 1 -- 172.21.15.110:0/860515382 shutdown_connections 2024-04-27T03:18:05.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.053+0000 7f167b351980 1 -- 172.21.15.110:0/860515382 wait complete. 2024-04-27T03:18:05.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 Processor -- start 2024-04-27T03:18:05.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- start start 2024-04-27T03:18:05.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae3783e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae376542d0 con 0x55ae376c5850 2024-04-27T03:18:05.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1678e6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae3783e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1678e6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae3783e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50002/0 (socket says 172.21.15.110:50002) 2024-04-27T03:18:05.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1678e6c640 1 -- 172.21.15.110:0/3681402816 learned_addr learned my addr 172.21.15.110:0/3681402816 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1678e6c640 1 -- 172.21.15.110:0/3681402816 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae375daae0 con 0x55ae376c5850 2024-04-27T03:18:05.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1678e6c640 1 --2- 172.21.15.110:0/3681402816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae3783e5c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f166c0ae040 tx=0x7f166c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529fc197a23fd7a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1566e05640 1 -- 172.21.15.110:0/3681402816 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c0b0040 con 0x55ae376c5850 2024-04-27T03:18:05.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1566e05640 1 -- 172.21.15.110:0/3681402816 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f166c0ab9b0 con 0x55ae376c5850 2024-04-27T03:18:05.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f1566e05640 1 -- 172.21.15.110:0/3681402816 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c0abc90 con 0x55ae376c5850 2024-04-27T03:18:05.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- 172.21.15.110:0/3681402816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 msgr2=0x55ae3783e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 --2- 172.21.15.110:0/3681402816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae3783e5c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f166c0ae040 tx=0x7f166c003030 comp rx=0 tx=0).stop 2024-04-27T03:18:05.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- 172.21.15.110:0/3681402816 shutdown_connections 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 --2- 172.21.15.110:0/3681402816 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae3783e5c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- 172.21.15.110:0/3681402816 >> 172.21.15.110:0/3681402816 conn(0x55ae3774ba20 msgr2=0x55ae3783a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- 172.21.15.110:0/3681402816 shutdown_connections 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- 172.21.15.110:0/3681402816 wait complete. 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 Processor -- start 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- start start 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37845540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.069+0000 7f167b351980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae3775cdf0 con 0x55ae376c5850 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1678e6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37845540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1678e6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37845540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50006/0 (socket says 172.21.15.110:50006) 2024-04-27T03:18:05.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1678e6c640 1 -- 172.21.15.110:0/3535974978 learned_addr learned my addr 172.21.15.110:0/3535974978 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1678e6c640 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae37845a80 con 0x55ae376c5850 2024-04-27T03:18:05.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1678e6c640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37845540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f166c01bb20 tx=0x7f166c018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1565602640 1 -- 172.21.15.110:0/3535974978 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c0c5070 con 0x55ae376c5850 2024-04-27T03:18:05.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1565602640 1 -- 172.21.15.110:0/3535974978 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f166c0b6970 con 0x55ae376c5850 2024-04-27T03:18:05.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1565602640 1 -- 172.21.15.110:0/3535974978 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c0b0050 con 0x55ae376c5850 2024-04-27T03:18:05.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ae37845d10 con 0x55ae376c5850 2024-04-27T03:18:05.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae378488f0 con 0x55ae376c5850 2024-04-27T03:18:05.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1565602640 1 -- 172.21.15.110:0/3535974978 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f166c0c8070 con 0x55ae376c5850 2024-04-27T03:18:05.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1565602640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1560034650 0x7f1560036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1565602640 1 -- 172.21.15.110:0/3535974978 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f166c0bc070 con 0x55ae376c5850 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f1673fff640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1560034650 0x7f1560036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37756880 0x55ae3784c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae3784c9a0 con 0x55ae37756880 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.073+0000 7f167966d640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37756880 0x55ae3784c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f1673fff640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1560034650 0x7f1560036b10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1664000f70 tx=0x7f16640012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167966d640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37756880 0x55ae3784c3d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167966d640 1 -- 172.21.15.110:0/3535974978 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f167400a4d0 con 0x55ae37756880 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae3784d770 0x55ae3784fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae378501a0 con 0x55ae3784d770 2024-04-27T03:18:05.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f1678e6c640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae3784d770 0x55ae3784fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f1678e6c640 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae3784d770 0x55ae3784fb60 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f1678e6c640 1 -- 172.21.15.110:0/3535974978 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae3784d770 2024-04-27T03:18:05.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37851a50 con 0x55ae37756880 2024-04-27T03:18:05.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167966d640 1 -- 172.21.15.110:0/3535974978 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f167400a4d0 con 0x55ae37756880 2024-04-27T03:18:05.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37850c80 con 0x55ae3784d770 2024-04-27T03:18:05.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f1678e6c640 1 -- 172.21.15.110:0/3535974978 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae3784d770 2024-04-27T03:18:05.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37851070 con 0x55ae37756880 2024-04-27T03:18:05.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.077+0000 7f167966d640 1 -- 172.21.15.110:0/3535974978 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f167400a4d0 con 0x55ae37756880 2024-04-27T03:18:05.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167b351980 1 Processor -- start 2024-04-27T03:18:05.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167b351980 1 -- start start 2024-04-27T03:18:05.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167b351980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae37860350 0x55ae37860770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167b351980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae37658310 con 0x55ae37860350 2024-04-27T03:18:05.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167966d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae37860350 0x55ae37860770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167966d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae37860350 0x55ae37860770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50016/0 (socket says 172.21.15.110:50016) 2024-04-27T03:18:05.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167966d640 1 -- 172.21.15.110:0/3179184086 learned_addr learned my addr 172.21.15.110:0/3179184086 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167966d640 1 -- 172.21.15.110:0/3179184086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae37860cb0 con 0x55ae37860350 2024-04-27T03:18:05.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167966d640 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae37860350 0x55ae37860770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f167400dfc0 tx=0x7f16740178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f155e7fc640 1 -- 172.21.15.110:0/3179184086 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16740157d0 con 0x55ae37860350 2024-04-27T03:18:05.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ae37861750 con 0x55ae37860350 2024-04-27T03:18:05.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f155e7fc640 1 -- 172.21.15.110:0/3179184086 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1674015970 con 0x55ae37860350 2024-04-27T03:18:05.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae37861c20 con 0x55ae37860350 2024-04-27T03:18:05.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f155e7fc640 1 -- 172.21.15.110:0/3179184086 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1674015c50 con 0x55ae37860350 2024-04-27T03:18:05.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f155e7fc640 1 -- 172.21.15.110:0/3179184086 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f1674020440 con 0x55ae37860350 2024-04-27T03:18:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f155e7fc640 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1550034650 0x7f1550036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.081+0000 7f155e7fc640 1 -- 172.21.15.110:0/3179184086 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f167400e040 con 0x55ae37860350 2024-04-27T03:18:05.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f1678e6c640 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1550034650 0x7f1550036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 Processor -- start 2024-04-27T03:18:05.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- start start 2024-04-27T03:18:05.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae3786d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae3783ec50 con 0x55ae3786d550 2024-04-27T03:18:05.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f1678e6c640 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1550034650 0x7f1550036b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x55ae378507a0 tx=0x7f166c000bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167966d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae3786d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167966d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae3786d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50022/0 (socket says 172.21.15.110:50022) 2024-04-27T03:18:05.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167966d640 1 -- 172.21.15.110:0/660767148 learned_addr learned my addr 172.21.15.110:0/660767148 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167966d640 1 -- 172.21.15.110:0/660767148 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae3786df20 con 0x55ae3786d550 2024-04-27T03:18:05.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167966d640 1 --2- 172.21.15.110:0/660767148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae3786d950 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f167404aaa0 tx=0x7f1674013870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3546b9f4979a981f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f154ffff640 1 -- 172.21.15.110:0/660767148 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1674041070 con 0x55ae3786d550 2024-04-27T03:18:05.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f154ffff640 1 -- 172.21.15.110:0/660767148 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1674020b50 con 0x55ae3786d550 2024-04-27T03:18:05.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f154ffff640 1 -- 172.21.15.110:0/660767148 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1674011cc0 con 0x55ae3786d550 2024-04-27T03:18:05.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- 172.21.15.110:0/660767148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 msgr2=0x55ae3786d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 --2- 172.21.15.110:0/660767148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae3786d950 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f167404aaa0 tx=0x7f1674013870 comp rx=0 tx=0).stop 2024-04-27T03:18:05.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- 172.21.15.110:0/660767148 shutdown_connections 2024-04-27T03:18:05.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 --2- 172.21.15.110:0/660767148 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae3786d950 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- 172.21.15.110:0/660767148 >> 172.21.15.110:0/660767148 conn(0x55ae37869890 msgr2=0x55ae378678f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- 172.21.15.110:0/660767148 shutdown_connections 2024-04-27T03:18:05.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- 172.21.15.110:0/660767148 wait complete. 2024-04-27T03:18:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 Processor -- start 2024-04-27T03:18:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- start start 2024-04-27T03:18:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae378749a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.085+0000 7f167b351980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ae3786e610 con 0x55ae3786d550 2024-04-27T03:18:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167966d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae378749a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167966d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae378749a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50036/0 (socket says 172.21.15.110:50036) 2024-04-27T03:18:05.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 learned_addr learned my addr 172.21.15.110:0/2682685025 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae37874ee0 con 0x55ae3786d550 2024-04-27T03:18:05.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167966d640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae378749a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1674046040 tx=0x7f167404aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f154e7fc640 1 -- 172.21.15.110:0/2682685025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1674038070 con 0x55ae3786d550 2024-04-27T03:18:05.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ae37875170 con 0x55ae3786d550 2024-04-27T03:18:05.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f154e7fc640 1 -- 172.21.15.110:0/2682685025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1674011730 con 0x55ae3786d550 2024-04-27T03:18:05.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f154e7fc640 1 -- 172.21.15.110:0/2682685025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1674041040 con 0x55ae3786d550 2024-04-27T03:18:05.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae37877e50 con 0x55ae3786d550 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f154e7fc640 1 -- 172.21.15.110:0/2682685025 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f167403b070 con 0x55ae3786d550 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f154e7fc640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1544034650 0x7f1544036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f154e7fc640 1 -- 172.21.15.110:0/2682685025 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f167404b030 con 0x55ae3786d550 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f1678e6c640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1544034650 0x7f1544036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37880010 0x55ae378824b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37882c00 con 0x55ae37880010 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f1673fff640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37880010 0x55ae378824b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f1678e6c640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1544034650 0x7f1544036b10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f166c0b7a90 tx=0x7f166c0dc000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.089+0000 7f1673fff640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37880010 0x55ae378824b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f166400e5a0 con 0x55ae37880010 2024-04-27T03:18:05.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 realm 2024-04-27T03:18:05.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae3787f4e0 con 0x55ae37880010 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f166400e5a0 con 0x55ae37880010 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37883700 con 0x55ae37880010 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f166400e5a0 con 0x55ae37880010 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae378844e0 0x55ae378868d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37886f80 con 0x55ae378844e0 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167966d640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae378844e0 0x55ae378868d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167966d640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae378844e0 0x55ae378868d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae378877d0 con 0x55ae37880010 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f166400e5a0 con 0x55ae37880010 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 20 searching for the correct realm 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ae37889540 0x55ae3788b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37889540 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f1678e6c640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ae37889540 0x55ae3788b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.093+0000 7f1678e6c640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ae37889540 0x55ae3788b930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f167966d640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16740133b0 0x7f167401fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x7f16740133b0 2024-04-27T03:18:05.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16740133b0 0x7f167401fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16740133b0 0x7f167401fab0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166400e5a0 con 0x7f16740133b0 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166400e5a0 con 0x55ae37880010 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37889540 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.097+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x7f16740133b0 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x7f16740133b0 2024-04-27T03:18:05.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x7f16740133b0 2024-04-27T03:18:05.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f1664029040 con 0x7f16740133b0 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37889540 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v20'1 uv1 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37889540 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.101+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37889540 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x7f16740133b0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x7f16740133b0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37880010 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x7f16740133b0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x7f16740133b0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37889540 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x7f16740133b0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1664029040 con 0x7f16740133b0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.105+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae37889540 2024-04-27T03:18:05.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788bfb0 con 0x55ae378844e0 2024-04-27T03:18:05.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37888110 con 0x55ae37880010 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f1664029040 con 0x55ae37880010 2024-04-27T03:18:05.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae378889b0 con 0x55ae378844e0 2024-04-27T03:18:05.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37887d50 con 0x55ae37880010 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 zone default found 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 4 Realm: () 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 zonegroup default 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788c430 con 0x7f16740133b0 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.109+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788e250 con 0x55ae37889540 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788e620 con 0x55ae37889540 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 20 started sync module instance, tier type = 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:05.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788d6f0 con 0x55ae378844e0 2024-04-27T03:18:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94206744386320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788ef30 con 0x55ae378844e0 2024-04-27T03:18:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.113+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788f780 con 0x55ae37880010 2024-04-27T03:18:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94206744391472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788f8a0 con 0x55ae37880010 2024-04-27T03:18:05.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378903b0 con 0x55ae378844e0 2024-04-27T03:18:05.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94206744386320] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94206744391472] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94206744395360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37890370 con 0x55ae378844e0 2024-04-27T03:18:05.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37890c50 con 0x7f16740133b0 2024-04-27T03:18:05.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94206744397728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37890370 con 0x7f16740133b0 2024-04-27T03:18:05.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.117+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37891730 con 0x55ae378844e0 2024-04-27T03:18:05.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94206744395360] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94206744397728] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94206744400448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37891730 con 0x55ae378844e0 2024-04-27T03:18:05.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37891b50 con 0x55ae37889540 2024-04-27T03:18:05.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94206744398784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378923a0 con 0x55ae37889540 2024-04-27T03:18:05.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37892d60 con 0x55ae37889540 2024-04-27T03:18:05.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.121+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94206744400448] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94206744406192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37893180 con 0x55ae37889540 2024-04-27T03:18:05.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37893b40 con 0x7f16740133b0 2024-04-27T03:18:05.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94206744398784] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'12 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94206744403696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37893b40 con 0x7f16740133b0 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=0 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=1 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=2 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=3 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=4 2024-04-27T03:18:05.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=5 2024-04-27T03:18:05.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94206744406192] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=6 2024-04-27T03:18:05.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94206744403696] v27'13 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 20 add_watcher() i=7 2024-04-27T03:18:05.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:05.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae37893b40 con 0x55ae37880010 2024-04-27T03:18:05.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.125+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94206744409744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37893ab0 con 0x55ae37880010 2024-04-27T03:18:05.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.129+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94206744409744] v27'42 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.129+0000 7f152efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.129+0000 7f167b351980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:05.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.129+0000 7f167b351980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:05.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.129+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789f540 con 0x55ae378844e0 2024-04-27T03:18:05.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.129+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'42 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.133+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789f5c0 con 0x7f16740133b0 2024-04-27T03:18:05.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.133+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'18 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.133+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789fcb0 con 0x55ae37889540 2024-04-27T03:18:05.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.133+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'18 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.133+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a0140 con 0x55ae37880010 2024-04-27T03:18:05.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.137+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'43 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.137+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789b2b0 con 0x55ae378844e0 2024-04-27T03:18:05.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.137+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'34 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.137+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789b2b0 con 0x55ae37889540 2024-04-27T03:18:05.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.141+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'29 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.141+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789b2b0 con 0x55ae378844e0 2024-04-27T03:18:05.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.141+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'20 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.141+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789b2b0 con 0x55ae37880010 2024-04-27T03:18:05.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.145+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'24 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.145+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a0c20 con 0x55ae378844e0 2024-04-27T03:18:05.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.145+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'12 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.145+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a0c20 con 0x55ae378844e0 2024-04-27T03:18:05.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.149+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'18 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.149+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a0c20 con 0x55ae378844e0 2024-04-27T03:18:05.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.153+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'35 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.153+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a0c20 con 0x55ae37889540 2024-04-27T03:18:05.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.153+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'14 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.153+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.153+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'18 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.153+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37880010 2024-04-27T03:18:05.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.157+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'35 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.157+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37880010 2024-04-27T03:18:05.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.157+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'44 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.157+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.161+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'26 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.161+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.165+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'36 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.165+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37889540 2024-04-27T03:18:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.165+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'24 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.165+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.169+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'20 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.169+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37880010 2024-04-27T03:18:05.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.169+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'45 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.169+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.173+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'27 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.173+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.177+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'28 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.177+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37889540 2024-04-27T03:18:05.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.177+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'20 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.177+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.181+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'30 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.181+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37889540 2024-04-27T03:18:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.181+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'14 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.181+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37889540 2024-04-27T03:18:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.185+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'30 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.185+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37889540 2024-04-27T03:18:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.185+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'20 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.185+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.189+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'24 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.189+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37880010 2024-04-27T03:18:05.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.189+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'25 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.189+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae378844e0 2024-04-27T03:18:05.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.193+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'28 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.193+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x55ae37880010 2024-04-27T03:18:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.193+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'36 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378a11d0 con 0x7f16740133b0 2024-04-27T03:18:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'18 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f152dfe3640 20 reqs_thread_entry: start 2024-04-27T03:18:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:05.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 20 Filter name: none 2024-04-27T03:18:05.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f149f7fe640 20 reqs_thread_entry: start 2024-04-27T03:18:05.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:18:05.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789e7a0 con 0x7f16740133b0 2024-04-27T03:18:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(94 fud [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2024-04-27T03:18:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.197+0000 7f167b351980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:could not remove user: unable to remove user, user does not exist 2024-04-27T03:18:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.201+0000 7f152efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.201+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94206744409744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3789ced0 con 0x55ae37880010 2024-04-27T03:18:05.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.201+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(95 data_loggenerations_metadata [watch unwatch cookie 94206744409744] v27'46 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.201+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94206744386320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37893690 con 0x55ae378844e0 2024-04-27T03:18:05.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.205+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(96 notify.0 [watch unwatch cookie 94206744386320] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.205+0000 7f167b351980 20 remove_watcher() i=0 2024-04-27T03:18:05.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.205+0000 7f167b351980 2 removed watcher, disabling cache 2024-04-27T03:18:05.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.205+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:97 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94206744391472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788db10 con 0x55ae37880010 2024-04-27T03:18:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.205+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(97 notify.1 [watch unwatch cookie 94206744391472] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x55ae37880010 2024-04-27T03:18:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.205+0000 7f167b351980 20 remove_watcher() i=1 2024-04-27T03:18:05.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.205+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94206744395360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788db10 con 0x55ae378844e0 2024-04-27T03:18:05.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.209+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(98 notify.2 [watch unwatch cookie 94206744395360] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.209+0000 7f167b351980 20 remove_watcher() i=2 2024-04-27T03:18:05.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.209+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:99 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94206744397728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788fe60 con 0x7f16740133b0 2024-04-27T03:18:05.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.209+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(99 notify.3 [watch unwatch cookie 94206744397728] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.209+0000 7f167b351980 20 remove_watcher() i=3 2024-04-27T03:18:05.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.209+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94206744400448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788db10 con 0x55ae378844e0 2024-04-27T03:18:05.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f167966d640 1 -- 172.21.15.110:0/2682685025 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(100 notify.4 [watch unwatch cookie 94206744400448] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f167403a030 con 0x55ae378844e0 2024-04-27T03:18:05.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f167b351980 20 remove_watcher() i=4 2024-04-27T03:18:05.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94206744398784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae378907a0 con 0x55ae37889540 2024-04-27T03:18:05.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(101 notify.5 [watch unwatch cookie 94206744398784] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f167b351980 20 remove_watcher() i=5 2024-04-27T03:18:05.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94206744406192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae3788fe60 con 0x55ae37889540 2024-04-27T03:18:05.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f1678e6c640 1 -- 172.21.15.110:0/2682685025 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(102 notify.6 [watch unwatch cookie 94206744406192] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166c0c7030 con 0x55ae37889540 2024-04-27T03:18:05.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f167b351980 20 remove_watcher() i=6 2024-04-27T03:18:05.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.213+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:103 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94206744403696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ae37891240 con 0x7f16740133b0 2024-04-27T03:18:05.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(103 notify.7 [watch unwatch cookie 94206744403696] v27'14 uv12 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f166403c1c0 con 0x7f16740133b0 2024-04-27T03:18:05.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 20 remove_watcher() i=7 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae378844e0 msgr2=0x55ae378868d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae378844e0 0x55ae378868d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37880010 msgr2=0x55ae378824b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37880010 0x55ae378824b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16740133b0 msgr2=0x7f167401fab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16740133b0 0x7f167401fab0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ae37889540 msgr2=0x55ae3788b930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ae37889540 0x55ae3788b930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1544034650 msgr2=0x7f1544036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1544034650 0x7f1544036b10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f166c0b7a90 tx=0x7f166c0dc000 comp rx=0 tx=0).stop 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 msgr2=0x55ae378749a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/2682685025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae3786d550 0x55ae378749a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1674046040 tx=0x7f167404aa30 comp rx=0 tx=0).stop 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f1673fff640 1 -- 172.21.15.110:0/2682685025 reap_dead start 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 shutdown_connections 2024-04-27T03:18:05.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 >> 172.21.15.110:0/2682685025 conn(0x55ae37869890 msgr2=0x55ae3786ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 shutdown_connections 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/2682685025 wait complete. 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1550034650 msgr2=0x7f1550036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1550034650 0x7f1550036b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x55ae378507a0 tx=0x7f166c000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae37860350 msgr2=0x55ae37860770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae37860350 0x55ae37860770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f167400dfc0 tx=0x7f16740178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 shutdown_connections 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1550034650 0x7f1550036b10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3179184086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae37860350 0x55ae37860770 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 >> 172.21.15.110:0/3179184086 conn(0x55ae37855160 msgr2=0x55ae37855c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 shutdown_connections 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3179184086 wait complete. 2024-04-27T03:18:05.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae3784d770 msgr2=0x55ae3784fb60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae3784d770 0x55ae3784fb60 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37756880 msgr2=0x55ae3784c3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37756880 0x55ae3784c3d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1560034650 msgr2=0x7f1560036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1560034650 0x7f1560036b10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1664000f70 tx=0x7f16640012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 msgr2=0x55ae37845540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37845540 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f166c01bb20 tx=0x7f166c018020 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 shutdown_connections 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ae3784d770 0x55ae3784fb60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ae37756880 0x55ae3784c3d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1560034650 0x7f1560036b10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 --2- 172.21.15.110:0/3535974978 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ae376c5850 0x55ae37845540 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f166c01bb20 tx=0x7f166c018020 comp rx=0 tx=0).stop 2024-04-27T03:18:05.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 >> 172.21.15.110:0/3535974978 conn(0x55ae3774ba20 msgr2=0x55ae37838c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 shutdown_connections 2024-04-27T03:18:05.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.217+0000 7f167b351980 1 -- 172.21.15.110:0/3535974978 wait complete. 2024-04-27T03:18:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-27T03:18:05.238 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo'] 2024-04-27T03:18:05.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo'] 2024-04-27T03:18:05.238 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo 2024-04-27T03:18:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:05.315 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b3aa46980 1 Processor -- start 2024-04-27T03:18:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b3aa46980 1 -- start start 2024-04-27T03:18:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b3aa46980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560f6cae0290 0x560f6cae0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b3aa46980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x560f6c99f0d0 con 0x560f6cae0bb0 2024-04-27T03:18:05.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b3aa46980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x560f6c994430 con 0x560f6cae0290 2024-04-27T03:18:05.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b38561640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560f6cae0290 0x560f6cae0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b37d60640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x560f6cae0bb0 0x560f6cae1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34548/0 (socket says 172.21.15.110:34548) 2024-04-27T03:18:05.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b38561640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560f6cae0290 0x560f6cae0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50038/0 (socket says 172.21.15.110:50038) 2024-04-27T03:18:05.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b37d60640 1 -- 172.21.15.110:0/948344252 learned_addr learned my addr 172.21.15.110:0/948344252 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:05.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b38561640 1 -- 172.21.15.110:0/948344252 >> v1:172.21.15.77:6789/0 conn(0x560f6cae0bb0 legacy=0x560f6cae1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.309+0000 7f7b38561640 1 -- 172.21.15.110:0/948344252 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6c98d510 con 0x560f6cae0290 2024-04-27T03:18:05.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b38561640 1 --2- 172.21.15.110:0/948344252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cae0670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800a420 tx=0x7f7b2801afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728eec21c0276595 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3755f640 1 -- 172.21.15.110:0/948344252 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b280af070 con 0x560f6cae0290 2024-04-27T03:18:05.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3755f640 1 -- 172.21.15.110:0/948344252 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b280aa530 con 0x560f6cae0290 2024-04-27T03:18:05.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3755f640 1 -- 172.21.15.110:0/948344252 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b280aa850 con 0x560f6cae0290 2024-04-27T03:18:05.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3aa46980 1 -- 172.21.15.110:0/948344252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 msgr2=0x560f6cae0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/948344252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cae0670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800a420 tx=0x7f7b2801afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3aa46980 1 -- 172.21.15.110:0/948344252 shutdown_connections 2024-04-27T03:18:05.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/948344252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cae0670 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3aa46980 1 -- 172.21.15.110:0/948344252 >> 172.21.15.110:0/948344252 conn(0x560f6cadbb60 msgr2=0x560f6caddf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3aa46980 1 -- 172.21.15.110:0/948344252 shutdown_connections 2024-04-27T03:18:05.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.313+0000 7f7b3aa46980 1 -- 172.21.15.110:0/948344252 wait complete. 2024-04-27T03:18:05.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b3aa46980 1 Processor -- start 2024-04-27T03:18:05.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b3aa46980 1 -- start start 2024-04-27T03:18:05.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b3aa46980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6ca566d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b3aa46980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f6c9e32d0 con 0x560f6cae0290 2024-04-27T03:18:05.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b38561640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6ca566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b38561640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6ca566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50052/0 (socket says 172.21.15.110:50052) 2024-04-27T03:18:05.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b38561640 1 -- 172.21.15.110:0/3854798722 learned_addr learned my addr 172.21.15.110:0/3854798722 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b38561640 1 -- 172.21.15.110:0/3854798722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6c969ae0 con 0x560f6cae0290 2024-04-27T03:18:05.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7b38561640 1 --2- 172.21.15.110:0/3854798722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6ca566d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800a420 tx=0x7f7b280033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8717fc6a56faae server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7a2660c640 1 -- 172.21.15.110:0/3854798722 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b280af050 con 0x560f6cae0290 2024-04-27T03:18:05.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7a2660c640 1 -- 172.21.15.110:0/3854798722 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b280b0040 con 0x560f6cae0290 2024-04-27T03:18:05.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.329+0000 7f7a2660c640 1 -- 172.21.15.110:0/3854798722 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b280aac60 con 0x560f6cae0290 2024-04-27T03:18:05.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- 172.21.15.110:0/3854798722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 msgr2=0x560f6ca566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/3854798722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6ca566d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7b2800a420 tx=0x7f7b280033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- 172.21.15.110:0/3854798722 shutdown_connections 2024-04-27T03:18:05.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/3854798722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6ca566d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- 172.21.15.110:0/3854798722 >> 172.21.15.110:0/3854798722 conn(0x560f6cadbb60 msgr2=0x560f6cbc7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- 172.21.15.110:0/3854798722 shutdown_connections 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- 172.21.15.110:0/3854798722 wait complete. 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 Processor -- start 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- start start 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cbd4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f6caea2e0 con 0x560f6cae0290 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b38561640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cbd4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b38561640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cbd4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50068/0 (socket says 172.21.15.110:50068) 2024-04-27T03:18:05.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b38561640 1 -- 172.21.15.110:0/524259102 learned_addr learned my addr 172.21.15.110:0/524259102 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b38561640 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6cbd51b0 con 0x560f6cae0290 2024-04-27T03:18:05.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b38561640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cbd4c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7b2801b010 tx=0x7f7b28019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7a24e09640 1 -- 172.21.15.110:0/524259102 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b280c5030 con 0x560f6cae0290 2024-04-27T03:18:05.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f6cbd5440 con 0x560f6cae0290 2024-04-27T03:18:05.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7a24e09640 1 -- 172.21.15.110:0/524259102 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b280c15b0 con 0x560f6cae0290 2024-04-27T03:18:05.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f6cbd8020 con 0x560f6cae0290 2024-04-27T03:18:05.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7a24e09640 1 -- 172.21.15.110:0/524259102 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b280c1890 con 0x560f6cae0290 2024-04-27T03:18:05.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7a24e09640 1 -- 172.21.15.110:0/524259102 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7b280c8070 con 0x560f6cae0290 2024-04-27T03:18:05.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7a24e09640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a20034650 0x7f7a20036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.333+0000 7f7a24e09640 1 -- 172.21.15.110:0/524259102 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7b280c6070 con 0x560f6cae0290 2024-04-27T03:18:05.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b37d60640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a20034650 0x7f7a20036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cae0bb0 0x560f6cbdbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cbdc1b0 con 0x560f6cae0bb0 2024-04-27T03:18:05.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b38d62640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cae0bb0 0x560f6cbdbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b37d60640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a20034650 0x7f7a20036b10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001da0 tx=0x7f7b2c001300 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b38d62640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cae0bb0 0x560f6cbdbc70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b38d62640 1 -- 172.21.15.110:0/524259102 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7b300083f0 con 0x560f6cae0bb0 2024-04-27T03:18:05.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cbdcf40 0x560f6cbdf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cbdf920 con 0x560f6cbdcf40 2024-04-27T03:18:05.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b38561640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cbdcf40 0x560f6cbdf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b38561640 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cbdcf40 0x560f6cbdf330 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b38561640 1 -- 172.21.15.110:0/524259102 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cbdcf40 2024-04-27T03:18:05.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cbe11d0 con 0x560f6cae0bb0 2024-04-27T03:18:05.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b38d62640 1 -- 172.21.15.110:0/524259102 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f7b300083f0 con 0x560f6cae0bb0 2024-04-27T03:18:05.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.337+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cbe0400 con 0x560f6cbdcf40 2024-04-27T03:18:05.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b38561640 1 -- 172.21.15.110:0/524259102 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cbdcf40 2024-04-27T03:18:05.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cbe07f0 con 0x560f6cae0bb0 2024-04-27T03:18:05.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b38d62640 1 -- 172.21.15.110:0/524259102 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f7b300083f0 con 0x560f6cae0bb0 2024-04-27T03:18:05.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b3aa46980 1 Processor -- start 2024-04-27T03:18:05.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b3aa46980 1 -- start start 2024-04-27T03:18:05.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b3aa46980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbefad0 0x560f6cbefef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b3aa46980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f6c9e7310 con 0x560f6cbefad0 2024-04-27T03:18:05.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b38d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbefad0 0x560f6cbefef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b38d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbefad0 0x560f6cbefef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50078/0 (socket says 172.21.15.110:50078) 2024-04-27T03:18:05.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b38d62640 1 -- 172.21.15.110:0/2414492246 learned_addr learned my addr 172.21.15.110:0/2414492246 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b38d62640 1 -- 172.21.15.110:0/2414492246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6cbf0430 con 0x560f6cbefad0 2024-04-27T03:18:05.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b38d62640 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbefad0 0x560f6cbefef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000aab0 tx=0x7f7b3000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7a1dffb640 1 -- 172.21.15.110:0/2414492246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b30007d30 con 0x560f6cbefad0 2024-04-27T03:18:05.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7a1dffb640 1 -- 172.21.15.110:0/2414492246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b3001d070 con 0x560f6cbefad0 2024-04-27T03:18:05.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f6cbf0ed0 con 0x560f6cbefad0 2024-04-27T03:18:05.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7a1dffb640 1 -- 172.21.15.110:0/2414492246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b30018440 con 0x560f6cbefad0 2024-04-27T03:18:05.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.341+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f6cbf13a0 con 0x560f6cbefad0 2024-04-27T03:18:05.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7a1dffb640 1 -- 172.21.15.110:0/2414492246 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7b30013070 con 0x560f6cbefad0 2024-04-27T03:18:05.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7a1dffb640 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a10034650 0x7f7a10036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7a1dffb640 1 -- 172.21.15.110:0/2414492246 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7b3004b5f0 con 0x560f6cbefad0 2024-04-27T03:18:05.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38561640 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a10034650 0x7f7a10036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 Processor -- start 2024-04-27T03:18:05.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- start start 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cbfd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f6ca56d60 con 0x560f6cbfccd0 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38561640 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a10034650 0x7f7a10036b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x560f6cbdff20 tx=0x7f7b280e3600 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cbfd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cbfd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50094/0 (socket says 172.21.15.110:50094) 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38d62640 1 -- 172.21.15.110:0/1985968783 learned_addr learned my addr 172.21.15.110:0/1985968783 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38d62640 1 -- 172.21.15.110:0/1985968783 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6cbfd6a0 con 0x560f6cbfccd0 2024-04-27T03:18:05.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38d62640 1 --2- 172.21.15.110:0/1985968783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cbfd0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7b30049970 tx=0x7f7b3003a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954c79cbca70fda8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7a177fe640 1 -- 172.21.15.110:0/1985968783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b3003f020 con 0x560f6cbfccd0 2024-04-27T03:18:05.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7a177fe640 1 -- 172.21.15.110:0/1985968783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b300177c0 con 0x560f6cbfccd0 2024-04-27T03:18:05.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7a177fe640 1 -- 172.21.15.110:0/1985968783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b30022460 con 0x560f6cbfccd0 2024-04-27T03:18:05.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1985968783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 msgr2=0x560f6cbfd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1985968783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cbfd0d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7b30049970 tx=0x7f7b3003a110 comp rx=0 tx=0).stop 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1985968783 shutdown_connections 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1985968783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cbfd0d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1985968783 >> 172.21.15.110:0/1985968783 conn(0x560f6cbf9010 msgr2=0x560f6cbf7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1985968783 shutdown_connections 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1985968783 wait complete. 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 Processor -- start 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- start start 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cc041b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b3aa46980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560f6cbfde40 con 0x560f6cbfccd0 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.345+0000 7f7b38d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cc041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b38d62640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cc041b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50108/0 (socket says 172.21.15.110:50108) 2024-04-27T03:18:05.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 learned_addr learned my addr 172.21.15.110:0/1796014628 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f6cc046f0 con 0x560f6cbfccd0 2024-04-27T03:18:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b38d62640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cc041b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b300168b0 tx=0x7f7b300498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7a15ffb640 1 -- 172.21.15.110:0/1796014628 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b3002f020 con 0x560f6cbfccd0 2024-04-27T03:18:05.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f6cc04980 con 0x560f6cbfccd0 2024-04-27T03:18:05.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7a15ffb640 1 -- 172.21.15.110:0/1796014628 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b30020360 con 0x560f6cbfccd0 2024-04-27T03:18:05.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7a15ffb640 1 -- 172.21.15.110:0/1796014628 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b3003f040 con 0x560f6cbfccd0 2024-04-27T03:18:05.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f6cc07660 con 0x560f6cbfccd0 2024-04-27T03:18:05.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7a15ffb640 1 -- 172.21.15.110:0/1796014628 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7b30032020 con 0x560f6cbfccd0 2024-04-27T03:18:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7a15ffb640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a080345b0 0x7f7a08036a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7a15ffb640 1 -- 172.21.15.110:0/1796014628 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7b30035c40 con 0x560f6cbfccd0 2024-04-27T03:18:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b38561640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a080345b0 0x7f7a08036a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cc0f7b0 0x560f6cc11c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc123c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b37d60640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cc0f7b0 0x560f6cc11c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b38561640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a080345b0 0x7f7a08036a70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b280b8040 tx=0x7f7b280e1000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.349+0000 7f7b37d60640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cc0f7b0 0x560f6cc11c70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7b2c00e640 con 0x560f6cc0f7b0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 realm 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc0ec00 con 0x560f6cc0f7b0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7b2c00e640 con 0x560f6cc0f7b0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc12ec0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7b2c00e640 con 0x560f6cc0f7b0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cc13ca0 0x560f6cc16090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc16710 con 0x560f6cc13ca0 2024-04-27T03:18:05.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b38d62640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cc13ca0 0x560f6cc16090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b38d62640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cc13ca0 0x560f6cc16090 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc16f60 con 0x560f6cc0f7b0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f7b2c00e640 con 0x560f6cc0f7b0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 20 searching for the correct realm 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f6cc18cd0 0x560f6cc1b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc18cd0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b38561640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f6cc18cd0 0x560f6cc1b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b38561640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f6cc18cd0 0x560f6cc1b0c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.353+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b38d62640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7b30034020 0x7f7b300137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x7f7b30034020 2024-04-27T03:18:05.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7b30034020 0x7f7b300137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7b30034020 0x7f7b300137e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c00e640 con 0x7f7b30034020 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c00e640 con 0x560f6cc0f7b0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc18cd0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.357+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x7f7b30034020 2024-04-27T03:18:05.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x7f7b30034020 2024-04-27T03:18:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x7f7b30034020 2024-04-27T03:18:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f7b2c029040 con 0x7f7b30034020 2024-04-27T03:18:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc18cd0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v20'1 uv1 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.361+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc18cd0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc18cd0 2024-04-27T03:18:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x7f7b30034020 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x7f7b30034020 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc0f7b0 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x7f7b30034020 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x7f7b30034020 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc18cd0 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x7f7b30034020 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b2c029040 con 0x7f7b30034020 2024-04-27T03:18:05.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.365+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc18cd0 2024-04-27T03:18:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1b740 con 0x560f6cc13ca0 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:05.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc178e0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7b2c029040 con 0x560f6cc0f7b0 2024-04-27T03:18:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc18180 con 0x560f6cc13ca0 2024-04-27T03:18:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc17520 con 0x560f6cc0f7b0 2024-04-27T03:18:05.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 zone default found 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 4 Realm: () 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 zonegroup default 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.369+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc17d90 con 0x7f7b30034020 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1daa0 con 0x560f6cc18cd0 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1de70 con 0x560f6cc18cd0 2024-04-27T03:18:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 20 started sync module instance, tier type = 2024-04-27T03:18:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:05.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1cf40 con 0x560f6cc13ca0 2024-04-27T03:18:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94624249140016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1cba0 con 0x560f6cc13ca0 2024-04-27T03:18:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.373+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1ed80 con 0x560f6cc0f7b0 2024-04-27T03:18:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94624249145552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1f1a0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1fc50 con 0x560f6cc13ca0 2024-04-27T03:18:05.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94624249140016] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94624249145552] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94624249143984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1ecf0 con 0x560f6cc13ca0 2024-04-27T03:18:05.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc204f0 con 0x7f7b30034020 2024-04-27T03:18:05.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94624249151552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1ecf0 con 0x7f7b30034020 2024-04-27T03:18:05.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.377+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc20fb0 con 0x560f6cc13ca0 2024-04-27T03:18:05.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94624249143984] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94624249151552] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94624249152608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc20f70 con 0x560f6cc13ca0 2024-04-27T03:18:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc213a0 con 0x560f6cc18cd0 2024-04-27T03:18:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94624249149232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc21c40 con 0x560f6cc18cd0 2024-04-27T03:18:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc22600 con 0x560f6cc18cd0 2024-04-27T03:18:05.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.381+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94624249152608] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94624249160016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc22570 con 0x560f6cc18cd0 2024-04-27T03:18:05.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc234c0 con 0x7f7b30034020 2024-04-27T03:18:05.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94624249149232] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94624249160016] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'15 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94624249162224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc23010 con 0x7f7b30034020 2024-04-27T03:18:05.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=0 2024-04-27T03:18:05.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=1 2024-04-27T03:18:05.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=2 2024-04-27T03:18:05.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=3 2024-04-27T03:18:05.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=4 2024-04-27T03:18:05.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=5 2024-04-27T03:18:05.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=6 2024-04-27T03:18:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94624249162224] v27'16 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 20 add_watcher() i=7 2024-04-27T03:18:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc23010 con 0x560f6cc0f7b0 2024-04-27T03:18:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.385+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.389+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94624249157520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc23bf0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.389+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94624249157520] v27'47 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.389+0000 7f79ee7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.389+0000 7f7b3aa46980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.389+0000 7f7b3aa46980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.389+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2eee0 con 0x560f6cc13ca0 2024-04-27T03:18:05.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.393+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'43 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.393+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2efc0 con 0x7f7b30034020 2024-04-27T03:18:05.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.393+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'19 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.393+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2f780 con 0x560f6cc18cd0 2024-04-27T03:18:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.397+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'19 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.397+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2fc30 con 0x560f6cc0f7b0 2024-04-27T03:18:05.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.397+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'48 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.397+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc13ca0 2024-04-27T03:18:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.401+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'37 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.401+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc18cd0 2024-04-27T03:18:05.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.401+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'31 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.401+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc13ca0 2024-04-27T03:18:05.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.401+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'21 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.405+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc0f7b0 2024-04-27T03:18:05.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.405+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'26 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.405+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30790 con 0x560f6cc13ca0 2024-04-27T03:18:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.405+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'13 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.405+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30790 con 0x560f6cc13ca0 2024-04-27T03:18:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.409+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'19 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.409+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc13ca0 2024-04-27T03:18:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.409+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'38 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.409+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc18cd0 2024-04-27T03:18:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.413+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'15 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.413+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc13ca0 2024-04-27T03:18:05.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.413+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'19 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.413+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc0f7b0 2024-04-27T03:18:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.417+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'37 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.417+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc0f7b0 2024-04-27T03:18:05.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.417+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'49 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.417+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc13ca0 2024-04-27T03:18:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.421+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'29 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.421+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc13ca0 2024-04-27T03:18:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.421+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'39 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.421+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ab30 con 0x560f6cc18cd0 2024-04-27T03:18:05.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.425+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'25 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.425+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2abe0 con 0x560f6cc13ca0 2024-04-27T03:18:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.429+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'21 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.429+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2abe0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.429+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'50 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.429+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc13ca0 2024-04-27T03:18:05.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.433+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'30 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.433+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc13ca0 2024-04-27T03:18:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.437+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'29 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.437+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc18cd0 2024-04-27T03:18:05.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.437+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'21 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.437+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc13ca0 2024-04-27T03:18:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.441+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'31 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.441+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc18cd0 2024-04-27T03:18:05.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.441+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'15 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.441+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc18cd0 2024-04-27T03:18:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.445+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'32 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.445+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc18cd0 2024-04-27T03:18:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.445+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'21 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.445+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc13ca0 2024-04-27T03:18:05.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.449+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'25 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.449+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.449+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'27 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.449+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc13ca0 2024-04-27T03:18:05.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.453+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'31 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.453+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.453+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'38 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.453+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc30ca0 con 0x7f7b30034020 2024-04-27T03:18:05.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'19 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f79ed7e2640 20 reqs_thread_entry: start 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 20 Filter name: none 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f795effd640 20 reqs_thread_entry: start 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2d1a0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(94 foo [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x0 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:05.466 INFO:teuthology.orchestra.run.smithi110.stderr:could not remove user: unable to remove user, user does not exist 2024-04-27T03:18:05.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f79ee7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.457+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94624249157520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc2ce30 con 0x560f6cc0f7b0 2024-04-27T03:18:05.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.461+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(95 data_loggenerations_metadata [watch unwatch cookie 94624249157520] v27'51 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.461+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94624249140016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc21790 con 0x560f6cc13ca0 2024-04-27T03:18:05.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(96 notify.0 [watch unwatch cookie 94624249140016] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b3aa46980 20 remove_watcher() i=0 2024-04-27T03:18:05.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b3aa46980 2 removed watcher, disabling cache 2024-04-27T03:18:05.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:97 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94624249145552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1d330 con 0x560f6cc0f7b0 2024-04-27T03:18:05.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(97 notify.1 [watch unwatch cookie 94624249145552] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x560f6cc0f7b0 2024-04-27T03:18:05.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b3aa46980 20 remove_watcher() i=1 2024-04-27T03:18:05.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94624249143984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1e8d0 con 0x560f6cc13ca0 2024-04-27T03:18:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(98 notify.2 [watch unwatch cookie 94624249143984] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b3aa46980 20 remove_watcher() i=2 2024-04-27T03:18:05.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.465+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:99 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94624249151552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1e8d0 con 0x7f7b30034020 2024-04-27T03:18:05.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.469+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(99 notify.3 [watch unwatch cookie 94624249151552] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.469+0000 7f7b3aa46980 20 remove_watcher() i=3 2024-04-27T03:18:05.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.469+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94624249152608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc20040 con 0x560f6cc13ca0 2024-04-27T03:18:05.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.469+0000 7f7b38d62640 1 -- 172.21.15.110:0/1796014628 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(100 notify.4 [watch unwatch cookie 94624249152608] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b30041050 con 0x560f6cc13ca0 2024-04-27T03:18:05.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.469+0000 7f7b3aa46980 20 remove_watcher() i=4 2024-04-27T03:18:05.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.469+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94624249149232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1e8d0 con 0x560f6cc18cd0 2024-04-27T03:18:05.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(101 notify.5 [watch unwatch cookie 94624249149232] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 20 remove_watcher() i=5 2024-04-27T03:18:05.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94624249160016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc1e8d0 con 0x560f6cc18cd0 2024-04-27T03:18:05.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b38561640 1 -- 172.21.15.110:0/1796014628 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(102 notify.6 [watch unwatch cookie 94624249160016] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b280b2070 con 0x560f6cc18cd0 2024-04-27T03:18:05.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 20 remove_watcher() i=6 2024-04-27T03:18:05.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:103 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94624249162224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x560f6cc22150 con 0x7f7b30034020 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(103 notify.7 [watch unwatch cookie 94624249162224] v27'17 uv15 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7b2c03c1c0 con 0x7f7b30034020 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 20 remove_watcher() i=7 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cc13ca0 msgr2=0x560f6cc16090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cc13ca0 0x560f6cc16090 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cc0f7b0 msgr2=0x560f6cc11c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cc0f7b0 0x560f6cc11c70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7b30034020 msgr2=0x7f7b300137e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7b30034020 0x7f7b300137e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f6cc18cd0 msgr2=0x560f6cc1b0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560f6cc18cd0 0x560f6cc1b0c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a080345b0 msgr2=0x7f7a08036a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a080345b0 0x7f7a08036a70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b280b8040 tx=0x7f7b280e1000 comp rx=0 tx=0).stop 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 msgr2=0x560f6cc041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/1796014628 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbfccd0 0x560f6cc041b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b300168b0 tx=0x7f7b300498d0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 reap_dead start 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b37d60640 1 -- 172.21.15.110:0/1796014628 reap_dead start 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 shutdown_connections 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 >> 172.21.15.110:0/1796014628 conn(0x560f6cbf9010 msgr2=0x560f6cbf6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 shutdown_connections 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.473+0000 7f7b3aa46980 1 -- 172.21.15.110:0/1796014628 wait complete. 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a10034650 msgr2=0x7f7a10036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a10034650 0x7f7a10036b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x560f6cbdff20 tx=0x7f7b280e3600 comp rx=0 tx=0).stop 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbefad0 msgr2=0x560f6cbefef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbefad0 0x560f6cbefef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000aab0 tx=0x7f7b3000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 shutdown_connections 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a10034650 0x7f7a10036b10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/2414492246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cbefad0 0x560f6cbefef0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 >> 172.21.15.110:0/2414492246 conn(0x560f6cbe48e0 msgr2=0x560f6cbe53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 shutdown_connections 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/2414492246 wait complete. 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cbdcf40 msgr2=0x560f6cbdf330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cbdcf40 0x560f6cbdf330 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cae0bb0 msgr2=0x560f6cbdbc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cae0bb0 0x560f6cbdbc70 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a20034650 msgr2=0x7f7a20036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a20034650 0x7f7a20036b10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001da0 tx=0x7f7b2c001300 comp rx=0 tx=0).stop 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 msgr2=0x560f6cbd4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cbd4c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7b2801b010 tx=0x7f7b28019810 comp rx=0 tx=0).stop 2024-04-27T03:18:05.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 shutdown_connections 2024-04-27T03:18:05.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560f6cbdcf40 0x560f6cbdf330 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560f6cae0bb0 0x560f6cbdbc70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7a20034650 0x7f7a20036b10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 --2- 172.21.15.110:0/524259102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560f6cae0290 0x560f6cbd4c70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 >> 172.21.15.110:0/524259102 conn(0x560f6cadbb60 msgr2=0x560f6ca55140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 shutdown_connections 2024-04-27T03:18:05.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.477+0000 7f7b3aa46980 1 -- 172.21.15.110:0/524259102 wait complete. 2024-04-27T03:18:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-27T03:18:05.496 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2024-04-27T03:18:05.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2024-04-27T03:18:05.496 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement rm --rgw-zone default --placement-id new-placement 2024-04-27T03:18:05.574 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:05.574 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:05.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.565+0000 7ff385fad980 1 Processor -- start 2024-04-27T03:18:05.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.565+0000 7ff385fad980 1 -- start start 2024-04-27T03:18:05.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.565+0000 7ff385fad980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5586f24cf9c0 0x5586f24d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.565+0000 7ff385fad980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5586f23880d0 con 0x5586f24cb4c0 2024-04-27T03:18:05.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.565+0000 7ff385fad980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5586f237d430 con 0x5586f24cf9c0 2024-04-27T03:18:05.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff3832c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5586f24cf9c0 0x5586f24d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff3832c7640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5586f24cf9c0 0x5586f24d1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50116/0 (socket says 172.21.15.110:50116) 2024-04-27T03:18:05.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff3832c7640 1 -- 172.21.15.110:0/2035597475 learned_addr learned my addr 172.21.15.110:0/2035597475 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff3832c7640 1 -- 172.21.15.110:0/2035597475 >> v1:172.21.15.77:6789/0 conn(0x5586f24cb4c0 legacy=0x5586f24cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff3832c7640 1 -- 172.21.15.110:0/2035597475 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f2376510 con 0x5586f24cf9c0 2024-04-27T03:18:05.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff3832c7640 1 --2- 172.21.15.110:0/2035597475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cf9c0 0x5586f24d1db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff37800a5c0 tx=0x7ff37801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d09bf68e1c46eca server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff382ac6640 1 -- 172.21.15.110:0/2035597475 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3780a6070 con 0x5586f24cf9c0 2024-04-27T03:18:05.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff382ac6640 1 -- 172.21.15.110:0/2035597475 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3780af070 con 0x5586f24cf9c0 2024-04-27T03:18:05.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.569+0000 7ff382ac6640 1 -- 172.21.15.110:0/2035597475 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3780aa5c0 con 0x5586f24cf9c0 2024-04-27T03:18:05.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.573+0000 7ff385fad980 1 -- 172.21.15.110:0/2035597475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cf9c0 msgr2=0x5586f24d1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.573+0000 7ff385fad980 1 --2- 172.21.15.110:0/2035597475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cf9c0 0x5586f24d1db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff37800a5c0 tx=0x7ff37801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:05.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.573+0000 7ff385fad980 1 -- 172.21.15.110:0/2035597475 shutdown_connections 2024-04-27T03:18:05.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.573+0000 7ff385fad980 1 --2- 172.21.15.110:0/2035597475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cf9c0 0x5586f24d1db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.573+0000 7ff385fad980 1 -- 172.21.15.110:0/2035597475 >> 172.21.15.110:0/2035597475 conn(0x5586f243ecc0 msgr2=0x5586f24c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.573+0000 7ff385fad980 1 -- 172.21.15.110:0/2035597475 shutdown_connections 2024-04-27T03:18:05.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.573+0000 7ff385fad980 1 -- 172.21.15.110:0/2035597475 wait complete. 2024-04-27T03:18:05.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.585+0000 7ff385fad980 1 Processor -- start 2024-04-27T03:18:05.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- start start 2024-04-27T03:18:05.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25b6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5586f23cc2d0 con 0x5586f24cb4c0 2024-04-27T03:18:05.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25b6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50118/0 (socket says 172.21.15.110:50118) 2024-04-27T03:18:05.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 -- 172.21.15.110:0/90911100 learned_addr learned my addr 172.21.15.110:0/90911100 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 -- 172.21.15.110:0/90911100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f2352ae0 con 0x5586f24cb4c0 2024-04-27T03:18:05.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 --2- 172.21.15.110:0/90911100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25b6620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff3740016c0 tx=0x7ff374001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79f1057f95b9192 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff271e0b640 1 -- 172.21.15.110:0/90911100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374016070 con 0x5586f24cb4c0 2024-04-27T03:18:05.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff271e0b640 1 -- 172.21.15.110:0/90911100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff37400f640 con 0x5586f24cb4c0 2024-04-27T03:18:05.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff271e0b640 1 -- 172.21.15.110:0/90911100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37400f9a0 con 0x5586f24cb4c0 2024-04-27T03:18:05.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- 172.21.15.110:0/90911100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 msgr2=0x5586f25b6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 --2- 172.21.15.110:0/90911100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25b6620 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff3740016c0 tx=0x7ff374001b80 comp rx=0 tx=0).stop 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- 172.21.15.110:0/90911100 shutdown_connections 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 --2- 172.21.15.110:0/90911100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25b6620 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- 172.21.15.110:0/90911100 >> 172.21.15.110:0/90911100 conn(0x5586f243ecc0 msgr2=0x5586f25b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- 172.21.15.110:0/90911100 shutdown_connections 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- 172.21.15.110:0/90911100 wait complete. 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 Processor -- start 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- start start 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25bd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff385fad980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5586f24d4de0 con 0x5586f24cb4c0 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25bd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50130/0 (socket says 172.21.15.110:50130) 2024-04-27T03:18:05.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.589+0000 7ff383ac8640 1 -- 172.21.15.110:0/578594560 learned_addr learned my addr 172.21.15.110:0/578594560 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff383ac8640 1 -- 172.21.15.110:0/578594560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f25bdc80 con 0x5586f24cb4c0 2024-04-27T03:18:05.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff383ac8640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25bd740 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff37400f610 tx=0x7ff374001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff26bfff640 1 -- 172.21.15.110:0/578594560 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374021020 con 0x5586f24cb4c0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff26bfff640 1 -- 172.21.15.110:0/578594560 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff374011770 con 0x5586f24cb4c0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff26bfff640 1 -- 172.21.15.110:0/578594560 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374016040 con 0x5586f24cb4c0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5586f25bdf10 con 0x5586f24cb4c0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586f25c0ad0 con 0x5586f24cb4c0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 Processor -- start 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- start start 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25caef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5586f23d0310 con 0x5586f25caad0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff3842c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff3842c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50144/0 (socket says 172.21.15.110:50144) 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff3842c9640 1 -- 172.21.15.110:0/801097877 learned_addr learned my addr 172.21.15.110:0/801097877 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff26bfff640 1 -- 172.21.15.110:0/578594560 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff374024020 con 0x5586f24cb4c0 2024-04-27T03:18:05.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff26bfff640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff26c0346a0 0x7ff26c036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff26bfff640 1 -- 172.21.15.110:0/578594560 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7ff374049620 con 0x5586f24cb4c0 2024-04-27T03:18:05.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff383ac8640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff26c0346a0 0x7ff26c036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff3842c9640 1 -- 172.21.15.110:0/801097877 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f25cb430 con 0x5586f25caad0 2024-04-27T03:18:05.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff3842c9640 1 --2- 172.21.15.110:0/801097877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25caef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00a440 tx=0x7ff37c00a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffb901c397702ab server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff383ac8640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff26c0346a0 0x7ff26c036b60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff374011be0 tx=0x7ff374011af0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff269ffb640 1 -- 172.21.15.110:0/801097877 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c008850 con 0x5586f25caad0 2024-04-27T03:18:05.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff269ffb640 1 -- 172.21.15.110:0/801097877 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff37c008a10 con 0x5586f25caad0 2024-04-27T03:18:05.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff269ffb640 1 -- 172.21.15.110:0/801097877 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c008d30 con 0x5586f25caad0 2024-04-27T03:18:05.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- 172.21.15.110:0/801097877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 msgr2=0x5586f25caef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 --2- 172.21.15.110:0/801097877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25caef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff37c00a440 tx=0x7ff37c00a900 comp rx=0 tx=0).stop 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- 172.21.15.110:0/801097877 shutdown_connections 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 --2- 172.21.15.110:0/801097877 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25caef0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- 172.21.15.110:0/801097877 >> 172.21.15.110:0/801097877 conn(0x5586f24cf9c0 msgr2=0x5586f25c5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- 172.21.15.110:0/801097877 shutdown_connections 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.593+0000 7ff385fad980 1 -- 172.21.15.110:0/801097877 wait complete. 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 Processor -- start 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 -- start start 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25d1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5586f25cbdf0 con 0x5586f25caad0 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff3842c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25d1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff3842c9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25d1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50156/0 (socket says 172.21.15.110:50156) 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff3842c9640 1 -- 172.21.15.110:0/1893796225 learned_addr learned my addr 172.21.15.110:0/1893796225 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff3842c9640 1 -- 172.21.15.110:0/1893796225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f25d23a0 con 0x5586f25caad0 2024-04-27T03:18:05.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff3842c9640 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25d1e60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5586f25cb8b0 tx=0x7ff37c003000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff263fff640 1 -- 172.21.15.110:0/1893796225 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c013a70 con 0x5586f25caad0 2024-04-27T03:18:05.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5586f25d2630 con 0x5586f25caad0 2024-04-27T03:18:05.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff263fff640 1 -- 172.21.15.110:0/1893796225 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff37c023070 con 0x5586f25caad0 2024-04-27T03:18:05.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff263fff640 1 -- 172.21.15.110:0/1893796225 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff37c013d40 con 0x5586f25caad0 2024-04-27T03:18:05.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586f25d5310 con 0x5586f25caad0 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff263fff640 1 -- 172.21.15.110:0/1893796225 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff37c028460 con 0x5586f25caad0 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff263fff640 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff25c034650 0x7ff25c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff263fff640 1 -- 172.21.15.110:0/1893796225 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7ff37c049f70 con 0x5586f25caad0 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 Processor -- start 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff3842c9640 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff25c034650 0x7ff25c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 -- start start 2024-04-27T03:18:05.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25ebcb0 0x5586f25ec0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.597+0000 7ff385fad980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5586f25b6d70 con 0x5586f25ebcb0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff3842c9640 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff25c034650 0x7ff25c036b10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff37c013fd0 tx=0x7ff37c013f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff383ac8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25ebcb0 0x5586f25ec0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff383ac8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25ebcb0 0x5586f25ec0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50172/0 (socket says 172.21.15.110:50172) 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff383ac8640 1 -- 172.21.15.110:0/2687902194 learned_addr learned my addr 172.21.15.110:0/2687902194 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff383ac8640 1 -- 172.21.15.110:0/2687902194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f25ec610 con 0x5586f25ebcb0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff383ac8640 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25ebcb0 0x5586f25ec0d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff374001340 tx=0x7ff3740479c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff1dbfff640 1 -- 172.21.15.110:0/2687902194 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374011c50 con 0x5586f25ebcb0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5586f25ed0b0 con 0x5586f25ebcb0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff1dbfff640 1 -- 172.21.15.110:0/2687902194 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff374011df0 con 0x5586f25ebcb0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff1dbfff640 1 -- 172.21.15.110:0/2687902194 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff374010b30 con 0x5586f25ebcb0 2024-04-27T03:18:05.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586f25ed580 con 0x5586f25ebcb0 2024-04-27T03:18:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff1dbfff640 1 -- 172.21.15.110:0/2687902194 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff37402d300 con 0x5586f25ebcb0 2024-04-27T03:18:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff1dbfff640 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff1bc034650 0x7ff1bc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff1dbfff640 1 -- 172.21.15.110:0/2687902194 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7ff374046c80 con 0x5586f25ebcb0 2024-04-27T03:18:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff3832c7640 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff1bc034650 0x7ff1bc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff1d9ffb640 20 reqs_thread_entry: start 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5586f25f1890 0x5586f25f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:1 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5586f25f4330 con 0x5586f25f1890 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff3832c7640 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff1bc034650 0x7ff1bc036b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3780ae000 tx=0x7ff3780a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.601+0000 7ff3842c9640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5586f25f1890 0x5586f25f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff3842c9640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5586f25f1890 0x5586f25f3c80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff3842c9640 1 -- 172.21.15.110:0/578594560 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(1 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ff37c027600 con 0x5586f25f1890 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5586f25f6100 0x5586f25f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5586f25f8bf0 con 0x5586f25f6100 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff383ac8640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5586f25f6100 0x5586f25f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff383ac8640 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5586f25f6100 0x5586f25f84f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff383ac8640 1 -- 172.21.15.110:0/578594560 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(2 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv1 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7ff37400f070 con 0x5586f25f6100 2024-04-27T03:18:05.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [stat,call version.check_conds in=122b,call version.inc in=38b,writefull 0~993 in=993b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5586f25fd990 con 0x5586f25f6100 2024-04-27T03:18:05.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.605+0000 7ff383ac8640 1 -- 172.21.15.110:0/578594560 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [stat,call,call,writefull 0~993] v27'2 uv2 ondisk = 0) ==== 316+0+0 (crc 0 0 0) 0x7ff37400f070 con 0x5586f25f6100 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff25c034650 msgr2=0x7ff25c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff25c034650 0x7ff25c036b10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff37c013fd0 tx=0x7ff37c013f60 comp rx=0 tx=0).stop 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 msgr2=0x5586f25d1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25d1e60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5586f25cb8b0 tx=0x7ff37c003000 comp rx=0 tx=0).stop 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 shutdown_connections 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff25c034650 0x7ff25c036b10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/1893796225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25caad0 0x5586f25d1e60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 >> 172.21.15.110:0/1893796225 conn(0x5586f24cf9c0 msgr2=0x5586f25c4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 shutdown_connections 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/1893796225 wait complete. 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff1bc034650 msgr2=0x7ff1bc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff1bc034650 0x7ff1bc036b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3780ae000 tx=0x7ff3780a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25ebcb0 msgr2=0x5586f25ec0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25ebcb0 0x5586f25ec0d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff374001340 tx=0x7ff3740479c0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 shutdown_connections 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff1bc034650 0x7ff1bc036b10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/2687902194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f25ebcb0 0x5586f25ec0d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 >> 172.21.15.110:0/2687902194 conn(0x5586f25e0890 msgr2=0x5586f25e12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 shutdown_connections 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/2687902194 wait complete. 2024-04-27T03:18:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5586f25f1890 msgr2=0x5586f25f3c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5586f25f1890 0x5586f25f3c80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5586f25f6100 msgr2=0x5586f25f84f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5586f25f6100 0x5586f25f84f0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff26c0346a0 msgr2=0x7ff26c036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff26c0346a0 0x7ff26c036b60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff374011be0 tx=0x7ff374011af0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 msgr2=0x5586f25bd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25bd740 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff37400f610 tx=0x7ff374001260 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 shutdown_connections 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5586f25f6100 0x5586f25f84f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5586f25f1890 0x5586f25f3c80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff26c0346a0 0x7ff26c036b60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 --2- 172.21.15.110:0/578594560 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5586f24cb4c0 0x5586f25bd740 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 >> 172.21.15.110:0/578594560 conn(0x5586f243ecc0 msgr2=0x5586f25b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 shutdown_connections 2024-04-27T03:18:05.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.609+0000 7ff385fad980 1 -- 172.21.15.110:0/578594560 wait complete. 2024-04-27T03:18:05.627 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"27a51273-3ccc-49e2-8682-1122abc3d898","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":""} 2024-04-27T03:18:05.627 DEBUG:tasks.util.rgw: json result: {'id': '27a51273-3ccc-49e2-8682-1122abc3d898', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': ''} 2024-04-27T03:18:05.628 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show'] 2024-04-27T03:18:05.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show'] 2024-04-27T03:18:05.629 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show 2024-04-27T03:18:05.701 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:05.701 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:05.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb78a4e7980 1 Processor -- start 2024-04-27T03:18:05.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb78a4e7980 1 -- start start 2024-04-27T03:18:05.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb78a4e7980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x555b48df99b0 0x555b48dfbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb78a4e7980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x555b48cb20d0 con 0x555b48df54a0 2024-04-27T03:18:05.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb78a4e7980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x555b48ca7430 con 0x555b48df99b0 2024-04-27T03:18:05.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb787801640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x555b48df99b0 0x555b48dfbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb787801640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x555b48df99b0 0x555b48dfbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50188/0 (socket says 172.21.15.110:50188) 2024-04-27T03:18:05.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb787801640 1 -- 172.21.15.110:0/3048438800 learned_addr learned my addr 172.21.15.110:0/3048438800 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb787801640 1 -- 172.21.15.110:0/3048438800 >> v1:172.21.15.77:6789/0 conn(0x555b48df54a0 legacy=0x555b48df5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.693+0000 7fb787801640 1 -- 172.21.15.110:0/3048438800 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48ca0510 con 0x555b48df99b0 2024-04-27T03:18:05.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb787801640 1 --2- 172.21.15.110:0/3048438800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df99b0 0x555b48dfbda0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00a5c0 tx=0x7fb77c01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a4562487984605 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb787000640 1 -- 172.21.15.110:0/3048438800 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb77c0a6070 con 0x555b48df99b0 2024-04-27T03:18:05.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb787000640 1 -- 172.21.15.110:0/3048438800 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77c0af070 con 0x555b48df99b0 2024-04-27T03:18:05.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb787000640 1 -- 172.21.15.110:0/3048438800 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb77c0aa5c0 con 0x555b48df99b0 2024-04-27T03:18:05.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3048438800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df99b0 msgr2=0x555b48dfbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3048438800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df99b0 0x555b48dfbda0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00a5c0 tx=0x7fb77c01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:05.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3048438800 shutdown_connections 2024-04-27T03:18:05.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3048438800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df99b0 0x555b48dfbda0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3048438800 >> 172.21.15.110:0/3048438800 conn(0x555b48d68b80 msgr2=0x555b48d68f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3048438800 shutdown_connections 2024-04-27T03:18:05.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.697+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3048438800 wait complete. 2024-04-27T03:18:05.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.713+0000 7fb78a4e7980 1 Processor -- start 2024-04-27T03:18:05.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.713+0000 7fb78a4e7980 1 -- start start 2024-04-27T03:18:05.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.713+0000 7fb78a4e7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.713+0000 7fb78a4e7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b48cf62d0 con 0x555b48df54a0 2024-04-27T03:18:05.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.713+0000 7fb788002640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.713+0000 7fb788002640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50200/0 (socket says 172.21.15.110:50200) 2024-04-27T03:18:05.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.713+0000 7fb788002640 1 -- 172.21.15.110:0/461678486 learned_addr learned my addr 172.21.15.110:0/461678486 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb788002640 1 -- 172.21.15.110:0/461678486 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48c7cae0 con 0x555b48df54a0 2024-04-27T03:18:05.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb788002640 1 --2- 172.21.15.110:0/461678486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee0620 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb7780015f0 tx=0x7fb778001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe0b73cacbbaa7b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb675e0b640 1 -- 172.21.15.110:0/461678486 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb778016070 con 0x555b48df54a0 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb675e0b640 1 -- 172.21.15.110:0/461678486 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77800f640 con 0x555b48df54a0 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb675e0b640 1 -- 172.21.15.110:0/461678486 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb77800f960 con 0x555b48df54a0 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 -- 172.21.15.110:0/461678486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 msgr2=0x555b48ee0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/461678486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee0620 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb7780015f0 tx=0x7fb778001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 -- 172.21.15.110:0/461678486 shutdown_connections 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/461678486 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee0620 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 -- 172.21.15.110:0/461678486 >> 172.21.15.110:0/461678486 conn(0x555b48d68b80 msgr2=0x555b48edc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 -- 172.21.15.110:0/461678486 shutdown_connections 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 -- 172.21.15.110:0/461678486 wait complete. 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 Processor -- start 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 -- start start 2024-04-27T03:18:05.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb78a4e7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b48dfedd0 con 0x555b48df54a0 2024-04-27T03:18:05.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb788002640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb788002640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50214/0 (socket says 172.21.15.110:50214) 2024-04-27T03:18:05.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb788002640 1 -- 172.21.15.110:0/3025675214 learned_addr learned my addr 172.21.15.110:0/3025675214 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb788002640 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48ee7c00 con 0x555b48df54a0 2024-04-27T03:18:05.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb788002640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee76c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb77800f610 tx=0x7fb778000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.717+0000 7fb66ffff640 1 -- 172.21.15.110:0/3025675214 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb778021020 con 0x555b48df54a0 2024-04-27T03:18:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b48ee7e90 con 0x555b48df54a0 2024-04-27T03:18:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb66ffff640 1 -- 172.21.15.110:0/3025675214 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb778011670 con 0x555b48df54a0 2024-04-27T03:18:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb66ffff640 1 -- 172.21.15.110:0/3025675214 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb778016040 con 0x555b48df54a0 2024-04-27T03:18:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b48eeab10 con 0x555b48df54a0 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb66ffff640 1 -- 172.21.15.110:0/3025675214 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb778024020 con 0x555b48df54a0 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb66ffff640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6700346a0 0x7fb670036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb66ffff640 1 -- 172.21.15.110:0/3025675214 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb778049780 con 0x555b48df54a0 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb787801640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6700346a0 0x7fb670036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48df99b0 0x555b48eee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48eeec10 con 0x555b48df99b0 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb788803640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48df99b0 0x555b48eee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb787801640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6700346a0 0x7fb670036b60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0ae000 tx=0x7fb77c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb788803640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48df99b0 0x555b48eee5f0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb788803640 1 -- 172.21.15.110:0/3025675214 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb7800083f0 con 0x555b48df99b0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48eefa00 0x555b48ef1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48ef2430 con 0x555b48eefa00 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb788002640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48eefa00 0x555b48ef1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.721+0000 7fb788002640 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48eefa00 0x555b48ef1df0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788002640 1 -- 172.21.15.110:0/3025675214 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fb77800f070 con 0x555b48eefa00 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48ef3ce0 con 0x555b48df99b0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788803640 1 -- 172.21.15.110:0/3025675214 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fb7800083f0 con 0x555b48df99b0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48ef2f10 con 0x555b48eefa00 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788002640 1 -- 172.21.15.110:0/3025675214 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fb77800f070 con 0x555b48eefa00 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48ef3330 con 0x555b48df99b0 2024-04-27T03:18:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788803640 1 -- 172.21.15.110:0/3025675214 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fb7800083f0 con 0x555b48df99b0 2024-04-27T03:18:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb78a4e7980 1 Processor -- start 2024-04-27T03:18:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb78a4e7980 1 -- start start 2024-04-27T03:18:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb78a4e7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f025f0 0x555b48f02a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb78a4e7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b48cfa310 con 0x555b48f025f0 2024-04-27T03:18:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f025f0 0x555b48f02a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f025f0 0x555b48f02a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50222/0 (socket says 172.21.15.110:50222) 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788803640 1 -- 172.21.15.110:0/2155471086 learned_addr learned my addr 172.21.15.110:0/2155471086 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788803640 1 -- 172.21.15.110:0/2155471086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48f02f50 con 0x555b48f025f0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.725+0000 7fb788803640 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f025f0 0x555b48f02a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb78000aab0 tx=0x7fb78000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb66d7fa640 1 -- 172.21.15.110:0/2155471086 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb780007d30 con 0x555b48f025f0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb66d7fa640 1 -- 172.21.15.110:0/2155471086 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb78001d070 con 0x555b48f025f0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b48f039f0 con 0x555b48f025f0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb66d7fa640 1 -- 172.21.15.110:0/2155471086 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb780018440 con 0x555b48f025f0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b48f03ec0 con 0x555b48f025f0 2024-04-27T03:18:05.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb66d7fa640 1 -- 172.21.15.110:0/2155471086 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb780013070 con 0x555b48f025f0 2024-04-27T03:18:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb66d7fa640 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6600346a0 0x7fb660036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb66d7fa640 1 -- 172.21.15.110:0/2155471086 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb78004b490 con 0x555b48f025f0 2024-04-27T03:18:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb788002640 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6600346a0 0x7fb660036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb78a4e7980 1 Processor -- start 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb78a4e7980 1 -- start start 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb788002640 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6600346a0 0x7fb660036b60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x555b48eef1c0 tx=0x7fb778018c70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb78a4e7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f0fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb78a4e7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b48ee0d70 con 0x555b48f0f7f0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb788803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f0fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb788803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f0fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50234/0 (socket says 172.21.15.110:50234) 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb788803640 1 -- 172.21.15.110:0/278895688 learned_addr learned my addr 172.21.15.110:0/278895688 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb788803640 1 -- 172.21.15.110:0/278895688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48f101e0 con 0x555b48f0f7f0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb788803640 1 --2- 172.21.15.110:0/278895688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f0fc10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb7800498a0 tx=0x7fb780018930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b355377474c0ba server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb666ffd640 1 -- 172.21.15.110:0/278895688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb780047070 con 0x555b48f0f7f0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb666ffd640 1 -- 172.21.15.110:0/278895688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb780018af0 con 0x555b48f0f7f0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.729+0000 7fb666ffd640 1 -- 172.21.15.110:0/278895688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb78004b0d0 con 0x555b48f0f7f0 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- 172.21.15.110:0/278895688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 msgr2=0x555b48f0fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/278895688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f0fc10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb7800498a0 tx=0x7fb780018930 comp rx=0 tx=0).stop 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- 172.21.15.110:0/278895688 shutdown_connections 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/278895688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f0fc10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- 172.21.15.110:0/278895688 >> 172.21.15.110:0/278895688 conn(0x555b48f0bb30 msgr2=0x555b48f09b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- 172.21.15.110:0/278895688 shutdown_connections 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- 172.21.15.110:0/278895688 wait complete. 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 Processor -- start 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- start start 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f16d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b48f10940 con 0x555b48f0f7f0 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb788803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f16d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb788803640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f16d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50242/0 (socket says 172.21.15.110:50242) 2024-04-27T03:18:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 learned_addr learned my addr 172.21.15.110:0/1313338619 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48f17250 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb788803640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f16d10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb780048770 tx=0x7fb780049830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb6657fa640 1 -- 172.21.15.110:0/1313338619 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb780031070 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b48f174e0 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb6657fa640 1 -- 172.21.15.110:0/1313338619 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb780016d60 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb6657fa640 1 -- 172.21.15.110:0/1313338619 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb780020860 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b48f1a1c0 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb6657fa640 1 -- 172.21.15.110:0/1313338619 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb780034070 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.733+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb6657fa640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb650034650 0x7fb650036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb6657fa640 1 -- 172.21.15.110:0/1313338619 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb780037c90 con 0x555b48f0f7f0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb788002640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb650034650 0x7fb650036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48f22360 0x555b48f24800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f24f50 con 0x555b48f22360 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb787801640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48f22360 0x555b48f24800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb788002640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb650034650 0x7fb650036b10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb778018d90 tx=0x7fb778039000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb787801640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48f22360 0x555b48f24800 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 realm 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f25380 con 0x555b48f22360 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f25c50 con 0x555b48f22360 2024-04-27T03:18:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48f26670 0x555b48f28a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f291d0 con 0x555b48f26670 2024-04-27T03:18:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb788803640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48f26670 0x555b48f28a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb788803640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48f26670 0x555b48f28a60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f29a70 con 0x555b48f22360 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 20 searching for the correct realm 2024-04-27T03:18:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b48f2b7e0 0x555b48f2dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.737+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f2b7e0 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788002640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b48f2b7e0 0x555b48f2dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788002640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b48f2b7e0 0x555b48f2dbd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788803640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb780036070 0x7fb7800137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x7fb780036070 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb780036070 0x7fb7800137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb780036070 0x7fb7800137e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f2b7e0 2024-04-27T03:18:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.741+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x7fb780036070 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x7fb780036070 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f2b7e0 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.745+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f2b7e0 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f2b7e0 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x7fb780036070 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f22360 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x7fb780036070 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f2b7e0 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x7fb780036070 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.749+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f2b7e0 2024-04-27T03:18:05.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2e1b0 con 0x555b48f26670 2024-04-27T03:18:05.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2a3f0 con 0x555b48f22360 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.753+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2a840 con 0x555b48f26670 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2ac60 con 0x555b48f22360 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 zone default found 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 4 Realm: () 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 zonegroup default 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2eae0 con 0x7fb780036070 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f30890 con 0x555b48f2b7e0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f30d30 con 0x555b48f2b7e0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 started sync module instance, tier type = 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:05.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2fd00 con 0x555b48f26670 2024-04-27T03:18:05.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93850554269984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f2f960 con 0x555b48f26670 2024-04-27T03:18:05.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.757+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f31c60 con 0x555b48f22360 2024-04-27T03:18:05.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93850554275760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f32080 con 0x555b48f22360 2024-04-27T03:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f32ab0 con 0x555b48f26670 2024-04-27T03:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93850554269984] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93850554275760] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93850554274192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f31bd0 con 0x555b48f26670 2024-04-27T03:18:05.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.761+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f33380 con 0x7fb780036070 2024-04-27T03:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93850554281680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f31bd0 con 0x7fb780036070 2024-04-27T03:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f33e40 con 0x555b48f26670 2024-04-27T03:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93850554274192] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93850554281680] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93850554282736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f33e00 con 0x555b48f26670 2024-04-27T03:18:05.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f34230 con 0x555b48f2b7e0 2024-04-27T03:18:05.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.765+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93850554279360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f34a80 con 0x555b48f2b7e0 2024-04-27T03:18:05.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f35480 con 0x555b48f2b7e0 2024-04-27T03:18:05.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93850554282736] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93850554279360] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93850554284304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f35870 con 0x555b48f2b7e0 2024-04-27T03:18:05.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f36100 con 0x7fb780036070 2024-04-27T03:18:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93850554284304] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'18 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93850554290064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f35d20 con 0x7fb780036070 2024-04-27T03:18:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 20 add_watcher() i=0 2024-04-27T03:18:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 20 add_watcher() i=1 2024-04-27T03:18:05.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 20 add_watcher() i=2 2024-04-27T03:18:05.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 20 add_watcher() i=3 2024-04-27T03:18:05.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 20 add_watcher() i=4 2024-04-27T03:18:05.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 20 add_watcher() i=5 2024-04-27T03:18:05.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.769+0000 7fb78a4e7980 20 add_watcher() i=6 2024-04-27T03:18:05.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93850554290064] v27'19 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 20 add_watcher() i=7 2024-04-27T03:18:05.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:05.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f39580 con 0x555b48f22360 2024-04-27T03:18:05.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93850554292800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f36a30 con 0x555b48f22360 2024-04-27T03:18:05.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93850554292800] v27'52 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb63dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 5 note: GC not initialized 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb63cfe1640 20 reqs_thread_entry: start 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 20 Filter name: none 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb5b67fc640 20 reqs_thread_entry: start 2024-04-27T03:18:05.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.773+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.e 6:74cd889a:usage::usage.0:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d220 con 0x555b48f26670 2024-04-27T03:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 usage.0 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:63 6.3 6:c0e78a64:usage::usage.1:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d220 con 0x555b48f26670 2024-04-27T03:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(63 usage.1 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 6.d 6:b564efa4:usage::usage.2:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(64 usage.2 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:65 6.2 6:42d64401:usage::usage.3:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(65 usage.3 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.7 6:e09b6f01:usage::usage.4:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(66 usage.4 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.6 6:672810f1:usage::usage.5:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 usage.5 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 6.4 6:25a86931:usage::usage.6:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(68 usage.6 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.17 6:ebbe7322:usage::usage.7:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 usage.7 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.777+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 6.6 6:607e1922:usage::usage.8:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(70 usage.8 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 6.b 6:d7809252:usage::usage.9:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(71 usage.9 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(72 usage.10 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:73 6.4 6:21f845ae:usage::usage.11:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(73 usage.11 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 6.f 6:f0386354:usage::usage.12:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x7fb780036070 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(74 usage.12 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 6.c 6:3735eec4:usage::usage.13:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(75 usage.13 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 6.18 6:1f520d0a:usage::usage.14:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x7fb780036070 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(76 usage.14 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:77 6.0 6:029cee12:usage::usage.15:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(77 usage.15 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:78 6.1e 6:7f3e030e:usage::usage.16:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(78 usage.16 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 6.10 6:0fbdcb32:usage::usage.17:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(79 usage.17 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.19 6:993b03e9:usage::usage.18:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(80 usage.18 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.781+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 6.1a 6:58fcd270:usage::usage.19:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(81 usage.19 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:82 6.1f 6:fcd63d49:usage::usage.20:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(82 usage.20 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.5 6:a2c1a412:usage::usage.21:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(83 usage.21 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.13 6:cdb71d68:usage::usage.22:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(84 usage.22 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.17 6:ef3bca03:usage::usage.23:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(85 usage.23 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:86 6.e 6:7412960a:usage::usage.24:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(86 usage.24 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:87 6.7 6:e62d1024:usage::usage.25:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(87 usage.25 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:88 6.d 6:b172cd59:usage::usage.26:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(88 usage.26 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.785+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:89 6.1 6:83912023:usage::usage.27:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(89 usage.27 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:90 6.1e 6:7c73bdb3:usage::usage.28:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(90 usage.28 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:91 6.16 6:6e59fa35:usage::usage.29:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(91 usage.29 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:92 6.1c 6:38fe6885:usage::usage.30:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(92 usage.30 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:93 6.1d 6:bb28dff0:usage::usage.31:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(93 usage.31 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb63dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.789+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93850554292800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.793+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(94 data_loggenerations_metadata [watch unwatch cookie 93850554292800] v27'53 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.793+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93850554269984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.793+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(95 notify.0 [watch unwatch cookie 93850554269984] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.793+0000 7fb78a4e7980 20 remove_watcher() i=0 2024-04-27T03:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.793+0000 7fb78a4e7980 2 removed watcher, disabling cache 2024-04-27T03:18:05.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.793+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93850554275760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f22360 2024-04-27T03:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(96 notify.1 [watch unwatch cookie 93850554275760] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x555b48f22360 2024-04-27T03:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb78a4e7980 20 remove_watcher() i=1 2024-04-27T03:18:05.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93850554274192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(97 notify.2 [watch unwatch cookie 93850554274192] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb78a4e7980 20 remove_watcher() i=2 2024-04-27T03:18:05.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93850554281680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x7fb780036070 2024-04-27T03:18:05.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(98 notify.3 [watch unwatch cookie 93850554281680] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb78a4e7980 20 remove_watcher() i=3 2024-04-27T03:18:05.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.797+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93850554282736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f26670 2024-04-27T03:18:05.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.801+0000 7fb788803640 1 -- 172.21.15.110:0/1313338619 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(99 notify.4 [watch unwatch cookie 93850554282736] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb780033030 con 0x555b48f26670 2024-04-27T03:18:05.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.801+0000 7fb78a4e7980 20 remove_watcher() i=4 2024-04-27T03:18:05.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.801+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93850554279360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.801+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(100 notify.5 [watch unwatch cookie 93850554279360] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.801+0000 7fb78a4e7980 20 remove_watcher() i=5 2024-04-27T03:18:05.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.801+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93850554284304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x555b48f2b7e0 2024-04-27T03:18:05.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb788002640 1 -- 172.21.15.110:0/1313338619 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(101 notify.6 [watch unwatch cookie 93850554284304] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb778050040 con 0x555b48f2b7e0 2024-04-27T03:18:05.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 20 remove_watcher() i=6 2024-04-27T03:18:05.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93850554290064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b48f5d1e0 con 0x7fb780036070 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(102 notify.7 [watch unwatch cookie 93850554290064] v27'20 uv18 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb77c0aaa40 con 0x7fb780036070 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 20 remove_watcher() i=7 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48f26670 msgr2=0x555b48f28a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48f26670 0x555b48f28a60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48f22360 msgr2=0x555b48f24800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48f22360 0x555b48f24800 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb780036070 msgr2=0x7fb7800137e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb780036070 0x7fb7800137e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b48f2b7e0 msgr2=0x555b48f2dbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b48f2b7e0 0x555b48f2dbd0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb650034650 msgr2=0x7fb650036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb650034650 0x7fb650036b10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb778018d90 tx=0x7fb778039000 comp rx=0 tx=0).stop 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 msgr2=0x555b48f16d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f16d10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb780048770 tx=0x7fb780049830 comp rx=0 tx=0).stop 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb787801640 1 -- 172.21.15.110:0/1313338619 reap_dead start 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 shutdown_connections 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/1313338619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f0f7f0 0x555b48f16d10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 >> 172.21.15.110:0/1313338619 conn(0x555b48f0bb30 msgr2=0x555b48f09190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 shutdown_connections 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/1313338619 wait complete. 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6600346a0 msgr2=0x7fb660036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6600346a0 0x7fb660036b60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x555b48eef1c0 tx=0x7fb778018c70 comp rx=0 tx=0).stop 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f025f0 msgr2=0x555b48f02a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f025f0 0x555b48f02a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb78000aab0 tx=0x7fb78000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:05.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 shutdown_connections 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6600346a0 0x7fb660036b60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/2155471086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48f025f0 0x555b48f02a10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 >> 172.21.15.110:0/2155471086 conn(0x555b48ef7400 msgr2=0x555b48ef7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 shutdown_connections 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/2155471086 wait complete. 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48eefa00 msgr2=0x555b48ef1df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48eefa00 0x555b48ef1df0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48df99b0 msgr2=0x555b48eee5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48df99b0 0x555b48eee5f0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6700346a0 msgr2=0x7fb670036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6700346a0 0x7fb670036b60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0ae000 tx=0x7fb77c0a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 msgr2=0x555b48ee76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee76c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb77800f610 tx=0x7fb778000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 shutdown_connections 2024-04-27T03:18:05.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48eefa00 0x555b48ef1df0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48df99b0 0x555b48eee5f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb6700346a0 0x7fb670036b60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 --2- 172.21.15.110:0/3025675214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48df54a0 0x555b48ee76c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 >> 172.21.15.110:0/3025675214 conn(0x555b48d68b80 msgr2=0x555b48edacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 shutdown_connections 2024-04-27T03:18:05.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.805+0000 7fb78a4e7980 1 -- 172.21.15.110:0/3025675214 wait complete. 2024-04-27T03:18:05.825 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[],"summary":[]} 2024-04-27T03:18:05.825 DEBUG:tasks.util.rgw: json result: {'entries': [], 'summary': []} 2024-04-27T03:18:05.826 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:05.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:05.826 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858e57c980 1 Processor -- start 2024-04-27T03:18:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858e57c980 1 -- start start 2024-04-27T03:18:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858e57c980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56055edc3790 0x56055edc5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858e57c980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x56055ec7e0d0 con 0x56055edbf2d0 2024-04-27T03:18:05.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858e57c980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x56055ec73430 con 0x56055edc3790 2024-04-27T03:18:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858b896640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56055edc3790 0x56055edc5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858b896640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56055edc3790 0x56055edc5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50248/0 (socket says 172.21.15.110:50248) 2024-04-27T03:18:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858b896640 1 -- 172.21.15.110:0/2550116379 learned_addr learned my addr 172.21.15.110:0/2550116379 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858c097640 1 --1- 172.21.15.110:0/2550116379 >> v1:172.21.15.77:6789/0 conn(0x56055edbf2d0 0x56055edbf6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34572/0 (socket says 172.21.15.110:34572) 2024-04-27T03:18:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858b896640 1 -- 172.21.15.110:0/2550116379 >> v1:172.21.15.77:6789/0 conn(0x56055edbf2d0 legacy=0x56055edbf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.897+0000 7f858b896640 1 -- 172.21.15.110:0/2550116379 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055ec6c510 con 0x56055edc3790 2024-04-27T03:18:05.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858b896640 1 --2- 172.21.15.110:0/2550116379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edc3790 0x56055edc5b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f858000a5c0 tx=0x7f858001b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca4ca009b014653 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858b095640 1 -- 172.21.15.110:0/2550116379 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85800a6070 con 0x56055edc3790 2024-04-27T03:18:05.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858b095640 1 -- 172.21.15.110:0/2550116379 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85800af070 con 0x56055edc3790 2024-04-27T03:18:05.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858b095640 1 -- 172.21.15.110:0/2550116379 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85800aa5c0 con 0x56055edc3790 2024-04-27T03:18:05.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858e57c980 1 -- 172.21.15.110:0/2550116379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edc3790 msgr2=0x56055edc5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858e57c980 1 --2- 172.21.15.110:0/2550116379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edc3790 0x56055edc5b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f858000a5c0 tx=0x7f858001b340 comp rx=0 tx=0).stop 2024-04-27T03:18:05.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858e57c980 1 -- 172.21.15.110:0/2550116379 shutdown_connections 2024-04-27T03:18:05.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858e57c980 1 --2- 172.21.15.110:0/2550116379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edc3790 0x56055edc5b80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858e57c980 1 -- 172.21.15.110:0/2550116379 >> 172.21.15.110:0/2550116379 conn(0x56055edbaba0 msgr2=0x56055edbcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858e57c980 1 -- 172.21.15.110:0/2550116379 shutdown_connections 2024-04-27T03:18:05.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.901+0000 7f858e57c980 1 -- 172.21.15.110:0/2550116379 wait complete. 2024-04-27T03:18:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858e57c980 1 Processor -- start 2024-04-27T03:18:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858e57c980 1 -- start start 2024-04-27T03:18:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858e57c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeaa420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858e57c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56055ecc22d0 con 0x56055edbf2d0 2024-04-27T03:18:05.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858c097640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeaa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858c097640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeaa420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50256/0 (socket says 172.21.15.110:50256) 2024-04-27T03:18:05.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858c097640 1 -- 172.21.15.110:0/1932497369 learned_addr learned my addr 172.21.15.110:0/1932497369 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858c097640 1 -- 172.21.15.110:0/1932497369 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055ec48ae0 con 0x56055edbf2d0 2024-04-27T03:18:05.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f858c097640 1 --2- 172.21.15.110:0/1932497369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeaa420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f857c001e50 tx=0x7f857c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23360a4ad51d736 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f847a60c640 1 -- 172.21.15.110:0/1932497369 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f857c00e5d0 con 0x56055edbf2d0 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f847a60c640 1 -- 172.21.15.110:0/1932497369 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857c00e770 con 0x56055edbf2d0 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.917+0000 7f847a60c640 1 -- 172.21.15.110:0/1932497369 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f857c00eab0 con 0x56055edbf2d0 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- 172.21.15.110:0/1932497369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 msgr2=0x56055eeaa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 --2- 172.21.15.110:0/1932497369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeaa420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f857c001e50 tx=0x7f857c001e80 comp rx=0 tx=0).stop 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- 172.21.15.110:0/1932497369 shutdown_connections 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 --2- 172.21.15.110:0/1932497369 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeaa420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- 172.21.15.110:0/1932497369 >> 172.21.15.110:0/1932497369 conn(0x56055edbaba0 msgr2=0x56055eea5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- 172.21.15.110:0/1932497369 shutdown_connections 2024-04-27T03:18:05.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- 172.21.15.110:0/1932497369 wait complete. 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 Processor -- start 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- start start 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeb1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56055edc8bb0 con 0x56055edbf2d0 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858c097640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeb1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858c097640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeb1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50270/0 (socket says 172.21.15.110:50270) 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858c097640 1 -- 172.21.15.110:0/1836231163 learned_addr learned my addr 172.21.15.110:0/1836231163 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858c097640 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055eeb19a0 con 0x56055edbf2d0 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858c097640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeb1460 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x56055eeab2b0 tx=0x7f857c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f8478e09640 1 -- 172.21.15.110:0/1836231163 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f857c001260 con 0x56055edbf2d0 2024-04-27T03:18:05.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f8478e09640 1 -- 172.21.15.110:0/1836231163 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857c00fa40 con 0x56055edbf2d0 2024-04-27T03:18:05.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56055eeb1c30 con 0x56055edbf2d0 2024-04-27T03:18:05.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f8478e09640 1 -- 172.21.15.110:0/1836231163 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f857c00fd80 con 0x56055edbf2d0 2024-04-27T03:18:05.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.921+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055eeb48b0 con 0x56055edbf2d0 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f8478e09640 1 -- 172.21.15.110:0/1836231163 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f857c017070 con 0x56055edbf2d0 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f8478e09640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8474034650 0x7f8474036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f8478e09640 1 -- 172.21.15.110:0/1836231163 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f857c015020 con 0x56055edbf2d0 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858b896640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8474034650 0x7f8474036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055edc3790 0x56055eeb83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eeb8a20 con 0x56055edc3790 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858c898640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055edc3790 0x56055eeb83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858b896640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8474034650 0x7f8474036b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f85800ae000 tx=0x7f85800a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858c898640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055edc3790 0x56055eeb83b0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858c898640 1 -- 172.21.15.110:0/1836231163 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f858400a4d0 con 0x56055edc3790 2024-04-27T03:18:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eeb97f0 0x56055eebbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eebc220 con 0x56055eeb97f0 2024-04-27T03:18:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858c097640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eeb97f0 0x56055eebbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858c097640 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eeb97f0 0x56055eebbbe0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858c097640 1 -- 172.21.15.110:0/1836231163 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f857c02e070 con 0x56055eeb97f0 2024-04-27T03:18:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eebdad0 con 0x56055edc3790 2024-04-27T03:18:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858c898640 1 -- 172.21.15.110:0/1836231163 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f858400a4d0 con 0x56055edc3790 2024-04-27T03:18:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.925+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eebcd00 con 0x56055eeb97f0 2024-04-27T03:18:05.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858c097640 1 -- 172.21.15.110:0/1836231163 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f857c02e070 con 0x56055eeb97f0 2024-04-27T03:18:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eebd0f0 con 0x56055edc3790 2024-04-27T03:18:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858c898640 1 -- 172.21.15.110:0/1836231163 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f858400a4d0 con 0x56055edc3790 2024-04-27T03:18:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858e57c980 1 Processor -- start 2024-04-27T03:18:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858e57c980 1 -- start start 2024-04-27T03:18:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858e57c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055eecc3d0 0x56055eecc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858e57c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56055ecc6310 con 0x56055eecc3d0 2024-04-27T03:18:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858c898640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055eecc3d0 0x56055eecc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858c898640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055eecc3d0 0x56055eecc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50284/0 (socket says 172.21.15.110:50284) 2024-04-27T03:18:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858c898640 1 -- 172.21.15.110:0/4219163412 learned_addr learned my addr 172.21.15.110:0/4219163412 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858c898640 1 -- 172.21.15.110:0/4219163412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055eeccd30 con 0x56055eecc3d0 2024-04-27T03:18:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858c898640 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055eecc3d0 0x56055eecc7f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f858400dfc0 tx=0x7f85840178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f8471ffb640 1 -- 172.21.15.110:0/4219163412 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85840157d0 con 0x56055eecc3d0 2024-04-27T03:18:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f8471ffb640 1 -- 172.21.15.110:0/4219163412 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8584015970 con 0x56055eecc3d0 2024-04-27T03:18:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f8471ffb640 1 -- 172.21.15.110:0/4219163412 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8584015c70 con 0x56055eecc3d0 2024-04-27T03:18:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56055eecd7d0 con 0x56055eecc3d0 2024-04-27T03:18:05.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.929+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055eecdca0 con 0x56055eecc3d0 2024-04-27T03:18:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f8471ffb640 1 -- 172.21.15.110:0/4219163412 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8584029440 con 0x56055eecc3d0 2024-04-27T03:18:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f8471ffb640 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8464034760 0x7f8464036c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f8471ffb640 1 -- 172.21.15.110:0/4219163412 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f858400e040 con 0x56055eecc3d0 2024-04-27T03:18:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c097640 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8464034760 0x7f8464036c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 Processor -- start 2024-04-27T03:18:05.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- start start 2024-04-27T03:18:05.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055ed35070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56055eeaab70 con 0x56055ed34c70 2024-04-27T03:18:05.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c097640 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8464034760 0x7f8464036c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x56055eebc820 tx=0x7f857c001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055ed35070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055ed35070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50292/0 (socket says 172.21.15.110:50292) 2024-04-27T03:18:05.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 -- 172.21.15.110:0/2781709520 learned_addr learned my addr 172.21.15.110:0/2781709520 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 -- 172.21.15.110:0/2781709520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055ed334c0 con 0x56055ed34c70 2024-04-27T03:18:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 --2- 172.21.15.110:0/2781709520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055ed35070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f858404ab40 tx=0x7f858401e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264e983dcaa024de server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f84637fe640 1 -- 172.21.15.110:0/2781709520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8584012440 con 0x56055ed34c70 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f84637fe640 1 -- 172.21.15.110:0/2781709520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85840283a0 con 0x56055ed34c70 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f84637fe640 1 -- 172.21.15.110:0/2781709520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85840125e0 con 0x56055ed34c70 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- 172.21.15.110:0/2781709520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 msgr2=0x56055ed35070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 --2- 172.21.15.110:0/2781709520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055ed35070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f858404ab40 tx=0x7f858401e8b0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- 172.21.15.110:0/2781709520 shutdown_connections 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 --2- 172.21.15.110:0/2781709520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055ed35070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- 172.21.15.110:0/2781709520 >> 172.21.15.110:0/2781709520 conn(0x56055eed58c0 msgr2=0x56055eed3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- 172.21.15.110:0/2781709520 shutdown_connections 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- 172.21.15.110:0/2781709520 wait complete. 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 Processor -- start 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- start start 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055eee2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858e57c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56055ed33990 con 0x56055ed34c70 2024-04-27T03:18:05.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055eee2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055eee2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50304/0 (socket says 172.21.15.110:50304) 2024-04-27T03:18:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.933+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 learned_addr learned my addr 172.21.15.110:0/3867473484 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055eee31d0 con 0x56055ed34c70 2024-04-27T03:18:05.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858c898640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055eee2c90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f858401eb20 tx=0x7f858404ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f8461ffb640 1 -- 172.21.15.110:0/3867473484 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8584013c60 con 0x56055ed34c70 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f8461ffb640 1 -- 172.21.15.110:0/3867473484 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8584038070 con 0x56055ed34c70 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56055eee3460 con 0x56055ed34c70 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f8461ffb640 1 -- 172.21.15.110:0/3867473484 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85840491d0 con 0x56055ed34c70 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055eee6140 con 0x56055ed34c70 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f8461ffb640 1 -- 172.21.15.110:0/3867473484 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8584032070 con 0x56055ed34c70 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f8461ffb640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8454034650 0x7f8454036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f8461ffb640 1 -- 172.21.15.110:0/3867473484 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f858404b6e0 con 0x56055ed34c70 2024-04-27T03:18:05.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858c097640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8454034650 0x7f8454036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055eeee2e0 0x56055eef0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef0ed0 con 0x56055eeee2e0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858b896640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055eeee2e0 0x56055eef0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858c097640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8454034650 0x7f8454036b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f857c024000 tx=0x7f857c03c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858b896640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055eeee2e0 0x56055eef0780 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 20 realm 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef1300 con 0x56055eeee2e0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:05.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.937+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef1bd0 con 0x56055eeee2e0 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eef25f0 0x56055eef49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef5150 con 0x56055eef25f0 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858c898640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eef25f0 0x56055eef49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858c898640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eef25f0 0x56055eef49e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef59f0 con 0x56055eeee2e0 2024-04-27T03:18:05.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 20 searching for the correct realm 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56055eef7760 0x56055eef9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef7760 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858c097640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56055eef7760 0x56055eef9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858c097640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56055eef7760 0x56055eef9b50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.941+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c898640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8584057040 0x7f858403f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x7f8584057040 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8584057040 0x7f858403f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:05.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8584057040 0x7f858403f870 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef7760 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.945+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x7f8584057040 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x7f8584057040 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef7760 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef7760 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.949+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef7760 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x7f8584057040 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eeee2e0 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x7f8584057040 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef7760 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x7f8584057040 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef7760 2024-04-27T03:18:05.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.953+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa0e0 con 0x56055eef25f0 2024-04-27T03:18:05.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:05.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:05.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:05.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:05.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef6340 con 0x56055eeee2e0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef67c0 con 0x56055eef25f0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eef6c20 con 0x56055eeee2e0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:05.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 zone default found 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 4 Realm: () 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 zonegroup default 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eefa9f0 con 0x7f8584057040 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eefc7a0 con 0x56055eef7760 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055eefcc60 con 0x56055eef7760 2024-04-27T03:18:05.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 started sync module instance, tier type = 2024-04-27T03:18:05.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:05.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.957+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eefbc10 con 0x56055eef25f0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.961+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.961+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94581067595824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eefd570 con 0x56055eef25f0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.961+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eefddc0 con 0x56055eeee2e0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.961+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.961+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94581067601264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eefdf40 con 0x56055eeee2e0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.961+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eefea60 con 0x56055eef25f0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.961+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94581067595824] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94581067601264] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94581067605296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eefea20 con 0x56055eef25f0 2024-04-27T03:18:05.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eeff300 con 0x7f8584057040 2024-04-27T03:18:05.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94581067607632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eefea20 con 0x7f8584057040 2024-04-27T03:18:05.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eeffd90 con 0x56055eef25f0 2024-04-27T03:18:05.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94581067607632] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.965+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94581067605296] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94581067600064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055eeffd90 con 0x56055eef25f0 2024-04-27T03:18:05.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef00180 con 0x56055eef7760 2024-04-27T03:18:05.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94581067608688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef009d0 con 0x56055eef7760 2024-04-27T03:18:05.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef013d0 con 0x56055eef7760 2024-04-27T03:18:05.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94581067600064] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94581067608688] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94581067613472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef00ee0 con 0x56055eef7760 2024-04-27T03:18:05.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.969+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef01de0 con 0x7f8584057040 2024-04-27T03:18:05.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94581067613472] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'21 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94581067610304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef01de0 con 0x7f8584057040 2024-04-27T03:18:05.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=0 2024-04-27T03:18:05.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=1 2024-04-27T03:18:05.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=2 2024-04-27T03:18:05.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=3 2024-04-27T03:18:05.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=4 2024-04-27T03:18:05.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=5 2024-04-27T03:18:05.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=6 2024-04-27T03:18:05.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94581067610304] v27'22 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 20 add_watcher() i=7 2024-04-27T03:18:05.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:05.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055ef05030 con 0x56055eeee2e0 2024-04-27T03:18:05.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94581067618976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef02370 con 0x56055eeee2e0 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.973+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94581067618976] v27'54 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f84427e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 5 note: GC not initialized 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f84417e2640 20 reqs_thread_entry: start 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:05.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 20 Filter name: none 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f83b2ffd640 20 reqs_thread_entry: start 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56055ef283d0 con 0x56055eeee2e0 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:could not fetch user info: no user info saved 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f84427e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:05.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.977+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94581067618976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x56055eeee2e0 2024-04-27T03:18:05.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94581067618976] v27'55 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94581067595824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x56055eef25f0 2024-04-27T03:18:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94581067595824] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858e57c980 20 remove_watcher() i=0 2024-04-27T03:18:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858e57c980 2 removed watcher, disabling cache 2024-04-27T03:18:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94581067601264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x56055eeee2e0 2024-04-27T03:18:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94581067601264] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x56055eeee2e0 2024-04-27T03:18:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858e57c980 20 remove_watcher() i=1 2024-04-27T03:18:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.981+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94581067605296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x56055eef25f0 2024-04-27T03:18:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.985+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94581067605296] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.985+0000 7f858e57c980 20 remove_watcher() i=2 2024-04-27T03:18:05.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.985+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94581067607632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x7f8584057040 2024-04-27T03:18:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.985+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94581067607632] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.985+0000 7f858e57c980 20 remove_watcher() i=3 2024-04-27T03:18:05.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.985+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94581067600064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x56055eef25f0 2024-04-27T03:18:05.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858c898640 1 -- 172.21.15.110:0/3867473484 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94581067600064] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8584026020 con 0x56055eef25f0 2024-04-27T03:18:05.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858e57c980 20 remove_watcher() i=4 2024-04-27T03:18:05.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94581067608688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x56055eef7760 2024-04-27T03:18:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94581067608688] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858e57c980 20 remove_watcher() i=5 2024-04-27T03:18:05.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94581067613472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x56055eef7760 2024-04-27T03:18:05.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858c097640 1 -- 172.21.15.110:0/3867473484 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94581067613472] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f857c02e070 con 0x56055eef7760 2024-04-27T03:18:05.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858e57c980 20 remove_watcher() i=6 2024-04-27T03:18:05.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.989+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94581067610304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56055ef28390 con 0x7f8584057040 2024-04-27T03:18:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94581067610304] v27'23 uv21 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f85800aa9c0 con 0x7f8584057040 2024-04-27T03:18:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 20 remove_watcher() i=7 2024-04-27T03:18:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eef25f0 msgr2=0x56055eef49e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eef25f0 0x56055eef49e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055eeee2e0 msgr2=0x56055eef0780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055eeee2e0 0x56055eef0780 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8584057040 msgr2=0x7f858403f870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8584057040 0x7f858403f870 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56055eef7760 msgr2=0x56055eef9b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56055eef7760 0x56055eef9b50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8454034650 msgr2=0x7f8454036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8454034650 0x7f8454036b10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f857c024000 tx=0x7f857c03c000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858b896640 1 -- 172.21.15.110:0/3867473484 reap_dead start 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 msgr2=0x56055eee2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055eee2c90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f858401eb20 tx=0x7f858404ab70 comp rx=0 tx=0).stop 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 shutdown_connections 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/3867473484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055ed34c70 0x56055eee2c90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 >> 172.21.15.110:0/3867473484 conn(0x56055eed58c0 msgr2=0x56055eed2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 shutdown_connections 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/3867473484 wait complete. 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8464034760 msgr2=0x7f8464036c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8464034760 0x7f8464036c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x56055eebc820 tx=0x7f857c001400 comp rx=0 tx=0).stop 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055eecc3d0 msgr2=0x56055eecc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055eecc3d0 0x56055eecc7f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f858400dfc0 tx=0x7f85840178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 shutdown_connections 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8464034760 0x7f8464036c20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/4219163412 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055eecc3d0 0x56055eecc7f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 >> 172.21.15.110:0/4219163412 conn(0x56055eec11e0 msgr2=0x56055eec1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 shutdown_connections 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/4219163412 wait complete. 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eeb97f0 msgr2=0x56055eebbbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eeb97f0 0x56055eebbbe0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055edc3790 msgr2=0x56055eeb83b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055edc3790 0x56055eeb83b0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8474034650 msgr2=0x7f8474036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8474034650 0x7f8474036b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f85800ae000 tx=0x7f85800a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 msgr2=0x56055eeb1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeb1460 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x56055eeab2b0 tx=0x7f857c000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 shutdown_connections 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56055eeb97f0 0x56055eebbbe0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56055edc3790 0x56055eeb83b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8474034650 0x7f8474036b10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 --2- 172.21.15.110:0/1836231163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56055edbf2d0 0x56055eeb1460 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 >> 172.21.15.110:0/1836231163 conn(0x56055edbaba0 msgr2=0x56055eea4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 shutdown_connections 2024-04-27T03:18:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:05.993+0000 7f858e57c980 1 -- 172.21.15.110:0/1836231163 wait complete. 2024-04-27T03:18:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-27T03:18:06.013 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'foo', '--display-name', 'Foo', '--email', 'foo@foo.com', '--access-key', '9te6NH5mcdcq0Tc5i8i1', '--secret', 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', '--max-buckets', '4'] 2024-04-27T03:18:06.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'foo', '--display-name', 'Foo', '--email', 'foo@foo.com', '--access-key', '9te6NH5mcdcq0Tc5i8i1', '--secret', 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', '--max-buckets', '4'] 2024-04-27T03:18:06.013 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid foo --display-name Foo --email foo@foo.com --access-key 9te6NH5mcdcq0Tc5i8i1 --secret Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu --max-buckets 4 2024-04-27T03:18:06.102 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:06.102 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:06.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c44b3980 1 Processor -- start 2024-04-27T03:18:06.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c44b3980 1 -- start start 2024-04-27T03:18:06.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c44b3980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a373d009c0 0x55a373d02db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c44b3980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a373bb90d0 con 0x55a373c6e830 2024-04-27T03:18:06.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c44b3980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a373bae430 con 0x55a373d009c0 2024-04-27T03:18:06.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c17cd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a373d009c0 0x55a373d02db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c17cd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a373d009c0 0x55a373d02db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50306/0 (socket says 172.21.15.110:50306) 2024-04-27T03:18:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c17cd640 1 -- 172.21.15.110:0/3142099692 learned_addr learned my addr 172.21.15.110:0/3142099692 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c1fce640 1 --1- 172.21.15.110:0/3142099692 >> v1:172.21.15.77:6789/0 conn(0x55a373c6e830 0x55a373cff2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34580/0 (socket says 172.21.15.110:34580) 2024-04-27T03:18:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c17cd640 1 -- 172.21.15.110:0/3142099692 >> v1:172.21.15.77:6789/0 conn(0x55a373c6e830 legacy=0x55a373cff2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.097+0000 7ff1c17cd640 1 -- 172.21.15.110:0/3142099692 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a373ba7510 con 0x55a373d009c0 2024-04-27T03:18:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c17cd640 1 --2- 172.21.15.110:0/3142099692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373d009c0 0x55a373d02db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800a5c0 tx=0x7ff1b801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9443442dc757a2ec server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c0fcc640 1 -- 172.21.15.110:0/3142099692 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1b80a6070 con 0x55a373d009c0 2024-04-27T03:18:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c0fcc640 1 -- 172.21.15.110:0/3142099692 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b80af070 con 0x55a373d009c0 2024-04-27T03:18:06.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c0fcc640 1 -- 172.21.15.110:0/3142099692 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1b80aa5c0 con 0x55a373d009c0 2024-04-27T03:18:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3142099692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373d009c0 msgr2=0x55a373d02db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/3142099692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373d009c0 0x55a373d02db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800a5c0 tx=0x7ff1b801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3142099692 shutdown_connections 2024-04-27T03:18:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/3142099692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373d009c0 0x55a373d02db0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3142099692 >> 172.21.15.110:0/3142099692 conn(0x55a373cf49c0 msgr2=0x55a373cf6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3142099692 shutdown_connections 2024-04-27T03:18:06.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.101+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3142099692 wait complete. 2024-04-27T03:18:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 Processor -- start 2024-04-27T03:18:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- start start 2024-04-27T03:18:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373de7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a373bfd2d0 con 0x55a373c6e830 2024-04-27T03:18:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373de7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373de7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50314/0 (socket says 172.21.15.110:50314) 2024-04-27T03:18:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 -- 172.21.15.110:0/3931862436 learned_addr learned my addr 172.21.15.110:0/3931862436 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 -- 172.21.15.110:0/3931862436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a373b83ae0 con 0x55a373c6e830 2024-04-27T03:18:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/3931862436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373de7810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4001e50 tx=0x7ff1b4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef38f4244b690792 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff0afe07640 1 -- 172.21.15.110:0/3931862436 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1b400e5d0 con 0x55a373c6e830 2024-04-27T03:18:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff0afe07640 1 -- 172.21.15.110:0/3931862436 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b400e770 con 0x55a373c6e830 2024-04-27T03:18:06.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff0afe07640 1 -- 172.21.15.110:0/3931862436 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1b400eab0 con 0x55a373c6e830 2024-04-27T03:18:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3931862436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 msgr2=0x55a373de7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/3931862436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373de7810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4001e50 tx=0x7ff1b4001e80 comp rx=0 tx=0).stop 2024-04-27T03:18:06.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3931862436 shutdown_connections 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/3931862436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373de7810 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3931862436 >> 172.21.15.110:0/3931862436 conn(0x55a373cf49c0 msgr2=0x55a373de3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3931862436 shutdown_connections 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3931862436 wait complete. 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 Processor -- start 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- start start 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373dee850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c44b3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a373d05db0 con 0x55a373c6e830 2024-04-27T03:18:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373dee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373dee850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50320/0 (socket says 172.21.15.110:50320) 2024-04-27T03:18:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.117+0000 7ff1c1fce640 1 -- 172.21.15.110:0/1102759164 learned_addr learned my addr 172.21.15.110:0/1102759164 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c1fce640 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a373deed90 con 0x55a373c6e830 2024-04-27T03:18:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373dee850 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55a373de86a0 tx=0x7ff1b4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff0ae604640 1 -- 172.21.15.110:0/1102759164 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1b4001260 con 0x55a373c6e830 2024-04-27T03:18:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff0ae604640 1 -- 172.21.15.110:0/1102759164 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b400fa40 con 0x55a373c6e830 2024-04-27T03:18:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a373def020 con 0x55a373c6e830 2024-04-27T03:18:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff0ae604640 1 -- 172.21.15.110:0/1102759164 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1b400fd80 con 0x55a373c6e830 2024-04-27T03:18:06.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a373df1ca0 con 0x55a373c6e830 2024-04-27T03:18:06.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff0ae604640 1 -- 172.21.15.110:0/1102759164 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff1b4017070 con 0x55a373c6e830 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff0ae604640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff0a8034650 0x7ff0a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff0ae604640 1 -- 172.21.15.110:0/1102759164 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7ff1b4015020 con 0x55a373c6e830 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c17cd640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff0a8034650 0x7ff0a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373df3770 0x55a373df5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373df6160 con 0x55a373df3770 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373df3770 0x55a373df5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c17cd640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff0a8034650 0x7ff0a8036b10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80ae000 tx=0x7ff1b80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373df3770 0x55a373df5b60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c27cf640 1 -- 172.21.15.110:0/1102759164 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff1bc00a4d0 con 0x55a373df3770 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373df6f30 0x55a373df9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373df9960 con 0x55a373df6f30 2024-04-27T03:18:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.121+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373df6f30 0x55a373df9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373df6f30 0x55a373df9320 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c1fce640 1 -- 172.21.15.110:0/1102759164 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373df6f30 2024-04-27T03:18:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373dfb210 con 0x55a373df3770 2024-04-27T03:18:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c27cf640 1 -- 172.21.15.110:0/1102759164 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7ff1bc00a4d0 con 0x55a373df3770 2024-04-27T03:18:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373dfa440 con 0x55a373df6f30 2024-04-27T03:18:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c1fce640 1 -- 172.21.15.110:0/1102759164 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373df6f30 2024-04-27T03:18:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373dfa830 con 0x55a373df3770 2024-04-27T03:18:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c27cf640 1 -- 172.21.15.110:0/1102759164 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7ff1bc00a4d0 con 0x55a373df3770 2024-04-27T03:18:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c44b3980 1 Processor -- start 2024-04-27T03:18:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c44b3980 1 -- start start 2024-04-27T03:18:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c44b3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e09b10 0x55a373e09f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c44b3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a373c01310 con 0x55a373e09b10 2024-04-27T03:18:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c27cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e09b10 0x55a373e09f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c27cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e09b10 0x55a373e09f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50324/0 (socket says 172.21.15.110:50324) 2024-04-27T03:18:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c27cf640 1 -- 172.21.15.110:0/179670632 learned_addr learned my addr 172.21.15.110:0/179670632 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.125+0000 7ff1c27cf640 1 -- 172.21.15.110:0/179670632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a373e0a470 con 0x55a373e09b10 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e09b10 0x55a373e09f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00dfc0 tx=0x7ff1bc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a77fe640 1 -- 172.21.15.110:0/179670632 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc0157d0 con 0x55a373e09b10 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a77fe640 1 -- 172.21.15.110:0/179670632 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc015970 con 0x55a373e09b10 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a373e0af10 con 0x55a373e09b10 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a77fe640 1 -- 172.21.15.110:0/179670632 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc015c70 con 0x55a373e09b10 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a373e0b3e0 con 0x55a373e09b10 2024-04-27T03:18:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a77fe640 1 -- 172.21.15.110:0/179670632 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff1bc029440 con 0x55a373e09b10 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a77fe640 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff09c034710 0x7ff09c036bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a77fe640 1 -- 172.21.15.110:0/179670632 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7ff1bc00e040 con 0x55a373e09b10 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff09c034710 0x7ff09c036bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c44b3980 1 Processor -- start 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c44b3980 1 -- start start 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff09c034710 0x7ff09c036bd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55a373df9f60 tx=0x7ff1b4001400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c44b3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e17080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c44b3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a373de7f60 con 0x55a373e16c60 2024-04-27T03:18:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c27cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e17080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c27cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e17080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50326/0 (socket says 172.21.15.110:50326) 2024-04-27T03:18:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c27cf640 1 -- 172.21.15.110:0/3134968257 learned_addr learned my addr 172.21.15.110:0/3134968257 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c27cf640 1 -- 172.21.15.110:0/3134968257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a373e17650 con 0x55a373e16c60 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/3134968257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e17080 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc015fa0 tx=0x7ff1bc015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81693fde441d6075 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a57fa640 1 -- 172.21.15.110:0/3134968257 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc0113a0 con 0x55a373e16c60 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a57fa640 1 -- 172.21.15.110:0/3134968257 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc011540 con 0x55a373e16c60 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.129+0000 7ff0a57fa640 1 -- 172.21.15.110:0/3134968257 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc0283a0 con 0x55a373e16c60 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3134968257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 msgr2=0x55a373e17080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/3134968257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e17080 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc015fa0 tx=0x7ff1bc015fd0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3134968257 shutdown_connections 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/3134968257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e17080 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3134968257 >> 172.21.15.110:0/3134968257 conn(0x55a373e12ff0 msgr2=0x55a373e11050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3134968257 shutdown_connections 2024-04-27T03:18:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- 172.21.15.110:0/3134968257 wait complete. 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 Processor -- start 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- start start 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e1e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a373e17da0 con 0x55a373e16c60 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c27cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e1e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c27cf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e1e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50338/0 (socket says 172.21.15.110:50338) 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 learned_addr learned my addr 172.21.15.110:0/4138979828 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a373e1e610 con 0x55a373e16c60 2024-04-27T03:18:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e1e0d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc01eb40 tx=0x7ff1bc04a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff0937fe640 1 -- 172.21.15.110:0/4138979828 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc0113a0 con 0x55a373e16c60 2024-04-27T03:18:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a373e1e8a0 con 0x55a373e16c60 2024-04-27T03:18:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff0937fe640 1 -- 172.21.15.110:0/4138979828 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc013a50 con 0x55a373e16c60 2024-04-27T03:18:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff0937fe640 1 -- 172.21.15.110:0/4138979828 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1bc013c40 con 0x55a373e16c60 2024-04-27T03:18:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a373e21580 con 0x55a373e16c60 2024-04-27T03:18:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff0937fe640 1 -- 172.21.15.110:0/4138979828 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff1bc0493c0 con 0x55a373e16c60 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff0937fe640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff08c034650 0x7ff08c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff0937fe640 1 -- 172.21.15.110:0/4138979828 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7ff1bc002390 con 0x55a373e16c60 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.133+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff08c034650 0x7ff08c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373e297d0 0x55a373e2bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e2c3e0 con 0x55a373e297d0 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff08c034650 0x7ff08c036b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4024000 tx=0x7ff1b403c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c17cd640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373e297d0 0x55a373e2bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c17cd640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373e297d0 0x55a373e2bc90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 realm 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e2c810 con 0x55a373e297d0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e2d0e0 con 0x55a373e297d0 2024-04-27T03:18:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373e2db00 0x55a373e2ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e306a0 con 0x55a373e2db00 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373e2db00 0x55a373e2ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373e2db00 0x55a373e2ff30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.137+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e30ef0 con 0x55a373e297d0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c44b3980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c44b3980 20 searching for the correct realm 2024-04-27T03:18:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a373e32c60 0x55a373e35050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e32c60 2024-04-27T03:18:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a373e32c60 0x55a373e35050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c1fce640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a373e32c60 0x55a373e35050 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c27cf640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff1bc014030 0x7ff1bc039300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x7ff1bc014030 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff1bc014030 0x7ff1bc039300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff1bc014030 0x7ff1bc039300 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e32c60 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.141+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x7ff1bc014030 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x7ff1bc014030 2024-04-27T03:18:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e32c60 2024-04-27T03:18:06.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.145+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e32c60 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e32c60 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x7ff1bc014030 2024-04-27T03:18:06.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e297d0 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x7ff1bc014030 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e32c60 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x7ff1bc014030 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.149+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e32c60 2024-04-27T03:18:06.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35660 con 0x55a373e2db00 2024-04-27T03:18:06.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e31870 con 0x55a373e297d0 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.153+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e32110 con 0x55a373e2db00 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e314b0 con 0x55a373e297d0 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 zone default found 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 4 Realm: () 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 zonegroup default 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e35f90 con 0x7ff1bc014030 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e38210 con 0x55a373e32c60 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e385b0 con 0x55a373e32c60 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 started sync module instance, tier type = 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:06.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e376a0 con 0x55a373e2db00 2024-04-27T03:18:06.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94160512318144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e37300 con 0x55a373e2db00 2024-04-27T03:18:06.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.157+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e39330 con 0x55a373e297d0 2024-04-27T03:18:06.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94160512323072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e392f0 con 0x55a373e297d0 2024-04-27T03:18:06.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3a260 con 0x55a373e2db00 2024-04-27T03:18:06.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94160512318144] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94160512323072] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94160512325408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3a260 con 0x55a373e2db00 2024-04-27T03:18:06.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.161+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3a6b0 con 0x7ff1bc014030 2024-04-27T03:18:06.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94160512321872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3a670 con 0x7ff1bc014030 2024-04-27T03:18:06.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3b570 con 0x55a373e2db00 2024-04-27T03:18:06.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94160512321872] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94160512325408] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94160512327024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3b0c0 con 0x55a373e2db00 2024-04-27T03:18:06.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3be40 con 0x55a373e32c60 2024-04-27T03:18:06.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94160512334224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3b0c0 con 0x55a373e32c60 2024-04-27T03:18:06.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3cba0 con 0x55a373e32c60 2024-04-27T03:18:06.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.165+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94160512327024] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94160512334224] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94160512330400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3c690 con 0x55a373e32c60 2024-04-27T03:18:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3d5b0 con 0x7ff1bc014030 2024-04-27T03:18:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94160512330400] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'24 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94160512336624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3d5b0 con 0x7ff1bc014030 2024-04-27T03:18:06.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=0 2024-04-27T03:18:06.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=1 2024-04-27T03:18:06.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=2 2024-04-27T03:18:06.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=3 2024-04-27T03:18:06.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=4 2024-04-27T03:18:06.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=5 2024-04-27T03:18:06.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=6 2024-04-27T03:18:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94160512336624] v27'25 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 20 add_watcher() i=7 2024-04-27T03:18:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e40820 con 0x55a373e297d0 2024-04-27T03:18:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.169+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94160512340592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e3db40 con 0x55a373e297d0 2024-04-27T03:18:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94160512340592] v27'56 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff0737fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:06.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 5 note: GC not initialized 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff0727fc640 20 reqs_thread_entry: start 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 Filter name: none 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7fefe3fff640 20 reqs_thread_entry: start 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e64130 con 0x55a373e297d0 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache get: name=default.rgw.meta+users.email+foo@foo.com : miss 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.3 8:c162d5d1:users.email::foo@foo.com:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e62770 con 0x55a373e297d0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo@foo.com [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 197+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 adding default.rgw.meta+users.email+foo@foo.com to cache LRU end 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache get: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 : miss 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.7 8:e4e0e270:users.keys::9te6NH5mcdcq0Tc5i8i1:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e62770 con 0x55a373e297d0 2024-04-27T03:18:06.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(64 9te6NH5mcdcq0Tc5i8i1 [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 206+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache put: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 info.flags=0x0 2024-04-27T03:18:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 adding default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 to cache LRU end 2024-04-27T03:18:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache get: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 : hit (negative entry) 2024-04-27T03:18:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 10 cache get: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 : hit (negative entry) 2024-04-27T03:18:06.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.173+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e63190 con 0x55a373e297d0 2024-04-27T03:18:06.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(65 foo [delete,create,call,writefull 0~391] v27'1 uv1 ondisk = 0) ==== 273+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c44b3980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c44b3980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c44b3980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.d 7:b4812045:::notify.4:head [notify cookie 94160512494512 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e67be0 con 0x55a373e2db00 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94160512327024 notify 115964116992 ret 0) ==== 609+0+0 (crc 0 0 0) 0x55a373e21580 con 0x55a373e2db00 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff0a57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116992 cookie 94160512327024 notifier 4282 bl.length()=567 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff0a57fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff0a57fa640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff0a57fa640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7ff084003eb0 con 0x55a373e2db00 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.4 [notify cookie 94160512494512 out=8b] v0'0 uv24 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(67 notify.4 [notify-ack] v0'0 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94160512494512 notify 115964116992 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a373e1e8a0 con 0x55a373e2db00 2024-04-27T03:18:06.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.177+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 8.3 8:c162d5d1:users.email::foo@foo.com:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e62c20 con 0x55a373e297d0 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(68 foo@foo.com [delete,create,writefull 0~7] v27'1 uv1 ondisk = 0) ==== 239+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 10 cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x7 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 10 moving default.rgw.meta+users.email+foo@foo.com to cache LRU end 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:foo@foo.com, ofs0, ns] 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [notify cookie 94160512505520 in=181b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e684c0 con 0x7ff1bc014030 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== watch-notify(notify (1) cookie 94160512321872 notify 115964116992 ret 0) ==== 211+0+0 (crc 0 0 0) 0x55a373e63530 con 0x7ff1bc014030 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116992 cookie 94160512321872 notifier 4282 bl.length()=169 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a4ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x7 2024-04-27T03:18:06.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a4ff9640 10 rgw watcher librados: moving default.rgw.meta+users.email+foo@foo.com to cache LRU end 2024-04-27T03:18:06.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a4ff9640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fefec003410 con 0x7ff1bc014030 2024-04-27T03:18:06.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [notify cookie 94160512505520 out=8b] v0'0 uv24 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(70 notify.3 [notify-ack] v0'0 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== watch-notify(notify_complete (2) cookie 94160512505520 notify 115964116992 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a373e63d50 con 0x7ff1bc014030 2024-04-27T03:18:06.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:71 8.7 8:e4e0e270:users.keys::9te6NH5mcdcq0Tc5i8i1:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e66170 con 0x55a373e297d0 2024-04-27T03:18:06.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(71 9te6NH5mcdcq0Tc5i8i1 [delete,create,writefull 0~7] v27'1 uv1 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 10 cache put: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 info.flags=0x7 2024-04-27T03:18:06.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 10 moving default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 to cache LRU end 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:9te6NH5mcdcq0Tc5i8i1, ofs0, ns] 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.e 7:7759931f:::notify.3:head [notify cookie 94160512509984 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a373e684c0 con 0x7ff1bc014030 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== watch-notify(notify (1) cookie 94160512321872 notify 115964116993 ret 0) ==== 219+0+0 (crc 0 0 0) 0x55a373e63d50 con 0x7ff1bc014030 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94160512321872 notifier 4282 bl.length()=177 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a57fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 info.flags=0x7 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a57fa640 10 rgw watcher librados: moving default.rgw.meta+users.keys+9te6NH5mcdcq0Tc5i8i1 to cache LRU end 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff0a57fa640 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7ff084003eb0 con 0x7ff1bc014030 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.181+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(72 notify.3 [notify cookie 94160512509984 out=8b] v0'0 uv24 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.185+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(73 notify.3 [notify-ack] v0'0 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.185+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== watch-notify(notify_complete (2) cookie 94160512509984 notify 115964116993 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a373ba7510 con 0x7ff1bc014030 2024-04-27T03:18:06.194 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:06.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.185+0000 7ff0737fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.185+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:74 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94160512340592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x55a373e297d0 2024-04-27T03:18:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.185+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(74 data_loggenerations_metadata [watch unwatch cookie 94160512340592] v27'57 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.185+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94160512318144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x55a373e2db00 2024-04-27T03:18:06.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.189+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(75 notify.0 [watch unwatch cookie 94160512318144] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.189+0000 7ff1c44b3980 20 remove_watcher() i=0 2024-04-27T03:18:06.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.189+0000 7ff1c44b3980 2 removed watcher, disabling cache 2024-04-27T03:18:06.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.189+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94160512323072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x55a373e297d0 2024-04-27T03:18:06.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.189+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(76 notify.1 [watch unwatch cookie 94160512323072] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x55a373e297d0 2024-04-27T03:18:06.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.189+0000 7ff1c44b3980 20 remove_watcher() i=1 2024-04-27T03:18:06.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.189+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94160512325408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x55a373e2db00 2024-04-27T03:18:06.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.193+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(77 notify.2 [watch unwatch cookie 94160512325408] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.193+0000 7ff1c44b3980 20 remove_watcher() i=2 2024-04-27T03:18:06.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.193+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94160512321872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x7ff1bc014030 2024-04-27T03:18:06.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.193+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(78 notify.3 [watch unwatch cookie 94160512321872] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.193+0000 7ff1c44b3980 20 remove_watcher() i=3 2024-04-27T03:18:06.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.193+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:79 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94160512327024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x55a373e2db00 2024-04-27T03:18:06.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c27cf640 1 -- 172.21.15.110:0/4138979828 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(79 notify.4 [watch unwatch cookie 94160512327024] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1bc026020 con 0x55a373e2db00 2024-04-27T03:18:06.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 20 remove_watcher() i=4 2024-04-27T03:18:06.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:80 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94160512334224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x55a373e32c60 2024-04-27T03:18:06.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(80 notify.5 [watch unwatch cookie 94160512334224] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 20 remove_watcher() i=5 2024-04-27T03:18:06.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:81 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94160512330400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x55a373e32c60 2024-04-27T03:18:06.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c1fce640 1 -- 172.21.15.110:0/4138979828 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(81 notify.6 [watch unwatch cookie 94160512330400] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b402e070 con 0x55a373e32c60 2024-04-27T03:18:06.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 20 remove_watcher() i=6 2024-04-27T03:18:06.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:82 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94160512336624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a373e61dd0 con 0x7ff1bc014030 2024-04-27T03:18:06.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(82 notify.7 [watch unwatch cookie 94160512336624] v27'26 uv24 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff1b80aa950 con 0x7ff1bc014030 2024-04-27T03:18:06.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 20 remove_watcher() i=7 2024-04-27T03:18:06.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373e2db00 msgr2=0x55a373e2ff30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373e2db00 0x55a373e2ff30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373e297d0 msgr2=0x55a373e2bc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373e297d0 0x55a373e2bc90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff1bc014030 msgr2=0x7ff1bc039300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff1bc014030 0x7ff1bc039300 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a373e32c60 msgr2=0x55a373e35050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.197+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a373e32c60 0x55a373e35050 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff08c034650 msgr2=0x7ff08c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff08c034650 0x7ff08c036b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4024000 tx=0x7ff1b403c000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 msgr2=0x55a373e1e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e1e0d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc01eb40 tx=0x7ff1bc04a880 comp rx=0 tx=0).stop 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c17cd640 1 -- 172.21.15.110:0/4138979828 reap_dead start 2024-04-27T03:18:06.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 shutdown_connections 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/4138979828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e16c60 0x55a373e1e0d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 >> 172.21.15.110:0/4138979828 conn(0x55a373e12ff0 msgr2=0x55a373e184c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 shutdown_connections 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/4138979828 wait complete. 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff09c034710 msgr2=0x7ff09c036bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff09c034710 0x7ff09c036bd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x55a373df9f60 tx=0x7ff1b4001400 comp rx=0 tx=0).stop 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e09b10 msgr2=0x55a373e09f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e09b10 0x55a373e09f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00dfc0 tx=0x7ff1bc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 shutdown_connections 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff09c034710 0x7ff09c036bd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/179670632 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373e09b10 0x55a373e09f30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 >> 172.21.15.110:0/179670632 conn(0x55a373dfe920 msgr2=0x55a373dff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 shutdown_connections 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/179670632 wait complete. 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373df6f30 msgr2=0x55a373df9320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373df6f30 0x55a373df9320 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373df3770 msgr2=0x55a373df5b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373df3770 0x55a373df5b60 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff0a8034650 msgr2=0x7ff0a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff0a8034650 0x7ff0a8036b10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80ae000 tx=0x7ff1b80a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 msgr2=0x55a373dee850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373dee850 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x55a373de86a0 tx=0x7ff1b4000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 shutdown_connections 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a373df6f30 0x55a373df9320 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a373df3770 0x55a373df5b60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff0a8034650 0x7ff0a8036b10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 --2- 172.21.15.110:0/1102759164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a373c6e830 0x55a373dee850 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 >> 172.21.15.110:0/1102759164 conn(0x55a373cf49c0 msgr2=0x55a373de1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 shutdown_connections 2024-04-27T03:18:06.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.201+0000 7ff1c44b3980 1 -- 172.21.15.110:0/1102759164 wait complete. 2024-04-27T03:18:06.219 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:06.219 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--email', 'foo@foo.com'] 2024-04-27T03:18:06.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--email', 'foo@foo.com'] 2024-04-27T03:18:06.220 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid fud --display-name Fud --email foo@foo.com 2024-04-27T03:18:06.253 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:06.254 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.245+0000 7f25965c8980 1 Processor -- start 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.245+0000 7f25965c8980 1 -- start start 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.245+0000 7f25965c8980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x561f90d56740 0x561f90d58b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.245+0000 7f25965c8980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x561f90c110d0 con 0x561f90d52230 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.245+0000 7f25965c8980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x561f90c06430 con 0x561f90d56740 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.245+0000 7f25938e2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x561f90d56740 0x561f90d58b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25940e3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x561f90d52230 0x561f90d52610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34588/0 (socket says 172.21.15.110:34588) 2024-04-27T03:18:06.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25940e3640 1 -- 172.21.15.110:0/2925715271 learned_addr learned my addr 172.21.15.110:0/2925715271 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25938e2640 1 -- 172.21.15.110:0/2925715271 >> v1:172.21.15.77:6789/0 conn(0x561f90d52230 legacy=0x561f90d52610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25938e2640 1 -- 172.21.15.110:0/2925715271 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f90bff510 con 0x561f90d56740 2024-04-27T03:18:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25938e2640 1 --2- 172.21.15.110:0/2925715271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d56740 0x561f90d58b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f258800a480 tx=0x7f258801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e953ed9464d96be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25930e1640 1 -- 172.21.15.110:0/2925715271 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25880b0070 con 0x561f90d56740 2024-04-27T03:18:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25930e1640 1 -- 172.21.15.110:0/2925715271 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25880ab540 con 0x561f90d56740 2024-04-27T03:18:06.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25930e1640 1 -- 172.21.15.110:0/2925715271 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25880ab840 con 0x561f90d56740 2024-04-27T03:18:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25965c8980 1 -- 172.21.15.110:0/2925715271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d56740 msgr2=0x561f90d58b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25965c8980 1 --2- 172.21.15.110:0/2925715271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d56740 0x561f90d58b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f258800a480 tx=0x7f258801b280 comp rx=0 tx=0).stop 2024-04-27T03:18:06.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25965c8980 1 -- 172.21.15.110:0/2925715271 shutdown_connections 2024-04-27T03:18:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25965c8980 1 --2- 172.21.15.110:0/2925715271 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d56740 0x561f90d58b30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25965c8980 1 -- 172.21.15.110:0/2925715271 >> 172.21.15.110:0/2925715271 conn(0x561f90d4db00 msgr2=0x561f90d4ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25965c8980 1 -- 172.21.15.110:0/2925715271 shutdown_connections 2024-04-27T03:18:06.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.249+0000 7f25965c8980 1 -- 172.21.15.110:0/2925715271 wait complete. 2024-04-27T03:18:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25965c8980 1 Processor -- start 2024-04-27T03:18:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25965c8980 1 -- start start 2024-04-27T03:18:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25965c8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90e3d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25965c8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561f90c552d0 con 0x561f90d52230 2024-04-27T03:18:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25940e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90e3d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25940e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90e3d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50350/0 (socket says 172.21.15.110:50350) 2024-04-27T03:18:06.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25940e3640 1 -- 172.21.15.110:0/1943899631 learned_addr learned my addr 172.21.15.110:0/1943899631 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25940e3640 1 -- 172.21.15.110:0/1943899631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f90bdbae0 con 0x561f90d52230 2024-04-27T03:18:06.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f25940e3640 1 --2- 172.21.15.110:0/1943899631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90e3d3d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2584001e20 tx=0x7f258400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=190e0e366f565a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.265+0000 7f248260c640 1 -- 172.21.15.110:0/1943899631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2584012020 con 0x561f90d52230 2024-04-27T03:18:06.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f248260c640 1 -- 172.21.15.110:0/1943899631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258400e7c0 con 0x561f90d52230 2024-04-27T03:18:06.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f248260c640 1 -- 172.21.15.110:0/1943899631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258400eb00 con 0x561f90d52230 2024-04-27T03:18:06.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- 172.21.15.110:0/1943899631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 msgr2=0x561f90e3d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 --2- 172.21.15.110:0/1943899631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90e3d3d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2584001e20 tx=0x7f258400b790 comp rx=0 tx=0).stop 2024-04-27T03:18:06.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- 172.21.15.110:0/1943899631 shutdown_connections 2024-04-27T03:18:06.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 --2- 172.21.15.110:0/1943899631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90e3d3d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- 172.21.15.110:0/1943899631 >> 172.21.15.110:0/1943899631 conn(0x561f90d4db00 msgr2=0x561f90e38f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- 172.21.15.110:0/1943899631 shutdown_connections 2024-04-27T03:18:06.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- 172.21.15.110:0/1943899631 wait complete. 2024-04-27T03:18:06.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 Processor -- start 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- start start 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90cc85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561f90d5bb60 con 0x561f90d52230 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25940e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90cc85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25940e3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90cc85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50358/0 (socket says 172.21.15.110:50358) 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25940e3640 1 -- 172.21.15.110:0/2626150417 learned_addr learned my addr 172.21.15.110:0/2626150417 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25940e3640 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f90cc8b30 con 0x561f90d52230 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25940e3640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90cc85f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2584001e20 tx=0x7f2584000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f2480e09640 1 -- 172.21.15.110:0/2626150417 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2584012030 con 0x561f90d52230 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561f90cc6e70 con 0x561f90d52230 2024-04-27T03:18:06.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f2480e09640 1 -- 172.21.15.110:0/2626150417 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2584001480 con 0x561f90d52230 2024-04-27T03:18:06.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f2480e09640 1 -- 172.21.15.110:0/2626150417 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258401a940 con 0x561f90d52230 2024-04-27T03:18:06.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.269+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f90cc7360 con 0x561f90d52230 2024-04-27T03:18:06.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f2480e09640 1 -- 172.21.15.110:0/2626150417 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2584015020 con 0x561f90d52230 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f2480e09640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f247c034650 0x7f247c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f2480e09640 1 -- 172.21.15.110:0/2626150417 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2584020030 con 0x561f90d52230 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25938e2640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f247c034650 0x7f247c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90d56740 0x561f90e4d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e4df30 con 0x561f90d56740 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25948e4640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90d56740 0x561f90e4d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25938e2640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f247c034650 0x7f247c036b10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f25880ad000 tx=0x7f25880a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25948e4640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90d56740 0x561f90e4d9f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25948e4640 1 -- 172.21.15.110:0/2626150417 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90d56740 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e4e830 0x561f90e50cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e51440 con 0x561f90e4e830 2024-04-27T03:18:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25940e3640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e4e830 0x561f90e50cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25940e3640 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e4e830 0x561f90e50cd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25940e3640 1 -- 172.21.15.110:0/2626150417 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f2584008030 con 0x561f90e4e830 2024-04-27T03:18:06.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e52cf0 con 0x561f90d56740 2024-04-27T03:18:06.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25948e4640 1 -- 172.21.15.110:0/2626150417 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90d56740 2024-04-27T03:18:06.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.273+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e51f20 con 0x561f90e4e830 2024-04-27T03:18:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25940e3640 1 -- 172.21.15.110:0/2626150417 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f2584008030 con 0x561f90e4e830 2024-04-27T03:18:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e52310 con 0x561f90d56740 2024-04-27T03:18:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25948e4640 1 -- 172.21.15.110:0/2626150417 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90d56740 2024-04-27T03:18:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25965c8980 1 Processor -- start 2024-04-27T03:18:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25965c8980 1 -- start start 2024-04-27T03:18:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25965c8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e61690 0x561f90e61ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25965c8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561f90c59310 con 0x561f90e61690 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25948e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e61690 0x561f90e61ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25948e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e61690 0x561f90e61ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50360/0 (socket says 172.21.15.110:50360) 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25948e4640 1 -- 172.21.15.110:0/3084374797 learned_addr learned my addr 172.21.15.110:0/3084374797 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25948e4640 1 -- 172.21.15.110:0/3084374797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f90e61ff0 con 0x561f90e61690 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25948e4640 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e61690 0x561f90e61ab0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f258c0056d0 tx=0x7f258c017820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f2479ffb640 1 -- 172.21.15.110:0/3084374797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258c012840 con 0x561f90e61690 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f2479ffb640 1 -- 172.21.15.110:0/3084374797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258c0129e0 con 0x561f90e61690 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561f90e62a90 con 0x561f90e61690 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f2479ffb640 1 -- 172.21.15.110:0/3084374797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258c012ce0 con 0x561f90e61690 2024-04-27T03:18:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.277+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f90e62f60 con 0x561f90e61690 2024-04-27T03:18:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f2479ffb640 1 -- 172.21.15.110:0/3084374797 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f258c020440 con 0x561f90e61690 2024-04-27T03:18:06.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f2479ffb640 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f246c034650 0x7f246c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f2479ffb640 1 -- 172.21.15.110:0/3084374797 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f258c00e040 con 0x561f90e61690 2024-04-27T03:18:06.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25940e3640 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f246c034650 0x7f246c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 Processor -- start 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- start start 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25940e3640 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f246c034650 0x7f246c036b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x561f90e51a40 tx=0x7f2584048500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e6ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561f90e3db20 con 0x561f90e6e810 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e6ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e6ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50374/0 (socket says 172.21.15.110:50374) 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 -- 172.21.15.110:0/360219020 learned_addr learned my addr 172.21.15.110:0/360219020 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 -- 172.21.15.110:0/360219020 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f90e6f200 con 0x561f90e6e810 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 --2- 172.21.15.110:0/360219020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e6ec30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f258c04a6a0 tx=0x7f258c014410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a54781815e0c4a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f24737fe640 1 -- 172.21.15.110:0/360219020 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258c01f3a0 con 0x561f90e6e810 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f24737fe640 1 -- 172.21.15.110:0/360219020 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258c01f540 con 0x561f90e6e810 2024-04-27T03:18:06.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f24737fe640 1 -- 172.21.15.110:0/360219020 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258c020880 con 0x561f90e6e810 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- 172.21.15.110:0/360219020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 msgr2=0x561f90e6ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 --2- 172.21.15.110:0/360219020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e6ec30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f258c04a6a0 tx=0x7f258c014410 comp rx=0 tx=0).stop 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- 172.21.15.110:0/360219020 shutdown_connections 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 --2- 172.21.15.110:0/360219020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e6ec30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- 172.21.15.110:0/360219020 >> 172.21.15.110:0/360219020 conn(0x561f90e6aba0 msgr2=0x561f90e68c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- 172.21.15.110:0/360219020 shutdown_connections 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- 172.21.15.110:0/360219020 wait complete. 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 Processor -- start 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- start start 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e75ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25965c8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561f90e6f950 con 0x561f90e6e810 2024-04-27T03:18:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e75ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e75ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50386/0 (socket says 172.21.15.110:50386) 2024-04-27T03:18:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.281+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 learned_addr learned my addr 172.21.15.110:0/2960610865 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f90e761e0 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25948e4640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e75ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f258c014720 tx=0x7f258c04a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f2471ffb640 1 -- 172.21.15.110:0/2960610865 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258c01e930 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f2471ffb640 1 -- 172.21.15.110:0/2960610865 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258c044040 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561f90e76470 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f2471ffb640 1 -- 172.21.15.110:0/2960610865 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f258c013b80 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f90e79150 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f2471ffb640 1 -- 172.21.15.110:0/2960610865 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f258c0133b0 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f2471ffb640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2464034650 0x7f2464036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f2471ffb640 1 -- 172.21.15.110:0/2960610865 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f258c034c30 con 0x561f90e6e810 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25940e3640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2464034650 0x7f2464036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90e813c0 0x561f90e83880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e83fd0 con 0x561f90e813c0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25938e2640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90e813c0 0x561f90e83880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25940e3640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2464034650 0x7f2464036b10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2584048890 tx=0x7f2584035000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25938e2640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90e813c0 0x561f90e83880 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 realm 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e84400 con 0x561f90e813c0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e84cd0 con 0x561f90e813c0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e856f0 0x561f90e87b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.285+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e88290 con 0x561f90e856f0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25948e4640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e856f0 0x561f90e87b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25948e4640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e856f0 0x561f90e87b20 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25965c8980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e88ae0 con 0x561f90e813c0 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25965c8980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25965c8980 20 searching for the correct realm 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561f90e8a850 0x561f90e8cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e8a850 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25940e3640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561f90e8a850 0x561f90e8cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25940e3640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561f90e8a850 0x561f90e8cc40 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25948e4640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f258c014030 0x7f258c03e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x7f258c014030 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.289+0000 7f25938e2640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f258c014030 0x7f258c03e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f258c014030 0x7f258c03e460 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e8a850 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x7f258c014030 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.293+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x7f258c014030 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e8a850 2024-04-27T03:18:06.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e8a850 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e8a850 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x7f258c014030 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.297+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e813c0 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x7f258c014030 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e8a850 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x7f258c014030 2024-04-27T03:18:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e8a850 2024-04-27T03:18:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.301+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8d2c0 con 0x561f90e856f0 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e89460 con 0x561f90e813c0 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e89d00 con 0x561f90e856f0 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e890a0 con 0x561f90e813c0 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 zone default found 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 4 Realm: () 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:06.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 zonegroup default 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8dbd0 con 0x7f258c014030 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8f870 con 0x561f90e8a850 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8fd30 con 0x561f90e8a850 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 started sync module instance, tier type = 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.305+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e8e740 con 0x561f90e856f0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94693575158576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e90640 con 0x561f90e856f0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e90e90 con 0x561f90e813c0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94693575165504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e90f90 con 0x561f90e813c0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e91ab0 con 0x561f90e856f0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94693575158576] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.309+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94693575165504] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94693575166560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e91a70 con 0x561f90e856f0 2024-04-27T03:18:06.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e92350 con 0x7f258c014030 2024-04-27T03:18:06.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94693575171744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e91a70 con 0x7f258c014030 2024-04-27T03:18:06.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e92e00 con 0x561f90e856f0 2024-04-27T03:18:06.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94693575171744] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94693575166560] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94693575169408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e92e00 con 0x561f90e856f0 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.313+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e931f0 con 0x561f90e8a850 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94693575172800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e93a40 con 0x561f90e8a850 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e94440 con 0x561f90e8a850 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94693575169408] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94693575172800] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94693575174416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e93f50 con 0x561f90e8a850 2024-04-27T03:18:06.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e94ce0 con 0x7f258c014030 2024-04-27T03:18:06.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94693575174416] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'27 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94693575177616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e93f50 con 0x7f258c014030 2024-04-27T03:18:06.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 20 add_watcher() i=0 2024-04-27T03:18:06.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 20 add_watcher() i=1 2024-04-27T03:18:06.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 20 add_watcher() i=2 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 20 add_watcher() i=3 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 20 add_watcher() i=4 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 20 add_watcher() i=5 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.317+0000 7f25965c8980 20 add_watcher() i=6 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94693575177616] v27'28 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 add_watcher() i=7 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90e97dd0 con 0x561f90e813c0 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94693575182384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e952f0 con 0x561f90e813c0 2024-04-27T03:18:06.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94693575182384] v27'58 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f244a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:06.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 5 note: GC not initialized 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f24497e2640 20 reqs_thread_entry: start 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 Filter name: none 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f23baffd640 20 reqs_thread_entry: start 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90ebb5d0 con 0x7f258c014030 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 fud [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2024-04-27T03:18:06.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 10 cache get: name=default.rgw.meta+users.email+foo@foo.com : miss 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.321+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.3 8:c162d5d1:users.email::foo@foo.com:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9c10 con 0x561f90e813c0 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 foo@foo.com [stat out=16b,read 0~7 out=7b] v0'0 uv1 ondisk = 0) ==== 197+0+23 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 20 rados_obj.operate() r=0 bl.length=7 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 10 cache put: name=default.rgw.meta+users.email+foo@foo.com info.flags=0x5 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 10 adding default.rgw.meta+users.email+foo@foo.com to cache LRU end 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561f90eba000 con 0x561f90e813c0 2024-04-27T03:18:06.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(64 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:06.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:06.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:06.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:06.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 10 chain_cache_entry: cache_locator=default.rgw.meta+users.uid+foo 2024-04-27T03:18:06.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 20 get_user_info_from_index found 0 xattrs 2024-04-27T03:18:06.336 INFO:teuthology.orchestra.run.smithi110.stderr:could not create user: unable to parse parameters, email: foo@foo.com is the email address of an existing user 2024-04-27T03:18:06.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f244a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94693575182384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90e7b020 con 0x561f90e813c0 2024-04-27T03:18:06.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.325+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 data_loggenerations_metadata [watch unwatch cookie 94693575182384] v27'59 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94693575158576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x561f90e856f0 2024-04-27T03:18:06.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(66 notify.0 [watch unwatch cookie 94693575158576] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25965c8980 20 remove_watcher() i=0 2024-04-27T03:18:06.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25965c8980 2 removed watcher, disabling cache 2024-04-27T03:18:06.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94693575165504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x561f90e813c0 2024-04-27T03:18:06.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(67 notify.1 [watch unwatch cookie 94693575165504] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x561f90e813c0 2024-04-27T03:18:06.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25965c8980 20 remove_watcher() i=1 2024-04-27T03:18:06.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.329+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94693575166560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x561f90e856f0 2024-04-27T03:18:06.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.333+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [watch unwatch cookie 94693575166560] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.333+0000 7f25965c8980 20 remove_watcher() i=2 2024-04-27T03:18:06.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.333+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94693575171744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x7f258c014030 2024-04-27T03:18:06.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.333+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [watch unwatch cookie 94693575171744] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.333+0000 7f25965c8980 20 remove_watcher() i=3 2024-04-27T03:18:06.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.333+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94693575169408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x561f90e856f0 2024-04-27T03:18:06.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25948e4640 1 -- 172.21.15.110:0/2960610865 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(70 notify.4 [watch unwatch cookie 94693575169408] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f258c00a4d0 con 0x561f90e856f0 2024-04-27T03:18:06.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25965c8980 20 remove_watcher() i=4 2024-04-27T03:18:06.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94693575172800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x561f90e8a850 2024-04-27T03:18:06.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(71 notify.5 [watch unwatch cookie 94693575172800] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25965c8980 20 remove_watcher() i=5 2024-04-27T03:18:06.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94693575174416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x561f90e8a850 2024-04-27T03:18:06.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25940e3640 1 -- 172.21.15.110:0/2960610865 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(72 notify.6 [watch unwatch cookie 94693575174416] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2584008030 con 0x561f90e8a850 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25965c8980 20 remove_watcher() i=6 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.337+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94693575177616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561f90eb9300 con 0x7f258c014030 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(73 notify.7 [watch unwatch cookie 94693575177616] v27'29 uv27 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f25880b3040 con 0x7f258c014030 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 20 remove_watcher() i=7 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e856f0 msgr2=0x561f90e87b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e856f0 0x561f90e87b20 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90e813c0 msgr2=0x561f90e83880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90e813c0 0x561f90e83880 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f258c014030 msgr2=0x7f258c03e460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f258c014030 0x7f258c03e460 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561f90e8a850 msgr2=0x561f90e8cc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561f90e8a850 0x561f90e8cc40 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2464034650 msgr2=0x7f2464036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2464034650 0x7f2464036b10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2584048890 tx=0x7f2584035000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 msgr2=0x561f90e75ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25938e2640 1 -- 172.21.15.110:0/2960610865 reap_dead start 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e75ca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f258c014720 tx=0x7f258c04a600 comp rx=0 tx=0).stop 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 shutdown_connections 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2960610865 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e6e810 0x561f90e75ca0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 >> 172.21.15.110:0/2960610865 conn(0x561f90e6aba0 msgr2=0x561f90e70070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 shutdown_connections 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2960610865 wait complete. 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f246c034650 msgr2=0x7f246c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f246c034650 0x7f246c036b10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x561f90e51a40 tx=0x7f2584048500 comp rx=0 tx=0).stop 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e61690 msgr2=0x561f90e61ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e61690 0x561f90e61ab0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f258c0056d0 tx=0x7f258c017820 comp rx=0 tx=0).stop 2024-04-27T03:18:06.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 shutdown_connections 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f246c034650 0x7f246c036b10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/3084374797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90e61690 0x561f90e61ab0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 >> 172.21.15.110:0/3084374797 conn(0x561f90e564a0 msgr2=0x561f90e56fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 shutdown_connections 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/3084374797 wait complete. 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e4e830 msgr2=0x561f90e50cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e4e830 0x561f90e50cd0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90d56740 msgr2=0x561f90e4d9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90d56740 0x561f90e4d9f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f247c034650 msgr2=0x7f247c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f247c034650 0x7f247c036b10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f25880ad000 tx=0x7f25880a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 msgr2=0x561f90cc85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90cc85f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2584001e20 tx=0x7f2584000ee0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 shutdown_connections 2024-04-27T03:18:06.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561f90e4e830 0x561f90e50cd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561f90d56740 0x561f90e4d9f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f247c034650 0x7f247c036b10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 --2- 172.21.15.110:0/2626150417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561f90d52230 0x561f90cc85f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 >> 172.21.15.110:0/2626150417 conn(0x561f90d4db00 msgr2=0x561f90e37a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 shutdown_connections 2024-04-27T03:18:06.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.341+0000 7f25965c8980 1 -- 172.21.15.110:0/2626150417 wait complete. 2024-04-27T03:18:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 240 2024-04-27T03:18:06.359 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:06.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:06.360 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:06.439 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392e06c980 1 Processor -- start 2024-04-27T03:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392e06c980 1 -- start start 2024-04-27T03:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392e06c980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f842621fc0 0x55f8426b0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392e06c980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55f84256d0d0 con 0x55f8426b14a0 2024-04-27T03:18:06.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392e06c980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55f842562430 con 0x55f842621fc0 2024-04-27T03:18:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392bb87640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f842621fc0 0x55f8426b0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392b386640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55f8426b14a0 0x55f8426b3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34594/0 (socket says 172.21.15.110:34594) 2024-04-27T03:18:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392bb87640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f842621fc0 0x55f8426b0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50398/0 (socket says 172.21.15.110:50398) 2024-04-27T03:18:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392b386640 1 -- 172.21.15.110:0/1795740746 learned_addr learned my addr 172.21.15.110:0/1795740746 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392bb87640 1 -- 172.21.15.110:0/1795740746 >> v1:172.21.15.77:6789/0 conn(0x55f8426b14a0 legacy=0x55f8426b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.433+0000 7f392bb87640 1 -- 172.21.15.110:0/1795740746 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f84255b510 con 0x55f842621fc0 2024-04-27T03:18:06.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392bb87640 1 --2- 172.21.15.110:0/1795740746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8426b0f60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f391c00a480 tx=0x7f391c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d416fa43143daf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392ab85640 1 -- 172.21.15.110:0/1795740746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f391c0b0070 con 0x55f842621fc0 2024-04-27T03:18:06.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392ab85640 1 -- 172.21.15.110:0/1795740746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f391c0ab540 con 0x55f842621fc0 2024-04-27T03:18:06.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392ab85640 1 -- 172.21.15.110:0/1795740746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f391c0ab840 con 0x55f842621fc0 2024-04-27T03:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392e06c980 1 -- 172.21.15.110:0/1795740746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 msgr2=0x55f8426b0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392e06c980 1 --2- 172.21.15.110:0/1795740746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8426b0f60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f391c00a480 tx=0x7f391c01b280 comp rx=0 tx=0).stop 2024-04-27T03:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392e06c980 1 -- 172.21.15.110:0/1795740746 shutdown_connections 2024-04-27T03:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392e06c980 1 --2- 172.21.15.110:0/1795740746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8426b0f60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392e06c980 1 -- 172.21.15.110:0/1795740746 >> 172.21.15.110:0/1795740746 conn(0x55f8426a8a40 msgr2=0x55f8426aae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392e06c980 1 -- 172.21.15.110:0/1795740746 shutdown_connections 2024-04-27T03:18:06.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.437+0000 7f392e06c980 1 -- 172.21.15.110:0/1795740746 wait complete. 2024-04-27T03:18:06.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 Processor -- start 2024-04-27T03:18:06.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- start start 2024-04-27T03:18:06.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427994e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f8425b12d0 con 0x55f842621fc0 2024-04-27T03:18:06.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392bb87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392bb87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50406/0 (socket says 172.21.15.110:50406) 2024-04-27T03:18:06.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392bb87640 1 -- 172.21.15.110:0/3490771617 learned_addr learned my addr 172.21.15.110:0/3490771617 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392bb87640 1 -- 172.21.15.110:0/3490771617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f842537ae0 con 0x55f842621fc0 2024-04-27T03:18:06.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392bb87640 1 --2- 172.21.15.110:0/3490771617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427994e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f391c0ae040 tx=0x7f391c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a647a124ba6a1f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f3819e0b640 1 -- 172.21.15.110:0/3490771617 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f391c0b0040 con 0x55f842621fc0 2024-04-27T03:18:06.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f3819e0b640 1 -- 172.21.15.110:0/3490771617 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f391c00a300 con 0x55f842621fc0 2024-04-27T03:18:06.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f3819e0b640 1 -- 172.21.15.110:0/3490771617 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f391c0ab820 con 0x55f842621fc0 2024-04-27T03:18:06.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- 172.21.15.110:0/3490771617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 msgr2=0x55f8427994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 --2- 172.21.15.110:0/3490771617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427994e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f391c0ae040 tx=0x7f391c003030 comp rx=0 tx=0).stop 2024-04-27T03:18:06.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- 172.21.15.110:0/3490771617 shutdown_connections 2024-04-27T03:18:06.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 --2- 172.21.15.110:0/3490771617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427994e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- 172.21.15.110:0/3490771617 >> 172.21.15.110:0/3490771617 conn(0x55f8426a8a40 msgr2=0x55f842795070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- 172.21.15.110:0/3490771617 shutdown_connections 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- 172.21.15.110:0/3490771617 wait complete. 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 Processor -- start 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- start start 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.453+0000 7f392e06c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f8426b5ac0 con 0x55f842621fc0 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392bb87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392bb87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50410/0 (socket says 172.21.15.110:50410) 2024-04-27T03:18:06.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392bb87640 1 -- 172.21.15.110:0/2899405375 learned_addr learned my addr 172.21.15.110:0/2899405375 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392bb87640 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8427a09a0 con 0x55f842621fc0 2024-04-27T03:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392bb87640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427a0460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f391c019930 tx=0x7f391c018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f3813fff640 1 -- 172.21.15.110:0/2899405375 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f391c0b0040 con 0x55f842621fc0 2024-04-27T03:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f3813fff640 1 -- 172.21.15.110:0/2899405375 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f391c0c0870 con 0x55f842621fc0 2024-04-27T03:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f8427a0c30 con 0x55f842621fc0 2024-04-27T03:18:06.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f3813fff640 1 -- 172.21.15.110:0/2899405375 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f391c0c0b50 con 0x55f842621fc0 2024-04-27T03:18:06.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8427a3810 con 0x55f842621fc0 2024-04-27T03:18:06.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f3813fff640 1 -- 172.21.15.110:0/2899405375 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f391c0b6470 con 0x55f842621fc0 2024-04-27T03:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f3813fff640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3814034650 0x7f3814036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f3813fff640 1 -- 172.21.15.110:0/2899405375 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f391c0bb070 con 0x55f842621fc0 2024-04-27T03:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392b386640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3814034650 0x7f3814036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8426b14a0 0x55f8427a72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427a7870 con 0x55f8426b14a0 2024-04-27T03:18:06.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392c388640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8426b14a0 0x55f8427a72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392b386640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3814034650 0x7f3814036b10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3920000f70 tx=0x7f39200012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392c388640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8426b14a0 0x55f8427a72f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392c388640 1 -- 172.21.15.110:0/2899405375 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f392400a4d0 con 0x55f8426b14a0 2024-04-27T03:18:06.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427a8660 0x55f8427aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.457+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ab080 con 0x55f8427a8660 2024-04-27T03:18:06.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392bb87640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427a8660 0x55f8427aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392bb87640 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427a8660 0x55f8427aaa50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392bb87640 1 -- 172.21.15.110:0/2899405375 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427a8660 2024-04-27T03:18:06.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ac930 con 0x55f8426b14a0 2024-04-27T03:18:06.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392c388640 1 -- 172.21.15.110:0/2899405375 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f392400a4d0 con 0x55f8426b14a0 2024-04-27T03:18:06.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427abb60 con 0x55f8427a8660 2024-04-27T03:18:06.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392bb87640 1 -- 172.21.15.110:0/2899405375 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427a8660 2024-04-27T03:18:06.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427abf50 con 0x55f8426b14a0 2024-04-27T03:18:06.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392c388640 1 -- 172.21.15.110:0/2899405375 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f392400a4d0 con 0x55f8426b14a0 2024-04-27T03:18:06.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392e06c980 1 Processor -- start 2024-04-27T03:18:06.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.461+0000 7f392e06c980 1 -- start start 2024-04-27T03:18:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8427bb240 0x55f8427bb660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f8425b5310 con 0x55f8427bb240 2024-04-27T03:18:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392c388640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8427bb240 0x55f8427bb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392c388640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8427bb240 0x55f8427bb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50414/0 (socket says 172.21.15.110:50414) 2024-04-27T03:18:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392c388640 1 -- 172.21.15.110:0/85202477 learned_addr learned my addr 172.21.15.110:0/85202477 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392c388640 1 -- 172.21.15.110:0/85202477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8427bbba0 con 0x55f8427bb240 2024-04-27T03:18:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392c388640 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8427bb240 0x55f8427bb660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f392400dfc0 tx=0x7f39240178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f38117fa640 1 -- 172.21.15.110:0/85202477 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39240157d0 con 0x55f8427bb240 2024-04-27T03:18:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f38117fa640 1 -- 172.21.15.110:0/85202477 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3924015970 con 0x55f8427bb240 2024-04-27T03:18:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f8427bc640 con 0x55f8427bb240 2024-04-27T03:18:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f38117fa640 1 -- 172.21.15.110:0/85202477 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3924015c70 con 0x55f8427bb240 2024-04-27T03:18:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8427bcb10 con 0x55f8427bb240 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f38117fa640 1 -- 172.21.15.110:0/85202477 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f3924029440 con 0x55f8427bb240 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f38117fa640 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3804034650 0x7f3804036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f38117fa640 1 -- 172.21.15.110:0/85202477 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f392400e040 con 0x55f8427bb240 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392bb87640 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3804034650 0x7f3804036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 Processor -- start 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 -- start start 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392bb87640 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3804034650 0x7f3804036b10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55f8427ab680 tx=0x7f391c0e3290 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8426ae910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.465+0000 7f392e06c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f842799b70 con 0x55f8426ae4f0 2024-04-27T03:18:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8426ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8426ae910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50428/0 (socket says 172.21.15.110:50428) 2024-04-27T03:18:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 -- 172.21.15.110:0/2807562787 learned_addr learned my addr 172.21.15.110:0/2807562787 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 -- 172.21.15.110:0/2807562787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8426acd40 con 0x55f8426ae4f0 2024-04-27T03:18:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 --2- 172.21.15.110:0/2807562787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8426ae910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3924015fa0 tx=0x7f3924015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f08445b7361248 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f380affd640 1 -- 172.21.15.110:0/2807562787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39240113a0 con 0x55f8426ae4f0 2024-04-27T03:18:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f380affd640 1 -- 172.21.15.110:0/2807562787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3924011540 con 0x55f8426ae4f0 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f380affd640 1 -- 172.21.15.110:0/2807562787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39240283a0 con 0x55f8426ae4f0 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- 172.21.15.110:0/2807562787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 msgr2=0x55f8426ae910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 --2- 172.21.15.110:0/2807562787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8426ae910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3924015fa0 tx=0x7f3924015fd0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- 172.21.15.110:0/2807562787 shutdown_connections 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 --2- 172.21.15.110:0/2807562787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8426ae910 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- 172.21.15.110:0/2807562787 >> 172.21.15.110:0/2807562787 conn(0x55f8427c4780 msgr2=0x55f8427c27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- 172.21.15.110:0/2807562787 shutdown_connections 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- 172.21.15.110:0/2807562787 wait complete. 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 Processor -- start 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- start start 2024-04-27T03:18:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8427d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f8426ad210 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8427d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8427d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50430/0 (socket says 172.21.15.110:50430) 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 learned_addr learned my addr 172.21.15.110:0/4170975682 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8427d2030 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392c388640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8427d1af0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f392401eb40 tx=0x7f392404a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f38097fa640 1 -- 172.21.15.110:0/4170975682 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39240113a0 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f8427d22c0 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f38097fa640 1 -- 172.21.15.110:0/4170975682 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f392401ee30 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f38097fa640 1 -- 172.21.15.110:0/4170975682 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3924013c60 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.469+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8427d4fa0 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f38097fa640 1 -- 172.21.15.110:0/4170975682 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f39240491b0 con 0x55f8426ae4f0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f38097fa640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f37f4034650 0x7f37f4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f38097fa640 1 -- 172.21.15.110:0/4170975682 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f3924002390 con 0x55f8426ae4f0 2024-04-27T03:18:06.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392bb87640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f37f4034650 0x7f37f4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8427dd1a0 0x55f8427df640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427dfd90 con 0x55f8427dd1a0 2024-04-27T03:18:06.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392b386640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8427dd1a0 0x55f8427df640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392bb87640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f37f4034650 0x7f37f4036b10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f391c0c7510 tx=0x7f391c0df000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392b386640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8427dd1a0 0x55f8427df640 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f392000e5a0 con 0x55f8427dd1a0 2024-04-27T03:18:06.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 realm 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427dc670 con 0x55f8427dd1a0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f392000e5a0 con 0x55f8427dd1a0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e0890 con 0x55f8427dd1a0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f392000e5a0 con 0x55f8427dd1a0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427e1670 0x55f8427e3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e40e0 con 0x55f8427e1670 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392c388640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427e1670 0x55f8427e3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.473+0000 7f392c388640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427e1670 0x55f8427e3a60 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392e06c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e4980 con 0x55f8427dd1a0 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f392000e5a0 con 0x55f8427dd1a0 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392e06c980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392e06c980 20 searching for the correct realm 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f8427e66f0 0x55f8427e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e66f0 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392bb87640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f8427e66f0 0x55f8427e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392bb87640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f8427e66f0 0x55f8427e8ae0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392c388640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3924014030 0x7f39240133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x7f3924014030 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3924014030 0x7f39240133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3924014030 0x7f39240133b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f392000e5a0 con 0x7f3924014030 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f392000e5a0 con 0x55f8427dd1a0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.477+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e66f0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x7f3924014030 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x7f3924014030 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x7f3924014030 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f3920029040 con 0x7f3924014030 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.481+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e66f0 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e66f0 2024-04-27T03:18:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e66f0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x7f3924014030 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x7f3924014030 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x7f3924014030 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x7f3924014030 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e66f0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.485+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x7f3924014030 2024-04-27T03:18:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3920029040 con 0x7f3924014030 2024-04-27T03:18:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e66f0 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e90c0 con 0x55f8427e1670 2024-04-27T03:18:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e52d0 con 0x55f8427dd1a0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3920029040 con 0x55f8427dd1a0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.489+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e5750 con 0x55f8427e1670 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e5bb0 con 0x55f8427dd1a0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 zone default found 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 4 Realm: () 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 zonegroup default 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427e9520 con 0x7f3924014030 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x7f3924014030 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427eb3b0 con 0x55f8427e66f0 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427eb750 con 0x55f8427e66f0 2024-04-27T03:18:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 started sync module instance, tier type = 2024-04-27T03:18:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ea7f0 con 0x55f8427e1670 2024-04-27T03:18:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94524755845952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ebaf0 con 0x55f8427e1670 2024-04-27T03:18:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.493+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ec4b0 con 0x55f8427dd1a0 2024-04-27T03:18:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94524755853312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ec8d0 con 0x55f8427dd1a0 2024-04-27T03:18:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ed310 con 0x55f8427e1670 2024-04-27T03:18:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94524755845952] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94524755853312] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94524755856864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ec420 con 0x55f8427e1670 2024-04-27T03:18:06.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.497+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427edbb0 con 0x7f3924014030 2024-04-27T03:18:06.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x7f3924014030 2024-04-27T03:18:06.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94524755859200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ec420 con 0x7f3924014030 2024-04-27T03:18:06.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ee670 con 0x55f8427e1670 2024-04-27T03:18:06.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94524755856864] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94524755859200] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x7f3924014030 2024-04-27T03:18:06.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94524755860256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ee630 con 0x55f8427e1670 2024-04-27T03:18:06.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.501+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427eea60 con 0x55f8427e66f0 2024-04-27T03:18:06.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94524755848160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427eea60 con 0x55f8427e66f0 2024-04-27T03:18:06.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ef920 con 0x55f8427e66f0 2024-04-27T03:18:06.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94524755860256] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94524755848160] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94524755865168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ef470 con 0x55f8427e66f0 2024-04-27T03:18:06.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ef470 con 0x7f3924014030 2024-04-27T03:18:06.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94524755865168] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'30 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x7f3924014030 2024-04-27T03:18:06.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94524755861824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ef470 con 0x7f3924014030 2024-04-27T03:18:06.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 20 add_watcher() i=0 2024-04-27T03:18:06.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 20 add_watcher() i=1 2024-04-27T03:18:06.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 20 add_watcher() i=2 2024-04-27T03:18:06.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 20 add_watcher() i=3 2024-04-27T03:18:06.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 20 add_watcher() i=4 2024-04-27T03:18:06.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 20 add_watcher() i=5 2024-04-27T03:18:06.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.505+0000 7f392e06c980 20 add_watcher() i=6 2024-04-27T03:18:06.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94524755861824] v27'31 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x7f3924014030 2024-04-27T03:18:06.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 20 add_watcher() i=7 2024-04-27T03:18:06.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:06.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8427f32b0 con 0x55f8427dd1a0 2024-04-27T03:18:06.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94524755868944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427f07d0 con 0x55f8427dd1a0 2024-04-27T03:18:06.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94524755868944] v27'60 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f37e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 5 note: GC not initialized 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f37e0fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 20 Filter name: none 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f375a7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f8428166a0 con 0x55f8427dd1a0 2024-04-27T03:18:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.509+0000 7f392e06c980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:06.520 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.513+0000 7f37e1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.513+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94524755868944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f842815490 con 0x55f8427dd1a0 2024-04-27T03:18:06.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.513+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94524755868944] v27'61 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.513+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94524755845952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427efd10 con 0x55f8427e1670 2024-04-27T03:18:06.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.517+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94524755845952] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.517+0000 7f392e06c980 20 remove_watcher() i=0 2024-04-27T03:18:06.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.517+0000 7f392e06c980 2 removed watcher, disabling cache 2024-04-27T03:18:06.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.517+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94524755853312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ea340 con 0x55f8427dd1a0 2024-04-27T03:18:06.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.517+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94524755853312] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x55f8427dd1a0 2024-04-27T03:18:06.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.517+0000 7f392e06c980 20 remove_watcher() i=1 2024-04-27T03:18:06.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.517+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94524755856864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ea340 con 0x55f8427e1670 2024-04-27T03:18:06.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.521+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94524755856864] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.521+0000 7f392e06c980 20 remove_watcher() i=2 2024-04-27T03:18:06.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.521+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94524755859200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ecde0 con 0x7f3924014030 2024-04-27T03:18:06.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.521+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94524755859200] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x7f3924014030 2024-04-27T03:18:06.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.521+0000 7f392e06c980 20 remove_watcher() i=3 2024-04-27T03:18:06.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.521+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94524755860256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ea340 con 0x55f8427e1670 2024-04-27T03:18:06.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392c388640 1 -- 172.21.15.110:0/4170975682 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94524755860256] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3924026020 con 0x55f8427e1670 2024-04-27T03:18:06.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 20 remove_watcher() i=4 2024-04-27T03:18:06.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94524755848160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ecde0 con 0x55f8427e66f0 2024-04-27T03:18:06.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94524755848160] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 20 remove_watcher() i=5 2024-04-27T03:18:06.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94524755865168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ea340 con 0x55f8427e66f0 2024-04-27T03:18:06.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392bb87640 1 -- 172.21.15.110:0/4170975682 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94524755865168] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f391c0b3040 con 0x55f8427e66f0 2024-04-27T03:18:06.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 20 remove_watcher() i=6 2024-04-27T03:18:06.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94524755861824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f8427ecde0 con 0x7f3924014030 2024-04-27T03:18:06.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94524755861824] v27'32 uv30 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f392003c1c0 con 0x7f3924014030 2024-04-27T03:18:06.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 20 remove_watcher() i=7 2024-04-27T03:18:06.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427e1670 msgr2=0x55f8427e3a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427e1670 0x55f8427e3a60 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8427dd1a0 msgr2=0x55f8427df640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8427dd1a0 0x55f8427df640 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3924014030 msgr2=0x7f39240133b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3924014030 0x7f39240133b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f8427e66f0 msgr2=0x55f8427e8ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f8427e66f0 0x55f8427e8ae0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f37f4034650 msgr2=0x7f37f4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f37f4034650 0x7f37f4036b10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f391c0c7510 tx=0x7f391c0df000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392b386640 1 -- 172.21.15.110:0/4170975682 reap_dead start 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 msgr2=0x55f8427d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8427d1af0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f392401eb40 tx=0x7f392404a880 comp rx=0 tx=0).stop 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 shutdown_connections 2024-04-27T03:18:06.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 --2- 172.21.15.110:0/4170975682 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8426ae4f0 0x55f8427d1af0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.525+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 >> 172.21.15.110:0/4170975682 conn(0x55f8427c4780 msgr2=0x55f8427c1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 shutdown_connections 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/4170975682 wait complete. 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3804034650 msgr2=0x7f3804036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3804034650 0x7f3804036b10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55f8427ab680 tx=0x7f391c0e3290 comp rx=0 tx=0).stop 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8427bb240 msgr2=0x55f8427bb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8427bb240 0x55f8427bb660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f392400dfc0 tx=0x7f39240178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 shutdown_connections 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3804034650 0x7f3804036b10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/85202477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f8427bb240 0x55f8427bb660 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 >> 172.21.15.110:0/85202477 conn(0x55f8427b0050 msgr2=0x55f8427b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 shutdown_connections 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/85202477 wait complete. 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427a8660 msgr2=0x55f8427aaa50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427a8660 0x55f8427aaa50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8426b14a0 msgr2=0x55f8427a72f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8426b14a0 0x55f8427a72f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3814034650 msgr2=0x7f3814036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3814034650 0x7f3814036b10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3920000f70 tx=0x7f39200012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 msgr2=0x55f8427a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427a0460 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f391c019930 tx=0x7f391c018020 comp rx=0 tx=0).stop 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 shutdown_connections 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f8427a8660 0x55f8427aaa50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f8426b14a0 0x55f8427a72f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3814034650 0x7f3814036b10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 --2- 172.21.15.110:0/2899405375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f842621fc0 0x55f8427a0460 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 >> 172.21.15.110:0/2899405375 conn(0x55f8426a8a40 msgr2=0x55f842793b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 shutdown_connections 2024-04-27T03:18:06.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.529+0000 7f392e06c980 1 -- 172.21.15.110:0/2899405375 wait complete. 2024-04-27T03:18:06.547 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:06.547 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'suspend', '--uid', 'foo'] 2024-04-27T03:18:06.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'suspend', '--uid', 'foo'] 2024-04-27T03:18:06.547 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user suspend --uid foo 2024-04-27T03:18:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.613+0000 7f58894c5980 1 Processor -- start 2024-04-27T03:18:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.613+0000 7f58894c5980 1 -- start start 2024-04-27T03:18:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.613+0000 7f58894c5980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5615922257a0 0x561592227b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.613+0000 7f58894c5980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5615920e00d0 con 0x561592221290 2024-04-27T03:18:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.613+0000 7f58894c5980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5615920d5430 con 0x5615922257a0 2024-04-27T03:18:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f58867df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5615922257a0 0x561592227b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f58867df640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5615922257a0 0x561592227b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50446/0 (socket says 172.21.15.110:50446) 2024-04-27T03:18:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f5886fe0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x561592221290 0x561592221670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34600/0 (socket says 172.21.15.110:34600) 2024-04-27T03:18:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f58867df640 1 -- 172.21.15.110:0/1610894388 learned_addr learned my addr 172.21.15.110:0/1610894388 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f58867df640 1 -- 172.21.15.110:0/1610894388 >> v1:172.21.15.77:6789/0 conn(0x561592221290 legacy=0x561592221670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f58867df640 1 -- 172.21.15.110:0/1610894388 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615920ce510 con 0x5615922257a0 2024-04-27T03:18:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f58867df640 1 --2- 172.21.15.110:0/1610894388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615922257a0 0x561592227b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f587000a5c0 tx=0x7f587001b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b902472fa518eed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f5885fde640 1 -- 172.21.15.110:0/1610894388 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58700a6070 con 0x5615922257a0 2024-04-27T03:18:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f5885fde640 1 -- 172.21.15.110:0/1610894388 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58700af070 con 0x5615922257a0 2024-04-27T03:18:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.617+0000 7f5885fde640 1 -- 172.21.15.110:0/1610894388 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58700aa5c0 con 0x5615922257a0 2024-04-27T03:18:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.621+0000 7f58894c5980 1 -- 172.21.15.110:0/1610894388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615922257a0 msgr2=0x561592227b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.621+0000 7f58894c5980 1 --2- 172.21.15.110:0/1610894388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615922257a0 0x561592227b90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f587000a5c0 tx=0x7f587001b340 comp rx=0 tx=0).stop 2024-04-27T03:18:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.621+0000 7f58894c5980 1 -- 172.21.15.110:0/1610894388 shutdown_connections 2024-04-27T03:18:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.621+0000 7f58894c5980 1 --2- 172.21.15.110:0/1610894388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5615922257a0 0x561592227b90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.621+0000 7f58894c5980 1 -- 172.21.15.110:0/1610894388 >> 172.21.15.110:0/1610894388 conn(0x56159221cb60 msgr2=0x56159221ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.621+0000 7f58894c5980 1 -- 172.21.15.110:0/1610894388 shutdown_connections 2024-04-27T03:18:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.621+0000 7f58894c5980 1 -- 172.21.15.110:0/1610894388 wait complete. 2024-04-27T03:18:06.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.633+0000 7f58894c5980 1 Processor -- start 2024-04-27T03:18:06.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.633+0000 7f58894c5980 1 -- start start 2024-04-27T03:18:06.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.633+0000 7f58894c5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x56159230c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.633+0000 7f58894c5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5615921242d0 con 0x561592221290 2024-04-27T03:18:06.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x56159230c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x56159230c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50460/0 (socket says 172.21.15.110:50460) 2024-04-27T03:18:06.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 -- 172.21.15.110:0/1670041375 learned_addr learned my addr 172.21.15.110:0/1670041375 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 -- 172.21.15.110:0/1670041375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615920aaae0 con 0x561592221290 2024-04-27T03:18:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 --2- 172.21.15.110:0/1670041375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x56159230c3d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5878001690 tx=0x7f5878001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143819fd8d84922b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5774e09640 1 -- 172.21.15.110:0/1670041375 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5878003460 con 0x561592221290 2024-04-27T03:18:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5774e09640 1 -- 172.21.15.110:0/1670041375 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878003600 con 0x561592221290 2024-04-27T03:18:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5774e09640 1 -- 172.21.15.110:0/1670041375 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5878003920 con 0x561592221290 2024-04-27T03:18:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 -- 172.21.15.110:0/1670041375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 msgr2=0x56159230c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 --2- 172.21.15.110:0/1670041375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x56159230c3d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5878001690 tx=0x7f5878001b50 comp rx=0 tx=0).stop 2024-04-27T03:18:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 -- 172.21.15.110:0/1670041375 shutdown_connections 2024-04-27T03:18:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 --2- 172.21.15.110:0/1670041375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x56159230c3d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 -- 172.21.15.110:0/1670041375 >> 172.21.15.110:0/1670041375 conn(0x56159221cb60 msgr2=0x561592307f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 -- 172.21.15.110:0/1670041375 shutdown_connections 2024-04-27T03:18:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 -- 172.21.15.110:0/1670041375 wait complete. 2024-04-27T03:18:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 Processor -- start 2024-04-27T03:18:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 -- start start 2024-04-27T03:18:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x5615923133e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f58894c5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159222abc0 con 0x561592221290 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x5615923133e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x5615923133e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50466/0 (socket says 172.21.15.110:50466) 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 -- 172.21.15.110:0/3999325198 learned_addr learned my addr 172.21.15.110:0/3999325198 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.637+0000 7f5886fe0640 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561592313920 con 0x561592221290 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5886fe0640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x5615923133e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5878001360 tx=0x7f5878001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5773606640 1 -- 172.21.15.110:0/3999325198 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5878022020 con 0x561592221290 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5773606640 1 -- 172.21.15.110:0/3999325198 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878006840 con 0x561592221290 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561592313bb0 con 0x561592221290 2024-04-27T03:18:06.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5773606640 1 -- 172.21.15.110:0/3999325198 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f587801e800 con 0x561592221290 2024-04-27T03:18:06.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561592316830 con 0x561592221290 2024-04-27T03:18:06.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5773606640 1 -- 172.21.15.110:0/3999325198 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5878025020 con 0x561592221290 2024-04-27T03:18:06.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5773606640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f576c0346a0 0x7f576c036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5773606640 1 -- 172.21.15.110:0/3999325198 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5878049bb0 con 0x561592221290 2024-04-27T03:18:06.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58867df640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f576c0346a0 0x7f576c036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5615922257a0 0x56159231a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159231a930 con 0x5615922257a0 2024-04-27T03:18:06.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58877e1640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5615922257a0 0x56159231a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58867df640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f576c0346a0 0x7f576c036b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58700ae000 tx=0x7f58700a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58877e1640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5615922257a0 0x56159231a310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58877e1640 1 -- 172.21.15.110:0/3999325198 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f58800083f0 con 0x5615922257a0 2024-04-27T03:18:06.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159231b720 0x56159231db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159231e140 con 0x56159231b720 2024-04-27T03:18:06.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5886fe0640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159231b720 0x56159231db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.641+0000 7f5886fe0640 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159231b720 0x56159231db10 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f5886fe0640 1 -- 172.21.15.110:0/3999325198 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f587801b040 con 0x56159231b720 2024-04-27T03:18:06.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159231f9f0 con 0x5615922257a0 2024-04-27T03:18:06.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58877e1640 1 -- 172.21.15.110:0/3999325198 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f58800083f0 con 0x5615922257a0 2024-04-27T03:18:06.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159231ec20 con 0x56159231b720 2024-04-27T03:18:06.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f5886fe0640 1 -- 172.21.15.110:0/3999325198 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f587801b040 con 0x56159231b720 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159231f010 con 0x5615922257a0 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58877e1640 1 -- 172.21.15.110:0/3999325198 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f58800083f0 con 0x5615922257a0 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58894c5980 1 Processor -- start 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58894c5980 1 -- start start 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58894c5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159232e350 0x56159232e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58894c5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561592128310 con 0x56159232e350 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58877e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159232e350 0x56159232e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58877e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159232e350 0x56159232e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50474/0 (socket says 172.21.15.110:50474) 2024-04-27T03:18:06.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58877e1640 1 -- 172.21.15.110:0/1568181024 learned_addr learned my addr 172.21.15.110:0/1568181024 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.645+0000 7f58877e1640 1 -- 172.21.15.110:0/1568181024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159232ecb0 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58877e1640 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159232e350 0x56159232e770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f588000aab0 tx=0x7f588000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5770e01640 1 -- 172.21.15.110:0/1568181024 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5880007d30 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5770e01640 1 -- 172.21.15.110:0/1568181024 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588001d070 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56159232f750 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5770e01640 1 -- 172.21.15.110:0/1568181024 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5880018440 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56159232fc20 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5770e01640 1 -- 172.21.15.110:0/1568181024 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5880013070 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5770e01640 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5760034650 0x7f5760036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5770e01640 1 -- 172.21.15.110:0/1568181024 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f588004b5e0 con 0x56159232e350 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5886fe0640 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5760034650 0x7f5760036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58894c5980 1 Processor -- start 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58894c5980 1 -- start start 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f5886fe0640 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5760034650 0x7f5760036b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x56159231e740 tx=0x7f5878032040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58894c5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x56159233b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58894c5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159230cac0 con 0x56159233b520 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58877e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x56159233b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58877e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x56159233b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50476/0 (socket says 172.21.15.110:50476) 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58877e1640 1 -- 172.21.15.110:0/3410201010 learned_addr learned my addr 172.21.15.110:0/3410201010 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58877e1640 1 -- 172.21.15.110:0/3410201010 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159233bf10 con 0x56159233b520 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f58877e1640 1 --2- 172.21.15.110:0/3410201010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x56159233b940 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5880049930 tx=0x7f5880049b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8caac7a48d893ee2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f576a7fc640 1 -- 172.21.15.110:0/3410201010 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5880047070 con 0x56159233b520 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f576a7fc640 1 -- 172.21.15.110:0/3410201010 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5880022460 con 0x56159233b520 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.649+0000 7f576a7fc640 1 -- 172.21.15.110:0/3410201010 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5880018bc0 con 0x56159233b520 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- 172.21.15.110:0/3410201010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 msgr2=0x56159233b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 --2- 172.21.15.110:0/3410201010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x56159233b940 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5880049930 tx=0x7f5880049b30 comp rx=0 tx=0).stop 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- 172.21.15.110:0/3410201010 shutdown_connections 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 --2- 172.21.15.110:0/3410201010 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x56159233b940 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- 172.21.15.110:0/3410201010 >> 172.21.15.110:0/3410201010 conn(0x561592337810 msgr2=0x5615923358c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- 172.21.15.110:0/3410201010 shutdown_connections 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- 172.21.15.110:0/3410201010 wait complete. 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 Processor -- start 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- start start 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x561592342990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159233c640 con 0x56159233b520 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58877e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x561592342990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58877e1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x561592342990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50484/0 (socket says 172.21.15.110:50484) 2024-04-27T03:18:06.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 learned_addr learned my addr 172.21.15.110:0/1334962756 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561592342ed0 con 0x56159233b520 2024-04-27T03:18:06.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58877e1640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x561592342990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5880049930 tx=0x7f58800498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f5768ff9640 1 -- 172.21.15.110:0/1334962756 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5880031070 con 0x56159233b520 2024-04-27T03:18:06.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561592343160 con 0x56159233b520 2024-04-27T03:18:06.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f5768ff9640 1 -- 172.21.15.110:0/1334962756 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5880016d60 con 0x56159233b520 2024-04-27T03:18:06.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f5768ff9640 1 -- 172.21.15.110:0/1334962756 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5880020820 con 0x56159233b520 2024-04-27T03:18:06.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561592345e40 con 0x56159233b520 2024-04-27T03:18:06.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f5768ff9640 1 -- 172.21.15.110:0/1334962756 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5880034070 con 0x56159233b520 2024-04-27T03:18:06.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f5768ff9640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5750034650 0x7f5750036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f5768ff9640 1 -- 172.21.15.110:0/1334962756 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5880037cb0 con 0x56159233b520 2024-04-27T03:18:06.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f5886fe0640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5750034650 0x7f5750036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.653+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159234e040 0x561592350500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592350c50 con 0x56159234e040 2024-04-27T03:18:06.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58867df640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159234e040 0x561592350500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f5886fe0640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5750034650 0x7f5750036b10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f587801ebd0 tx=0x7f587803b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58867df640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159234e040 0x561592350500 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 realm 2024-04-27T03:18:06.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592351080 con 0x56159234e040 2024-04-27T03:18:06.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:06.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592351950 con 0x56159234e040 2024-04-27T03:18:06.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561592352370 0x5615923547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592354f10 con 0x561592352370 2024-04-27T03:18:06.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58877e1640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561592352370 0x5615923547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58877e1640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561592352370 0x5615923547a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592355760 con 0x56159234e040 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 20 searching for the correct realm 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5615923574d0 0x5615923598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x5615923574d0 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.657+0000 7f5886fe0640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5615923574d0 0x5615923598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f5886fe0640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5615923574d0 0x5615923598c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58877e1640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5880036070 0x7f58800218a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x7f5880036070 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5880036070 0x7f58800218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5880036070 0x7f58800218a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x5615923574d0 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.661+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x7f5880036070 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x7f5880036070 2024-04-27T03:18:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x5615923574d0 2024-04-27T03:18:06.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.665+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x5615923574d0 2024-04-27T03:18:06.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x5615923574d0 2024-04-27T03:18:06.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x7f5880036070 2024-04-27T03:18:06.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x56159234e040 2024-04-27T03:18:06.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x7f5880036070 2024-04-27T03:18:06.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x5615923574d0 2024-04-27T03:18:06.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x7f5880036070 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.669+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x5615923574d0 2024-04-27T03:18:06.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561592359f40 con 0x561592352370 2024-04-27T03:18:06.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:06.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:06.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:06.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:06.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:06.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:06.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592356110 con 0x56159234e040 2024-04-27T03:18:06.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5615923569b0 con 0x561592352370 2024-04-27T03:18:06.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592355d50 con 0x56159234e040 2024-04-27T03:18:06.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:06.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 zone default found 2024-04-27T03:18:06.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 4 Realm: () 2024-04-27T03:18:06.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:06.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 zonegroup default 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.673+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159235a850 con 0x7f5880036070 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159235ca80 con 0x5615923574d0 2024-04-27T03:18:06.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159235ce20 con 0x5615923574d0 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 20 started sync module instance, tier type = 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235bf10 con 0x561592352370 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94650647298864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235bb70 con 0x561592352370 2024-04-27T03:18:06.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.677+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235dc90 con 0x56159234e040 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94650647304160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235e0b0 con 0x56159234e040 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235eb80 con 0x561592352370 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94650647298864] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94650647304160] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94650647307888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235dc00 con 0x561592352370 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235f450 con 0x7f5880036070 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94650647310240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235dc00 con 0x7f5880036070 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.681+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235fef0 con 0x561592352370 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94650647307888] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94650647310240] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94650647311296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159235feb0 con 0x561592352370 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5615923602e0 con 0x5615923574d0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94650647302592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592360b30 con 0x5615923574d0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5615923614f0 con 0x5615923574d0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.685+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94650647311296] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94650647318592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592361460 con 0x5615923574d0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5615923623b0 con 0x7f5880036070 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94650647302592] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'33 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94650647316096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5615923623b0 con 0x7f5880036070 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=0 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=1 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=2 2024-04-27T03:18:06.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=3 2024-04-27T03:18:06.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=4 2024-04-27T03:18:06.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=5 2024-04-27T03:18:06.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94650647318592] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=6 2024-04-27T03:18:06.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94650647316096] v27'34 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 20 add_watcher() i=7 2024-04-27T03:18:06.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5615923623b0 con 0x56159234e040 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.689+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94650647322368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5615923627d0 con 0x56159234e040 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94650647322368] v27'62 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f573d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 5 note: GC not initialized 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f56b7fff640 20 reqs_thread_entry: start 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 20 Filter name: none 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f56b5ffb640 20 reqs_thread_entry: start 2024-04-27T03:18:06.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561592388ad0 con 0x56159234e040 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5615923885b0 con 0x7f5880036070 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 155+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.693+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56159238ed40 con 0x56159234e040 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(64 foo [delete,create,call,call,writefull 0~391] v27'2 uv2 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58894c5980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58894c5980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58894c5980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:06.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify cookie 94650647505360 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56159238ba60 con 0x561592352370 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94650647311296 notify 115964116993 ret 0) ==== 609+0+0 (crc 0 0 0) 0x561592345e40 con 0x561592352370 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(65 notify.4 [notify cookie 94650647505360 out=8b] v0'0 uv33 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f576a7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94650647311296 notifier 4303 bl.length()=567 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f576a7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f576a7fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f576a7fc640 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f574c003eb0 con 0x561592352370 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(66 notify.4 [notify-ack] v0'0 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94650647505360 notify 115964116993 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f5880063020 con 0x561592352370 2024-04-27T03:18:06.716 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":1,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f573d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.697+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94650647322368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x56159234e040 2024-04-27T03:18:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.701+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(67 data_loggenerations_metadata [watch unwatch cookie 94650647322368] v27'63 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.701+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94650647298864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x561592352370 2024-04-27T03:18:06.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.701+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(68 notify.0 [watch unwatch cookie 94650647298864] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.701+0000 7f58894c5980 20 remove_watcher() i=0 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.701+0000 7f58894c5980 2 removed watcher, disabling cache 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.701+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94650647304160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x56159234e040 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(69 notify.1 [watch unwatch cookie 94650647304160] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x56159234e040 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58894c5980 20 remove_watcher() i=1 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94650647307888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x561592352370 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(70 notify.2 [watch unwatch cookie 94650647307888] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58894c5980 20 remove_watcher() i=2 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94650647310240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x7f5880036070 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.3 [watch unwatch cookie 94650647310240] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58894c5980 20 remove_watcher() i=3 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.705+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94650647311296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x561592352370 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.709+0000 7f58877e1640 1 -- 172.21.15.110:0/1334962756 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(72 notify.4 [watch unwatch cookie 94650647311296] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5880033030 con 0x561592352370 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.709+0000 7f58894c5980 20 remove_watcher() i=4 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.709+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94650647302592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x5615923574d0 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.709+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(73 notify.5 [watch unwatch cookie 94650647302592] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.709+0000 7f58894c5980 20 remove_watcher() i=5 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.709+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:74 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94650647318592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x5615923574d0 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f5886fe0640 1 -- 172.21.15.110:0/1334962756 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(74 notify.6 [watch unwatch cookie 94650647318592] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f587801b040 con 0x5615923574d0 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 20 remove_watcher() i=6 2024-04-27T03:18:06.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:75 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94650647316096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561592386800 con 0x7f5880036070 2024-04-27T03:18:06.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(75 notify.7 [watch unwatch cookie 94650647316096] v27'35 uv33 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f58700aaa90 con 0x7f5880036070 2024-04-27T03:18:06.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 20 remove_watcher() i=7 2024-04-27T03:18:06.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561592352370 msgr2=0x5615923547a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561592352370 0x5615923547a0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159234e040 msgr2=0x561592350500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159234e040 0x561592350500 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5880036070 msgr2=0x7f58800218a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5880036070 0x7f58800218a0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5615923574d0 msgr2=0x5615923598c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5615923574d0 0x5615923598c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5750034650 msgr2=0x7f5750036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5750034650 0x7f5750036b10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f587801ebd0 tx=0x7f587803b000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 msgr2=0x561592342990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58867df640 1 -- 172.21.15.110:0/1334962756 reap_dead start 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x561592342990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5880049930 tx=0x7f58800498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 shutdown_connections 2024-04-27T03:18:06.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1334962756 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159233b520 0x561592342990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 >> 172.21.15.110:0/1334962756 conn(0x561592337810 msgr2=0x561592334ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 shutdown_connections 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1334962756 wait complete. 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5760034650 msgr2=0x7f5760036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5760034650 0x7f5760036b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x56159231e740 tx=0x7f5878032040 comp rx=0 tx=0).stop 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159232e350 msgr2=0x56159232e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159232e350 0x56159232e770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f588000aab0 tx=0x7f588000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 shutdown_connections 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5760034650 0x7f5760036b10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/1568181024 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159232e350 0x56159232e770 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 >> 172.21.15.110:0/1568181024 conn(0x561592323160 msgr2=0x561592323c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 shutdown_connections 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/1568181024 wait complete. 2024-04-27T03:18:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159231b720 msgr2=0x56159231db10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159231b720 0x56159231db10 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5615922257a0 msgr2=0x56159231a310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5615922257a0 0x56159231a310 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f576c0346a0 msgr2=0x7f576c036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f576c0346a0 0x7f576c036b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f58700ae000 tx=0x7f58700a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 msgr2=0x5615923133e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x5615923133e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5878001360 tx=0x7f5878001260 comp rx=0 tx=0).stop 2024-04-27T03:18:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 shutdown_connections 2024-04-27T03:18:06.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159231b720 0x56159231db10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5615922257a0 0x56159231a310 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f576c0346a0 0x7f576c036b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 --2- 172.21.15.110:0/3999325198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561592221290 0x5615923133e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 >> 172.21.15.110:0/3999325198 conn(0x56159221cb60 msgr2=0x561592306a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 shutdown_connections 2024-04-27T03:18:06.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.713+0000 7f58894c5980 1 -- 172.21.15.110:0/3999325198 wait complete. 2024-04-27T03:18:06.733 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:06.733 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:06.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:06.733 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:06.811 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:06.811 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:06.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f8777980 1 Processor -- start 2024-04-27T03:18:06.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f8777980 1 -- start start 2024-04-27T03:18:06.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f8777980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c3a17439b0 0x55c3a1745da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f8777980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55c3a15fc0d0 con 0x55c3a173f490 2024-04-27T03:18:06.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f8777980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55c3a15f1430 con 0x55c3a17439b0 2024-04-27T03:18:06.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f5a91640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c3a17439b0 0x55c3a1745da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f5a91640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c3a17439b0 0x55c3a1745da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50500/0 (socket says 172.21.15.110:50500) 2024-04-27T03:18:06.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f5a91640 1 -- 172.21.15.110:0/2634313875 learned_addr learned my addr 172.21.15.110:0/2634313875 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f6292640 1 --1- 172.21.15.110:0/2634313875 >> v1:172.21.15.77:6789/0 conn(0x55c3a173f490 0x55c3a173f870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34614/0 (socket says 172.21.15.110:34614) 2024-04-27T03:18:06.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f5a91640 1 -- 172.21.15.110:0/2634313875 >> v1:172.21.15.77:6789/0 conn(0x55c3a173f490 legacy=0x55c3a173f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.805+0000 7f98f5a91640 1 -- 172.21.15.110:0/2634313875 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a15ea510 con 0x55c3a17439b0 2024-04-27T03:18:06.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f5a91640 1 --2- 172.21.15.110:0/2634313875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a17439b0 0x55c3a1745da0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98e000a5c0 tx=0x7f98e001b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3d122ea7f664d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f5290640 1 -- 172.21.15.110:0/2634313875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e00a6070 con 0x55c3a17439b0 2024-04-27T03:18:06.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f5290640 1 -- 172.21.15.110:0/2634313875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e00af070 con 0x55c3a17439b0 2024-04-27T03:18:06.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f5290640 1 -- 172.21.15.110:0/2634313875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e00aa5c0 con 0x55c3a17439b0 2024-04-27T03:18:06.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f8777980 1 -- 172.21.15.110:0/2634313875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a17439b0 msgr2=0x55c3a1745da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f8777980 1 --2- 172.21.15.110:0/2634313875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a17439b0 0x55c3a1745da0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f98e000a5c0 tx=0x7f98e001b340 comp rx=0 tx=0).stop 2024-04-27T03:18:06.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f8777980 1 -- 172.21.15.110:0/2634313875 shutdown_connections 2024-04-27T03:18:06.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f8777980 1 --2- 172.21.15.110:0/2634313875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a17439b0 0x55c3a1745da0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f8777980 1 -- 172.21.15.110:0/2634313875 >> 172.21.15.110:0/2634313875 conn(0x55c3a16b2b90 msgr2=0x55c3a16b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f8777980 1 -- 172.21.15.110:0/2634313875 shutdown_connections 2024-04-27T03:18:06.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.809+0000 7f98f8777980 1 -- 172.21.15.110:0/2634313875 wait complete. 2024-04-27T03:18:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 Processor -- start 2024-04-27T03:18:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 -- start start 2024-04-27T03:18:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a182a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c3a16402d0 con 0x55c3a173f490 2024-04-27T03:18:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f6292640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a182a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f6292640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a182a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50506/0 (socket says 172.21.15.110:50506) 2024-04-27T03:18:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f6292640 1 -- 172.21.15.110:0/33083111 learned_addr learned my addr 172.21.15.110:0/33083111 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f6292640 1 -- 172.21.15.110:0/33083111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a15c6ae0 con 0x55c3a173f490 2024-04-27T03:18:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f6292640 1 --2- 172.21.15.110:0/33083111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a182a5e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f98e8001690 tx=0x7f98e8001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466ff18524f42e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f97e4608640 1 -- 172.21.15.110:0/33083111 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8003460 con 0x55c3a173f490 2024-04-27T03:18:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f97e4608640 1 -- 172.21.15.110:0/33083111 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e8003600 con 0x55c3a173f490 2024-04-27T03:18:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f97e4608640 1 -- 172.21.15.110:0/33083111 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8003920 con 0x55c3a173f490 2024-04-27T03:18:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 -- 172.21.15.110:0/33083111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 msgr2=0x55c3a182a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 --2- 172.21.15.110:0/33083111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a182a5e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f98e8001690 tx=0x7f98e8001b50 comp rx=0 tx=0).stop 2024-04-27T03:18:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 -- 172.21.15.110:0/33083111 shutdown_connections 2024-04-27T03:18:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 --2- 172.21.15.110:0/33083111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a182a5e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 -- 172.21.15.110:0/33083111 >> 172.21.15.110:0/33083111 conn(0x55c3a16b2b90 msgr2=0x55c3a1826170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 -- 172.21.15.110:0/33083111 shutdown_connections 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.825+0000 7f98f8777980 1 -- 172.21.15.110:0/33083111 wait complete. 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 Processor -- start 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 -- start start 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a18315f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c3a1748e00 con 0x55c3a173f490 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f6292640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a18315f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f6292640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a18315f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50518/0 (socket says 172.21.15.110:50518) 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f6292640 1 -- 172.21.15.110:0/4178833859 learned_addr learned my addr 172.21.15.110:0/4178833859 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f6292640 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a1831b30 con 0x55c3a173f490 2024-04-27T03:18:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f6292640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a18315f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000f90 tx=0x7f98e8001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f97e2e05640 1 -- 172.21.15.110:0/4178833859 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8022020 con 0x55c3a173f490 2024-04-27T03:18:06.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c3a1831dc0 con 0x55c3a173f490 2024-04-27T03:18:06.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f97e2e05640 1 -- 172.21.15.110:0/4178833859 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e8006840 con 0x55c3a173f490 2024-04-27T03:18:06.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f97e2e05640 1 -- 172.21.15.110:0/4178833859 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e801e800 con 0x55c3a173f490 2024-04-27T03:18:06.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3a1834a40 con 0x55c3a173f490 2024-04-27T03:18:06.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f97e2e05640 1 -- 172.21.15.110:0/4178833859 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f98e8025020 con 0x55c3a173f490 2024-04-27T03:18:06.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f97e2e05640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97dc034600 0x7f97dc036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f97e2e05640 1 -- 172.21.15.110:0/4178833859 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f98e8049c20 con 0x55c3a173f490 2024-04-27T03:18:06.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f5a91640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97dc034600 0x7f97dc036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a17439b0 0x55c3a1838520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1838af0 con 0x55c3a17439b0 2024-04-27T03:18:06.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.829+0000 7f98f6a93640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a17439b0 0x55c3a1838520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f5a91640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97dc034600 0x7f97dc036ac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f98e00ae000 tx=0x7f98e00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6a93640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a17439b0 0x55c3a1838520 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6a93640 1 -- 172.21.15.110:0/4178833859 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98f00083f0 con 0x55c3a17439b0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18398c0 0x55c3a183bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a183c2e0 con 0x55c3a18398c0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6292640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18398c0 0x55c3a183bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6292640 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18398c0 0x55c3a183bcb0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6292640 1 -- 172.21.15.110:0/4178833859 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a18398c0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a183db90 con 0x55c3a17439b0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6a93640 1 -- 172.21.15.110:0/4178833859 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f98f00083f0 con 0x55c3a17439b0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a183cdf0 con 0x55c3a18398c0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6292640 1 -- 172.21.15.110:0/4178833859 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a18398c0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a183d1e0 con 0x55c3a17439b0 2024-04-27T03:18:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.833+0000 7f98f6a93640 1 -- 172.21.15.110:0/4178833859 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f98f00083f0 con 0x55c3a17439b0 2024-04-27T03:18:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 Processor -- start 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 -- start start 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a184c500 0x55c3a184c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c3a1644310 con 0x55c3a184c500 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f6a93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a184c500 0x55c3a184c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f6a93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a184c500 0x55c3a184c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50520/0 (socket says 172.21.15.110:50520) 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f6a93640 1 -- 172.21.15.110:0/1443872919 learned_addr learned my addr 172.21.15.110:0/1443872919 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f6a93640 1 -- 172.21.15.110:0/1443872919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a184ce60 con 0x55c3a184c500 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f6a93640 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a184c500 0x55c3a184c920 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f98f000aab0 tx=0x7f98f000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f97dbfff640 1 -- 172.21.15.110:0/1443872919 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f0007d30 con 0x55c3a184c500 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c3a184d900 con 0x55c3a184c500 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f97dbfff640 1 -- 172.21.15.110:0/1443872919 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f001d070 con 0x55c3a184c500 2024-04-27T03:18:06.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f97dbfff640 1 -- 172.21.15.110:0/1443872919 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f0018440 con 0x55c3a184c500 2024-04-27T03:18:06.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3a184ddd0 con 0x55c3a184c500 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f97dbfff640 1 -- 172.21.15.110:0/1443872919 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f98f0013070 con 0x55c3a184c500 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f97dbfff640 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d0034650 0x7f97d0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f97dbfff640 1 -- 172.21.15.110:0/1443872919 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f98f004b530 con 0x55c3a184c500 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f6292640 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d0034650 0x7f97d0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 Processor -- start 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f8777980 1 -- start start 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.837+0000 7f98f6292640 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d0034650 0x7f97d0036b10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55c3a183c910 tx=0x7f98e801c3a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1859b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c3a182acd0 con 0x55c3a1859700 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1859b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1859b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50536/0 (socket says 172.21.15.110:50536) 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 -- 172.21.15.110:0/744766496 learned_addr learned my addr 172.21.15.110:0/744766496 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 -- 172.21.15.110:0/744766496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a185a0f0 con 0x55c3a1859700 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 --2- 172.21.15.110:0/744766496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1859b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98f0049970 tx=0x7f98f003a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd02266d5c819c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f97d9ffb640 1 -- 172.21.15.110:0/744766496 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f003f020 con 0x55c3a1859700 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f97d9ffb640 1 -- 172.21.15.110:0/744766496 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f003b310 con 0x55c3a1859700 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f97d9ffb640 1 -- 172.21.15.110:0/744766496 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f0022460 con 0x55c3a1859700 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- 172.21.15.110:0/744766496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 msgr2=0x55c3a1859b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 --2- 172.21.15.110:0/744766496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1859b20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98f0049970 tx=0x7f98f003a110 comp rx=0 tx=0).stop 2024-04-27T03:18:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- 172.21.15.110:0/744766496 shutdown_connections 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 --2- 172.21.15.110:0/744766496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1859b20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- 172.21.15.110:0/744766496 >> 172.21.15.110:0/744766496 conn(0x55c3a1855a40 msgr2=0x55c3a1853aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- 172.21.15.110:0/744766496 shutdown_connections 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- 172.21.15.110:0/744766496 wait complete. 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 Processor -- start 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- start start 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1860bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c3a185a850 con 0x55c3a1859700 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1860bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1860bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50550/0 (socket says 172.21.15.110:50550) 2024-04-27T03:18:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 learned_addr learned my addr 172.21.15.110:0/3562536390 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a18610f0 con 0x55c3a1859700 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f6a93640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1860bb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55c3a1856140 tx=0x7f98f00498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f97c7fff640 1 -- 172.21.15.110:0/3562536390 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f002f020 con 0x55c3a1859700 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c3a1861380 con 0x55c3a1859700 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f97c7fff640 1 -- 172.21.15.110:0/3562536390 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98f0020c80 con 0x55c3a1859700 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f97c7fff640 1 -- 172.21.15.110:0/3562536390 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98f003f040 con 0x55c3a1859700 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.841+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3a1864060 con 0x55c3a1859700 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f97c7fff640 1 -- 172.21.15.110:0/3562536390 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f98f0032020 con 0x55c3a1859700 2024-04-27T03:18:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f97c7fff640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97c0034650 0x7f97c0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f97c7fff640 1 -- 172.21.15.110:0/3562536390 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f98f004ad50 con 0x55c3a1859700 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f6292640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97c0034650 0x7f97c0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a186c1f0 0x55c3a186e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a186edd0 con 0x55c3a186c1f0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f5a91640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a186c1f0 0x55c3a186e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f6292640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97c0034650 0x7f97c0036b10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f98e804afd0 tx=0x7f98e8043000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f5a91640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a186c1f0 0x55c3a186e680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 realm 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a186f200 con 0x55c3a186c1f0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a186fad0 con 0x55c3a186c1f0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18704f0 0x55c3a18728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1873050 con 0x55c3a18704f0 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f6a93640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18704f0 0x55c3a18728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.845+0000 7f98f6a93640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18704f0 0x55c3a18728e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f8777980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a18738f0 con 0x55c3a186c1f0 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f8777980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f8777980 20 searching for the correct realm 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c3a1875660 0x55c3a1877a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a1875660 2024-04-27T03:18:06.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6292640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c3a1875660 0x55c3a1877a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6292640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c3a1875660 0x55c3a1877a50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6a93640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f98f0034020 0x7f98f00481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x7f98f0034020 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f5a91640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f98f0034020 0x7f98f00481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f5a91640 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f98f0034020 0x7f98f00481f0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.849+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a1875660 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x7f98f0034020 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.853+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x7f98f0034020 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a1875660 2024-04-27T03:18:06.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a1875660 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a1875660 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.857+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x7f98f0034020 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a186c1f0 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x7f98f0034020 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a1875660 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x7f98f0034020 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a1875660 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.861+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1878030 con 0x55c3a18704f0 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:06.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1874240 con 0x55c3a186c1f0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a18746c0 con 0x55c3a18704f0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1874b20 con 0x55c3a186c1f0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 zone default found 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 4 Realm: () 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 zonegroup default 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1873e90 con 0x7f98f0034020 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187a850 con 0x55c3a1875660 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187abf0 con 0x55c3a1875660 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 started sync module instance, tier type = 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:06.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.865+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1879ce0 con 0x55c3a18704f0 2024-04-27T03:18:06.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.869+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.869+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94298717004032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187b490 con 0x55c3a18704f0 2024-04-27T03:18:06.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.869+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187bce0 con 0x55c3a186c1f0 2024-04-27T03:18:06.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.869+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.869+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94298717009040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187c530 con 0x55c3a186c1f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.869+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187cd80 con 0x55c3a18704f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.869+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94298717004032] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94298717009040] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94298717007840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187ca30 con 0x55c3a18704f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187d280 con 0x7f98f0034020 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94298717014848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187d5d0 con 0x7f98f0034020 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187e0f0 con 0x55c3a18704f0 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94298717014848] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94298717007840] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94298717019072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187e0b0 con 0x55c3a18704f0 2024-04-27T03:18:06.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.873+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187e4e0 con 0x55c3a1875660 2024-04-27T03:18:06.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94298717013296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187ed30 con 0x55c3a1875660 2024-04-27T03:18:06.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187f730 con 0x55c3a1875660 2024-04-27T03:18:06.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94298717019072] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94298717013296] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94298717016096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187fb20 con 0x55c3a1875660 2024-04-27T03:18:06.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1880370 con 0x7f98f0034020 2024-04-27T03:18:06.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94298717016096] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'36 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94298717027104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1880760 con 0x7f98f0034020 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 20 add_watcher() i=0 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 20 add_watcher() i=1 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 20 add_watcher() i=2 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 20 add_watcher() i=3 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 20 add_watcher() i=4 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 20 add_watcher() i=5 2024-04-27T03:18:06.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.877+0000 7f98f8777980 20 add_watcher() i=6 2024-04-27T03:18:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94298717027104] v27'37 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f8777980 20 add_watcher() i=7 2024-04-27T03:18:06.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f8777980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:06.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1883790 con 0x55c3a186c1f0 2024-04-27T03:18:06.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94298717028672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1880c70 con 0x55c3a186c1f0 2024-04-27T03:18:06.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94298717028672] v27'64 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f97acfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f8777980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:06.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f8777980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:06.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f98f8777980 5 note: GC not initialized 2024-04-27T03:18:06.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.881+0000 7f97277fe640 20 reqs_thread_entry: start 2024-04-27T03:18:06.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f8777980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:06.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f8777980 20 Filter name: none 2024-04-27T03:18:06.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f97257fa640 20 reqs_thread_entry: start 2024-04-27T03:18:06.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f8777980 20 rados->read ofs=0 len=0 2024-04-27T03:18:06.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c3a18a6af0 con 0x55c3a186c1f0 2024-04-27T03:18:06.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv2 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f8777980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:06.893 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":1,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:06.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f97acfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:06.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94298717028672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a18a58e0 con 0x55c3a186c1f0 2024-04-27T03:18:06.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94298717028672] v27'65 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.885+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94298717004032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a1880110 con 0x55c3a18704f0 2024-04-27T03:18:06.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.889+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94298717004032] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.889+0000 7f98f8777980 20 remove_watcher() i=0 2024-04-27T03:18:06.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.889+0000 7f98f8777980 2 removed watcher, disabling cache 2024-04-27T03:18:06.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.889+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94298717009040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187a100 con 0x55c3a186c1f0 2024-04-27T03:18:06.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.889+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94298717009040] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x55c3a186c1f0 2024-04-27T03:18:06.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.889+0000 7f98f8777980 20 remove_watcher() i=1 2024-04-27T03:18:06.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.889+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94298717007840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187a100 con 0x55c3a18704f0 2024-04-27T03:18:06.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.893+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94298717007840] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.893+0000 7f98f8777980 20 remove_watcher() i=2 2024-04-27T03:18:06.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.893+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94298717014848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187afe0 con 0x7f98f0034020 2024-04-27T03:18:06.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.893+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94298717014848] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.893+0000 7f98f8777980 20 remove_watcher() i=3 2024-04-27T03:18:06.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.893+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94298717019072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187a100 con 0x55c3a18704f0 2024-04-27T03:18:06.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f6a93640 1 -- 172.21.15.110:0/3562536390 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94298717019072] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f0041050 con 0x55c3a18704f0 2024-04-27T03:18:06.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 20 remove_watcher() i=4 2024-04-27T03:18:06.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94298717013296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187cb40 con 0x55c3a1875660 2024-04-27T03:18:06.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94298717013296] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 20 remove_watcher() i=5 2024-04-27T03:18:06.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94298717016096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187a100 con 0x55c3a1875660 2024-04-27T03:18:06.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f6292640 1 -- 172.21.15.110:0/3562536390 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94298717016096] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e801b040 con 0x55c3a1875660 2024-04-27T03:18:06.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 20 remove_watcher() i=6 2024-04-27T03:18:06.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94298717027104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c3a187cb40 con 0x7f98f0034020 2024-04-27T03:18:06.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94298717027104] v27'38 uv36 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98e00aaa10 con 0x7f98f0034020 2024-04-27T03:18:06.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 20 remove_watcher() i=7 2024-04-27T03:18:06.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18704f0 msgr2=0x55c3a18728e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18704f0 0x55c3a18728e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a186c1f0 msgr2=0x55c3a186e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a186c1f0 0x55c3a186e680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f98f0034020 msgr2=0x7f98f00481f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f98f0034020 0x7f98f00481f0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c3a1875660 msgr2=0x55c3a1877a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c3a1875660 0x55c3a1877a50 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97c0034650 msgr2=0x7f97c0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97c0034650 0x7f97c0036b10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f98e804afd0 tx=0x7f98e8043000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f5a91640 1 -- 172.21.15.110:0/3562536390 reap_dead start 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 msgr2=0x55c3a1860bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.897+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1860bb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55c3a1856140 tx=0x7f98f00498d0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 shutdown_connections 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/3562536390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a1859700 0x55c3a1860bb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 >> 172.21.15.110:0/3562536390 conn(0x55c3a1855a40 msgr2=0x55c3a18530a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 shutdown_connections 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/3562536390 wait complete. 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d0034650 msgr2=0x7f97d0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d0034650 0x7f97d0036b10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55c3a183c910 tx=0x7f98e801c3a0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a184c500 msgr2=0x55c3a184c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a184c500 0x55c3a184c920 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f98f000aab0 tx=0x7f98f000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 shutdown_connections 2024-04-27T03:18:06.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d0034650 0x7f97d0036b10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/1443872919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a184c500 0x55c3a184c920 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 >> 172.21.15.110:0/1443872919 conn(0x55c3a1841310 msgr2=0x55c3a1841e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 shutdown_connections 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/1443872919 wait complete. 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18398c0 msgr2=0x55c3a183bcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18398c0 0x55c3a183bcb0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a17439b0 msgr2=0x55c3a1838520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a17439b0 0x55c3a1838520 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97dc034600 msgr2=0x7f97dc036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97dc034600 0x7f97dc036ac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f98e00ae000 tx=0x7f98e00a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 msgr2=0x55c3a18315f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a18315f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000f90 tx=0x7f98e8001260 comp rx=0 tx=0).stop 2024-04-27T03:18:06.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 shutdown_connections 2024-04-27T03:18:06.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c3a18398c0 0x55c3a183bcb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c3a17439b0 0x55c3a1838520 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97dc034600 0x7f97dc036ac0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 --2- 172.21.15.110:0/4178833859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c3a173f490 0x55c3a18315f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 >> 172.21.15.110:0/4178833859 conn(0x55c3a16b2b90 msgr2=0x55c3a1824c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 shutdown_connections 2024-04-27T03:18:06.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.901+0000 7f98f8777980 1 -- 172.21.15.110:0/4178833859 wait complete. 2024-04-27T03:18:06.919 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:06.919 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'enable', '--uid', 'foo'] 2024-04-27T03:18:06.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'enable', '--uid', 'foo'] 2024-04-27T03:18:06.920 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user enable --uid foo 2024-04-27T03:18:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:06.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af7a29980 1 Processor -- start 2024-04-27T03:18:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af7a29980 1 -- start start 2024-04-27T03:18:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af7a29980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d36950a6b0 0x55d36950caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af7a29980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55d3693c60d0 con 0x55d369506150 2024-04-27T03:18:06.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af7a29980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55d3693bb430 con 0x55d36950a6b0 2024-04-27T03:18:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af4d43640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d36950a6b0 0x55d36950caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af5544640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55d369506150 0x55d369506530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34622/0 (socket says 172.21.15.110:34622) 2024-04-27T03:18:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af4d43640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d36950a6b0 0x55d36950caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50552/0 (socket says 172.21.15.110:50552) 2024-04-27T03:18:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af5544640 1 -- 172.21.15.110:0/203858534 learned_addr learned my addr 172.21.15.110:0/203858534 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af4d43640 1 -- 172.21.15.110:0/203858534 >> v1:172.21.15.77:6789/0 conn(0x55d369506150 legacy=0x55d369506530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.985+0000 7f9af4d43640 1 -- 172.21.15.110:0/203858534 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3693b4510 con 0x55d36950a6b0 2024-04-27T03:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af4d43640 1 --2- 172.21.15.110:0/203858534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d36950a6b0 0x55d36950caa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00a610 tx=0x7f9aec01d140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eabc8bd0562c975 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:06.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9ae7fff640 1 -- 172.21.15.110:0/203858534 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec0b1070 con 0x55d36950a6b0 2024-04-27T03:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9ae7fff640 1 -- 172.21.15.110:0/203858534 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aec00a200 con 0x55d36950a6b0 2024-04-27T03:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9ae7fff640 1 -- 172.21.15.110:0/203858534 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec0085a0 con 0x55d36950a6b0 2024-04-27T03:18:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af7a29980 1 -- 172.21.15.110:0/203858534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d36950a6b0 msgr2=0x55d36950caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af7a29980 1 --2- 172.21.15.110:0/203858534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d36950a6b0 0x55d36950caa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00a610 tx=0x7f9aec01d140 comp rx=0 tx=0).stop 2024-04-27T03:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af7a29980 1 -- 172.21.15.110:0/203858534 shutdown_connections 2024-04-27T03:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af7a29980 1 --2- 172.21.15.110:0/203858534 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d36950a6b0 0x55d36950caa0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af7a29980 1 -- 172.21.15.110:0/203858534 >> 172.21.15.110:0/203858534 conn(0x55d369501a20 msgr2=0x55d369503e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af7a29980 1 -- 172.21.15.110:0/203858534 shutdown_connections 2024-04-27T03:18:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:06.989+0000 7f9af7a29980 1 -- 172.21.15.110:0/203858534 wait complete. 2024-04-27T03:18:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af7a29980 1 Processor -- start 2024-04-27T03:18:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af7a29980 1 -- start start 2024-04-27T03:18:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af7a29980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af7a29980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d36940a2d0 con 0x55d369506150 2024-04-27T03:18:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af5544640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af5544640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50568/0 (socket says 172.21.15.110:50568) 2024-04-27T03:18:07.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af5544640 1 -- 172.21.15.110:0/3271333670 learned_addr learned my addr 172.21.15.110:0/3271333670 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af5544640 1 -- 172.21.15.110:0/3271333670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d369390ae0 con 0x55d369506150 2024-04-27T03:18:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f9af5544640 1 --2- 172.21.15.110:0/3271333670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f0390 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80015f0 tx=0x7f9ae8001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2b5f83146d09dc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f99e3606640 1 -- 172.21.15.110:0/3271333670 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ae8004530 con 0x55d369506150 2024-04-27T03:18:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f99e3606640 1 -- 172.21.15.110:0/3271333670 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae80046f0 con 0x55d369506150 2024-04-27T03:18:07.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.005+0000 7f99e3606640 1 -- 172.21.15.110:0/3271333670 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ae8004a90 con 0x55d369506150 2024-04-27T03:18:07.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- 172.21.15.110:0/3271333670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 msgr2=0x55d3695f0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3271333670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f0390 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80015f0 tx=0x7f9ae8001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- 172.21.15.110:0/3271333670 shutdown_connections 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3271333670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f0390 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- 172.21.15.110:0/3271333670 >> 172.21.15.110:0/3271333670 conn(0x55d369501a20 msgr2=0x55d3695ebf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- 172.21.15.110:0/3271333670 shutdown_connections 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- 172.21.15.110:0/3271333670 wait complete. 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 Processor -- start 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- start start 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d36947b720 con 0x55d369506150 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af5544640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af5544640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50584/0 (socket says 172.21.15.110:50584) 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af5544640 1 -- 172.21.15.110:0/3051598169 learned_addr learned my addr 172.21.15.110:0/3051598169 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af5544640 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3695f78e0 con 0x55d369506150 2024-04-27T03:18:07.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af5544640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f73a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55d3695f1220 tx=0x7f9ae8000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f99e1e03640 1 -- 172.21.15.110:0/3051598169 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ae8013760 con 0x55d369506150 2024-04-27T03:18:07.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f99e1e03640 1 -- 172.21.15.110:0/3051598169 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae8013900 con 0x55d369506150 2024-04-27T03:18:07.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d3695f7b70 con 0x55d369506150 2024-04-27T03:18:07.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f99e1e03640 1 -- 172.21.15.110:0/3051598169 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ae8013be0 con 0x55d369506150 2024-04-27T03:18:07.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3695fa7f0 con 0x55d369506150 2024-04-27T03:18:07.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.009+0000 7f99e1e03640 1 -- 172.21.15.110:0/3051598169 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9ae8010070 con 0x55d369506150 2024-04-27T03:18:07.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f99e1e03640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99dc034650 0x7f99dc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f99e1e03640 1 -- 172.21.15.110:0/3051598169 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9ae8018070 con 0x55d369506150 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af4d43640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99dc034650 0x7f99dc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d36950a6b0 0x55d3695fe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d3695fe960 con 0x55d36950a6b0 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5d45640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d36950a6b0 0x55d3695fe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af4d43640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99dc034650 0x7f99dc036b10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0ae000 tx=0x7f9aec0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5d45640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d36950a6b0 0x55d3695fe2f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5d45640 1 -- 172.21.15.110:0/3051598169 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9af00083f0 con 0x55d36950a6b0 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d3695ff730 0x55d369601b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d369602160 con 0x55d3695ff730 2024-04-27T03:18:07.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5544640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d3695ff730 0x55d369601b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5544640 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d3695ff730 0x55d369601b20 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5544640 1 -- 172.21.15.110:0/3051598169 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f9ae801b030 con 0x55d3695ff730 2024-04-27T03:18:07.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d369603a10 con 0x55d36950a6b0 2024-04-27T03:18:07.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5d45640 1 -- 172.21.15.110:0/3051598169 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f9af00083f0 con 0x55d36950a6b0 2024-04-27T03:18:07.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d369602c40 con 0x55d3695ff730 2024-04-27T03:18:07.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af5544640 1 -- 172.21.15.110:0/3051598169 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f9ae801b030 con 0x55d3695ff730 2024-04-27T03:18:07.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.013+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d369603030 con 0x55d36950a6b0 2024-04-27T03:18:07.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af5d45640 1 -- 172.21.15.110:0/3051598169 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f9af00083f0 con 0x55d36950a6b0 2024-04-27T03:18:07.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af7a29980 1 Processor -- start 2024-04-27T03:18:07.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af7a29980 1 -- start start 2024-04-27T03:18:07.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af7a29980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369612310 0x55d369612730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af7a29980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d36940e310 con 0x55d369612310 2024-04-27T03:18:07.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af5d45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369612310 0x55d369612730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af5d45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369612310 0x55d369612730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50594/0 (socket says 172.21.15.110:50594) 2024-04-27T03:18:07.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af5d45640 1 -- 172.21.15.110:0/1352387696 learned_addr learned my addr 172.21.15.110:0/1352387696 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af5d45640 1 -- 172.21.15.110:0/1352387696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d369612c70 con 0x55d369612310 2024-04-27T03:18:07.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af5d45640 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369612310 0x55d369612730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9af000aab0 tx=0x7f9af000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f99daffd640 1 -- 172.21.15.110:0/1352387696 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9af0007d30 con 0x55d369612310 2024-04-27T03:18:07.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d369613710 con 0x55d369612310 2024-04-27T03:18:07.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f99daffd640 1 -- 172.21.15.110:0/1352387696 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9af001d070 con 0x55d369612310 2024-04-27T03:18:07.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f99daffd640 1 -- 172.21.15.110:0/1352387696 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9af0018440 con 0x55d369612310 2024-04-27T03:18:07.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d369613be0 con 0x55d369612310 2024-04-27T03:18:07.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f99daffd640 1 -- 172.21.15.110:0/1352387696 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9af0013070 con 0x55d369612310 2024-04-27T03:18:07.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f99daffd640 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99cc034650 0x7f99cc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.017+0000 7f99daffd640 1 -- 172.21.15.110:0/1352387696 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9af004b520 con 0x55d369612310 2024-04-27T03:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5544640 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99cc034650 0x7f99cc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 Processor -- start 2024-04-27T03:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- start start 2024-04-27T03:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5544640 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99cc034650 0x7f99cc036b10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55d369602760 tx=0x7f9ae8010440 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d369623d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d3695f0ae0 con 0x55d369623920 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d369623d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d369623d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50610/0 (socket says 172.21.15.110:50610) 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 -- 172.21.15.110:0/981443211 learned_addr learned my addr 172.21.15.110:0/981443211 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 -- 172.21.15.110:0/981443211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d369624310 con 0x55d369623920 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 --2- 172.21.15.110:0/981443211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d369623d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9af0049930 tx=0x7f9af0018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbc89f2bf17f9d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f99d8ff9640 1 -- 172.21.15.110:0/981443211 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9af0047070 con 0x55d369623920 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f99d8ff9640 1 -- 172.21.15.110:0/981443211 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9af0018bc0 con 0x55d369623920 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f99d8ff9640 1 -- 172.21.15.110:0/981443211 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9af0020360 con 0x55d369623920 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- 172.21.15.110:0/981443211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 msgr2=0x55d369623d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 --2- 172.21.15.110:0/981443211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d369623d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9af0049930 tx=0x7f9af0018950 comp rx=0 tx=0).stop 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- 172.21.15.110:0/981443211 shutdown_connections 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 --2- 172.21.15.110:0/981443211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d369623d40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- 172.21.15.110:0/981443211 >> 172.21.15.110:0/981443211 conn(0x55d36961b850 msgr2=0x55d3696198b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- 172.21.15.110:0/981443211 shutdown_connections 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- 172.21.15.110:0/981443211 wait complete. 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 Processor -- start 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- start start 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d36962ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af7a29980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d369624a70 con 0x55d369623920 2024-04-27T03:18:07.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d36962ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d36962ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50614/0 (socket says 172.21.15.110:50614) 2024-04-27T03:18:07.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.021+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 learned_addr learned my addr 172.21.15.110:0/2378831552 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36962b320 con 0x55d369623920 2024-04-27T03:18:07.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af5d45640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d36962ade0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9af00168b0 tx=0x7f9af00498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f99caffd640 1 -- 172.21.15.110:0/2378831552 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9af0031070 con 0x55d369623920 2024-04-27T03:18:07.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d36962b5b0 con 0x55d369623920 2024-04-27T03:18:07.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f99caffd640 1 -- 172.21.15.110:0/2378831552 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9af0020900 con 0x55d369623920 2024-04-27T03:18:07.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36962e290 con 0x55d369623920 2024-04-27T03:18:07.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f99caffd640 1 -- 172.21.15.110:0/2378831552 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9af0049a70 con 0x55d369623920 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f99caffd640 1 -- 172.21.15.110:0/2378831552 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9af0034070 con 0x55d369623920 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f99caffd640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99c0034650 0x7f99c0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f99caffd640 1 -- 172.21.15.110:0/2378831552 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9af003c380 con 0x55d369623920 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af5544640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99c0034650 0x7f99c0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d3696364b0 0x55d369638950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d3696390a0 con 0x55d3696364b0 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af4d43640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d3696364b0 0x55d369638950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af5544640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99c0034650 0x7f99c0036b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8048440 tx=0x7f9ae8049000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af4d43640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d3696364b0 0x55d369638950 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 realm 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d3696394d0 con 0x55d3696364b0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d369639da0 con 0x55d3696364b0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.025+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d36963a7c0 0x55d36963cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36963d360 con 0x55d36963a7c0 2024-04-27T03:18:07.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af5d45640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d36963a7c0 0x55d36963cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af5d45640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d36963a7c0 0x55d36963cbf0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36963dbb0 con 0x55d3696364b0 2024-04-27T03:18:07.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 20 searching for the correct realm 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d36963f920 0x55d369641d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55d369642340 con 0x55d36963f920 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af5544640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d36963f920 0x55d369641d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af5544640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d36963f920 0x55d369641d10 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.029+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55d369642340 con 0x55d36963a7c0 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5d45640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9af0036070 0x7f9af00389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55d369642340 con 0x7f9af0036070 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9af0036070 0x7f9af00389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9af0036070 0x7f9af00389c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55d369642340 con 0x55d3696364b0 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55d369642340 con 0x55d3696364b0 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963f920 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x7f9af0036070 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.033+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x7f9af0036070 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963f920 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963f920 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963f920 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.037+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x7f9af0036070 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d3696364b0 2024-04-27T03:18:07.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x7f9af0036070 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963f920 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x7f9af0036070 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963f920 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.041+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f9aec0cf040 con 0x55d36963a7c0 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36963e560 con 0x55d3696364b0 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36963e950 con 0x55d36963a7c0 2024-04-27T03:18:07.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36963ed70 con 0x55d3696364b0 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 zone default found 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 4 Realm: () 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 zonegroup default 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d3696427c0 con 0x7f9af0036070 2024-04-27T03:18:07.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d369644200 con 0x55d36963f920 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d3696446e0 con 0x55d36963f920 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 started sync module instance, tier type = 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:07.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.045+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369643640 con 0x55d36963a7c0 2024-04-27T03:18:07.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94366494636592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369644ff0 con 0x55d36963a7c0 2024-04-27T03:18:07.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369645840 con 0x55d3696364b0 2024-04-27T03:18:07.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94366494642160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369645900 con 0x55d3696364b0 2024-04-27T03:18:07.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369646420 con 0x55d36963a7c0 2024-04-27T03:18:07.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94366494636592] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.049+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94366494642160] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94366494646000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d3696463e0 con 0x55d36963a7c0 2024-04-27T03:18:07.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369646cc0 con 0x7f9af0036070 2024-04-27T03:18:07.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94366494648336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d3696463e0 con 0x7f9af0036070 2024-04-27T03:18:07.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369647700 con 0x55d36963a7c0 2024-04-27T03:18:07.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94366494646000] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94366494648336] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94366494649392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369647250 con 0x55d36963a7c0 2024-04-27T03:18:07.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.053+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369647fa0 con 0x55d36963f920 2024-04-27T03:18:07.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94366494653168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369647250 con 0x55d36963f920 2024-04-27T03:18:07.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369648d20 con 0x55d36963f920 2024-04-27T03:18:07.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94366494649392] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94366494653168] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94366494640960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369648ce0 con 0x55d36963f920 2024-04-27T03:18:07.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d3696495c0 con 0x7f9af0036070 2024-04-27T03:18:07.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94366494640960] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'39 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94366494655296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369648ce0 con 0x7f9af0036070 2024-04-27T03:18:07.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 20 add_watcher() i=0 2024-04-27T03:18:07.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 20 add_watcher() i=1 2024-04-27T03:18:07.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 20 add_watcher() i=2 2024-04-27T03:18:07.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 20 add_watcher() i=3 2024-04-27T03:18:07.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 20 add_watcher() i=4 2024-04-27T03:18:07.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 20 add_watcher() i=5 2024-04-27T03:18:07.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.057+0000 7f9af7a29980 20 add_watcher() i=6 2024-04-27T03:18:07.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94366494655296] v27'40 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 20 add_watcher() i=7 2024-04-27T03:18:07.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:07.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36964c710 con 0x55d3696364b0 2024-04-27T03:18:07.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94366494658832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369649bd0 con 0x55d3696364b0 2024-04-27T03:18:07.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94366494658832] v27'66 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f99ab7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:07.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:07.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 5 note: GC not initialized 2024-04-27T03:18:07.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f99aa7e4640 20 reqs_thread_entry: start 2024-04-27T03:18:07.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:07.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 20 Filter name: none 2024-04-27T03:18:07.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f991bfff640 20 reqs_thread_entry: start 2024-04-27T03:18:07.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:07.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36966fe80 con 0x55d3696364b0 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv2 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d36966f9c0 con 0x7f9af0036070 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.061+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 155+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d369676130 con 0x55d3696364b0 2024-04-27T03:18:07.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(64 foo [delete,create,call,call,writefull 0~391] v27'3 uv3 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af7a29980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af7a29980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af7a29980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify cookie 94366494842304 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55d369672e50 con 0x55d36963a7c0 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94366494649392 notify 115964116994 ret 0) ==== 609+0+0 (crc 0 0 0) 0x55d36962e290 con 0x55d36963a7c0 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f99d8ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94366494649392 notifier 4317 bl.length()=567 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f99d8ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(65 notify.4 [notify cookie 94366494842304 out=8b] v0'0 uv39 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f99d8ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f99d8ff9640 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f99b8003eb0 con 0x55d36963a7c0 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(66 notify.4 [notify-ack] v0'0 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94366494842304 notify 115964116994 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55d36962b5b0 con 0x55d36963a7c0 2024-04-27T03:18:07.075 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:07.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f99ab7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.065+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94366494658832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x55d3696364b0 2024-04-27T03:18:07.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.069+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(67 data_loggenerations_metadata [watch unwatch cookie 94366494658832] v27'67 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.069+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94366494636592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x55d36963a7c0 2024-04-27T03:18:07.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(68 notify.0 [watch unwatch cookie 94366494636592] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af7a29980 20 remove_watcher() i=0 2024-04-27T03:18:07.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af7a29980 2 removed watcher, disabling cache 2024-04-27T03:18:07.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94366494642160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x55d3696364b0 2024-04-27T03:18:07.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(69 notify.1 [watch unwatch cookie 94366494642160] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x55d3696364b0 2024-04-27T03:18:07.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af7a29980 20 remove_watcher() i=1 2024-04-27T03:18:07.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94366494646000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x55d36963a7c0 2024-04-27T03:18:07.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(70 notify.2 [watch unwatch cookie 94366494646000] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af7a29980 20 remove_watcher() i=2 2024-04-27T03:18:07.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.073+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94366494648336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x7f9af0036070 2024-04-27T03:18:07.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.3 [watch unwatch cookie 94366494648336] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af7a29980 20 remove_watcher() i=3 2024-04-27T03:18:07.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94366494649392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x55d36963a7c0 2024-04-27T03:18:07.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af5d45640 1 -- 172.21.15.110:0/2378831552 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(72 notify.4 [watch unwatch cookie 94366494649392] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9af0033030 con 0x55d36963a7c0 2024-04-27T03:18:07.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af7a29980 20 remove_watcher() i=4 2024-04-27T03:18:07.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94366494653168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x55d36963f920 2024-04-27T03:18:07.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(73 notify.5 [watch unwatch cookie 94366494653168] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af7a29980 20 remove_watcher() i=5 2024-04-27T03:18:07.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.077+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:74 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94366494640960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x55d36963f920 2024-04-27T03:18:07.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af5544640 1 -- 172.21.15.110:0/2378831552 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(74 notify.6 [watch unwatch cookie 94366494640960] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9ae8050040 con 0x55d36963f920 2024-04-27T03:18:07.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 20 remove_watcher() i=6 2024-04-27T03:18:07.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:75 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94366494655296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55d36966dbb0 con 0x7f9af0036070 2024-04-27T03:18:07.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(75 notify.7 [watch unwatch cookie 94366494655296] v27'41 uv39 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9aec0b4040 con 0x7f9af0036070 2024-04-27T03:18:07.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 20 remove_watcher() i=7 2024-04-27T03:18:07.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d36963a7c0 msgr2=0x55d36963cbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d36963a7c0 0x55d36963cbf0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d3696364b0 msgr2=0x55d369638950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d3696364b0 0x55d369638950 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9af0036070 msgr2=0x7f9af00389c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9af0036070 0x7f9af00389c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d36963f920 msgr2=0x55d369641d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d36963f920 0x55d369641d10 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99c0034650 msgr2=0x7f99c0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99c0034650 0x7f99c0036b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8048440 tx=0x7f9ae8049000 comp rx=0 tx=0).stop 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 msgr2=0x55d36962ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af4d43640 1 -- 172.21.15.110:0/2378831552 reap_dead start 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d36962ade0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9af00168b0 tx=0x7f9af00498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 shutdown_connections 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/2378831552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369623920 0x55d36962ade0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 >> 172.21.15.110:0/2378831552 conn(0x55d36961b850 msgr2=0x55d369618eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 shutdown_connections 2024-04-27T03:18:07.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/2378831552 wait complete. 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99cc034650 msgr2=0x7f99cc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99cc034650 0x7f99cc036b10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55d369602760 tx=0x7f9ae8010440 comp rx=0 tx=0).stop 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369612310 msgr2=0x55d369612730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369612310 0x55d369612730 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9af000aab0 tx=0x7f9af000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 shutdown_connections 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99cc034650 0x7f99cc036b10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/1352387696 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369612310 0x55d369612730 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 >> 172.21.15.110:0/1352387696 conn(0x55d369607120 msgr2=0x55d369607c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 shutdown_connections 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/1352387696 wait complete. 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d3695ff730 msgr2=0x55d369601b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d3695ff730 0x55d369601b20 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d36950a6b0 msgr2=0x55d3695fe2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d36950a6b0 0x55d3695fe2f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99dc034650 msgr2=0x7f99dc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99dc034650 0x7f99dc036b10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0ae000 tx=0x7f9aec0a7000 comp rx=0 tx=0).stop 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 msgr2=0x55d3695f73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f73a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55d3695f1220 tx=0x7f9ae8000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 shutdown_connections 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d3695ff730 0x55d369601b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d36950a6b0 0x55d3695fe2f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f99dc034650 0x7f99dc036b10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 --2- 172.21.15.110:0/3051598169 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d369506150 0x55d3695f73a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 >> 172.21.15.110:0/3051598169 conn(0x55d369501a20 msgr2=0x55d3695eaa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 shutdown_connections 2024-04-27T03:18:07.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.081+0000 7f9af7a29980 1 -- 172.21.15.110:0/3051598169 wait complete. 2024-04-27T03:18:07.099 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:07.099 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:07.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:07.099 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:07.176 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:07.176 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:07.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dadb3b980 1 Processor -- start 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dadb3b980 1 -- start start 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dadb3b980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b1440b850 0x556b1449c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dadb3b980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x556b143560d0 con 0x556b1449c880 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dadb3b980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x556b1434b430 con 0x556b1440b850 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dab656640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b1440b850 0x556b1449c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7daae55640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x556b1449c880 0x556b1449ec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34632/0 (socket says 172.21.15.110:34632) 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dab656640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b1440b850 0x556b1449c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50630/0 (socket says 172.21.15.110:50630) 2024-04-27T03:18:07.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7daae55640 1 -- 172.21.15.110:0/1168763528 learned_addr learned my addr 172.21.15.110:0/1168763528 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:07.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dab656640 1 -- 172.21.15.110:0/1168763528 >> v1:172.21.15.77:6789/0 conn(0x556b1449c880 legacy=0x556b1449ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.169+0000 7f7dab656640 1 -- 172.21.15.110:0/1168763528 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b14344510 con 0x556b1440b850 2024-04-27T03:18:07.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dab656640 1 --2- 172.21.15.110:0/1168763528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1449c340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00a420 tx=0x7f7d9c01afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a802fbb517ad1ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7daa654640 1 -- 172.21.15.110:0/1168763528 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d9c0af070 con 0x556b1440b850 2024-04-27T03:18:07.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7daa654640 1 -- 172.21.15.110:0/1168763528 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9c0aa530 con 0x556b1440b850 2024-04-27T03:18:07.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7daa654640 1 -- 172.21.15.110:0/1168763528 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d9c0aa850 con 0x556b1440b850 2024-04-27T03:18:07.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1168763528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 msgr2=0x556b1449c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1168763528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1449c340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c00a420 tx=0x7f7d9c01afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1168763528 shutdown_connections 2024-04-27T03:18:07.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1168763528 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1449c340 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1168763528 >> 172.21.15.110:0/1168763528 conn(0x556b14491a20 msgr2=0x556b14493e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1168763528 shutdown_connections 2024-04-27T03:18:07.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.173+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1168763528 wait complete. 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.185+0000 7f7dadb3b980 1 Processor -- start 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.185+0000 7f7dadb3b980 1 -- start start 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.185+0000 7f7dadb3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b14584620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.185+0000 7f7dadb3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b1439a2d0 con 0x556b1440b850 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.185+0000 7f7dab656640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b14584620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.185+0000 7f7dab656640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b14584620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50646/0 (socket says 172.21.15.110:50646) 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.185+0000 7f7dab656640 1 -- 172.21.15.110:0/1214980168 learned_addr learned my addr 172.21.15.110:0/1214980168 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dab656640 1 -- 172.21.15.110:0/1214980168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b14320ae0 con 0x556b1440b850 2024-04-27T03:18:07.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dab656640 1 --2- 172.21.15.110:0/1214980168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b14584620 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0ad040 tx=0x7f7d9c0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93cfd8e396137f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7c9960a640 1 -- 172.21.15.110:0/1214980168 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d9c0af050 con 0x556b1440b850 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7c9960a640 1 -- 172.21.15.110:0/1214980168 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9c0b0040 con 0x556b1440b850 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7c9960a640 1 -- 172.21.15.110:0/1214980168 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d9c0aaca0 con 0x556b1440b850 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1214980168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 msgr2=0x556b14584620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1214980168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b14584620 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0ad040 tx=0x7f7d9c0033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1214980168 shutdown_connections 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1214980168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b14584620 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1214980168 >> 172.21.15.110:0/1214980168 conn(0x556b14491a20 msgr2=0x556b145801b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1214980168 shutdown_connections 2024-04-27T03:18:07.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1214980168 wait complete. 2024-04-27T03:18:07.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 Processor -- start 2024-04-27T03:18:07.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 -- start start 2024-04-27T03:18:07.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1458b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dadb3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b144a2df0 con 0x556b1440b850 2024-04-27T03:18:07.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dab656640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1458b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dab656640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1458b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50648/0 (socket says 172.21.15.110:50648) 2024-04-27T03:18:07.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dab656640 1 -- 172.21.15.110:0/76020137 learned_addr learned my addr 172.21.15.110:0/76020137 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dab656640 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b1458ba80 con 0x556b1440b850 2024-04-27T03:18:07.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7dab656640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1458b540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c018c00 tx=0x7f7d9c019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7c97e07640 1 -- 172.21.15.110:0/76020137 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d9c0c5070 con 0x556b1440b850 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7c97e07640 1 -- 172.21.15.110:0/76020137 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9c0b6d60 con 0x556b1440b850 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.189+0000 7f7c97e07640 1 -- 172.21.15.110:0/76020137 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d9c0c15f0 con 0x556b1440b850 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b1458bd10 con 0x556b1440b850 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b1458e940 con 0x556b1440b850 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7c97e07640 1 -- 172.21.15.110:0/76020137 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7d9c0c8070 con 0x556b1440b850 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7c97e07640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c900346a0 0x7f7c90036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7c97e07640 1 -- 172.21.15.110:0/76020137 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7d9c0bb070 con 0x556b1440b850 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7daae55640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c900346a0 0x7f7c90036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b1449c880 0x556b14592420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145929b0 con 0x556b1449c880 2024-04-27T03:18:07.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dabe57640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b1449c880 0x556b14592420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7daae55640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c900346a0 0x7f7c90036b60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7d94001da0 tx=0x7f7d94001300 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dabe57640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b1449c880 0x556b14592420 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dabe57640 1 -- 172.21.15.110:0/76020137 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7da40083f0 con 0x556b1449c880 2024-04-27T03:18:07.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b14593760 0x556b14595b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145961b0 con 0x556b14593760 2024-04-27T03:18:07.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dab656640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b14593760 0x556b14595b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.193+0000 7f7dab656640 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b14593760 0x556b14595b50 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dab656640 1 -- 172.21.15.110:0/76020137 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b14593760 2024-04-27T03:18:07.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b14597a60 con 0x556b1449c880 2024-04-27T03:18:07.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dabe57640 1 -- 172.21.15.110:0/76020137 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f7da40083f0 con 0x556b1449c880 2024-04-27T03:18:07.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b14596c90 con 0x556b14593760 2024-04-27T03:18:07.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dab656640 1 -- 172.21.15.110:0/76020137 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b14593760 2024-04-27T03:18:07.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b14597080 con 0x556b1449c880 2024-04-27T03:18:07.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dabe57640 1 -- 172.21.15.110:0/76020137 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f7da40083f0 con 0x556b1449c880 2024-04-27T03:18:07.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 Processor -- start 2024-04-27T03:18:07.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 -- start start 2024-04-27T03:18:07.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145a63a0 0x556b145a67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b1439e310 con 0x556b145a63a0 2024-04-27T03:18:07.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dabe57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145a63a0 0x556b145a67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dabe57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145a63a0 0x556b145a67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50654/0 (socket says 172.21.15.110:50654) 2024-04-27T03:18:07.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dabe57640 1 -- 172.21.15.110:0/1531904121 learned_addr learned my addr 172.21.15.110:0/1531904121 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dabe57640 1 -- 172.21.15.110:0/1531904121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b145a6d00 con 0x556b145a63a0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dabe57640 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145a63a0 0x556b145a67c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7da400aab0 tx=0x7f7da400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7c95602640 1 -- 172.21.15.110:0/1531904121 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da4007d30 con 0x556b145a63a0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b145a77a0 con 0x556b145a63a0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7c95602640 1 -- 172.21.15.110:0/1531904121 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da401d070 con 0x556b145a63a0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7c95602640 1 -- 172.21.15.110:0/1531904121 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da4018440 con 0x556b145a63a0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.197+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b145a7c70 con 0x556b145a63a0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7c95602640 1 -- 172.21.15.110:0/1531904121 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7da4013070 con 0x556b145a63a0 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7c95602640 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c84034650 0x7f7c84036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7c95602640 1 -- 172.21.15.110:0/1531904121 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7da404b530 con 0x556b145a63a0 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dab656640 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c84034650 0x7f7c84036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 Processor -- start 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- start start 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b14584cb0 con 0x556b145b3580 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dab656640 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c84034650 0x7f7c84036b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x556b145967b0 tx=0x7f7d9c0c83c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dabe57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dabe57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145b39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50660/0 (socket says 172.21.15.110:50660) 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dabe57640 1 -- 172.21.15.110:0/829942807 learned_addr learned my addr 172.21.15.110:0/829942807 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dabe57640 1 -- 172.21.15.110:0/829942807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b145b3f70 con 0x556b145b3580 2024-04-27T03:18:07.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dabe57640 1 --2- 172.21.15.110:0/829942807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145b39a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7da4049970 tx=0x7f7da403a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec646c39b86c79bc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7c8effd640 1 -- 172.21.15.110:0/829942807 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da403f020 con 0x556b145b3580 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7c8effd640 1 -- 172.21.15.110:0/829942807 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da403b310 con 0x556b145b3580 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7c8effd640 1 -- 172.21.15.110:0/829942807 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da4022460 con 0x556b145b3580 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- 172.21.15.110:0/829942807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 msgr2=0x556b145b39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/829942807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145b39a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7da4049970 tx=0x7f7da403a110 comp rx=0 tx=0).stop 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- 172.21.15.110:0/829942807 shutdown_connections 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/829942807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145b39a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- 172.21.15.110:0/829942807 >> 172.21.15.110:0/829942807 conn(0x556b145af8c0 msgr2=0x556b145ad920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- 172.21.15.110:0/829942807 shutdown_connections 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- 172.21.15.110:0/829942807 wait complete. 2024-04-27T03:18:07.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 Processor -- start 2024-04-27T03:18:07.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.201+0000 7f7dadb3b980 1 -- start start 2024-04-27T03:18:07.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145baa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b145b4690 con 0x556b145b3580 2024-04-27T03:18:07.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dabe57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145baa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dabe57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145baa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50662/0 (socket says 172.21.15.110:50662) 2024-04-27T03:18:07.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 learned_addr learned my addr 172.21.15.110:0/3335086422 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b145baf80 con 0x556b145b3580 2024-04-27T03:18:07.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dabe57640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145baa40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x556b145affc0 tx=0x7f7da40498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7c8d7fa640 1 -- 172.21.15.110:0/3335086422 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da402f020 con 0x556b145b3580 2024-04-27T03:18:07.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b145bb210 con 0x556b145b3580 2024-04-27T03:18:07.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b145bdef0 con 0x556b145b3580 2024-04-27T03:18:07.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7c8d7fa640 1 -- 172.21.15.110:0/3335086422 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da4020c80 con 0x556b145b3580 2024-04-27T03:18:07.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7c8d7fa640 1 -- 172.21.15.110:0/3335086422 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7da403f040 con 0x556b145b3580 2024-04-27T03:18:07.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7c8d7fa640 1 -- 172.21.15.110:0/3335086422 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7da4032020 con 0x556b145b3580 2024-04-27T03:18:07.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7c8d7fa640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c74034650 0x7f7c74036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7c8d7fa640 1 -- 172.21.15.110:0/3335086422 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7da4035020 con 0x556b145b3580 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dab656640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c74034650 0x7f7c74036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b145c60c0 0x556b145c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145c8cb0 con 0x556b145c60c0 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7daae55640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b145c60c0 0x556b145c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dab656640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c74034650 0x7f7c74036b10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0c8fd0 tx=0x7f7d9c0df000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7daae55640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b145c60c0 0x556b145c8560 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7d9400e640 con 0x556b145c60c0 2024-04-27T03:18:07.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 20 realm 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.205+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145c5590 con 0x556b145c60c0 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7d9400e640 con 0x556b145c60c0 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145c9780 con 0x556b145c60c0 2024-04-27T03:18:07.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7d9400e640 con 0x556b145c60c0 2024-04-27T03:18:07.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b145ca380 0x556b145cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145ccfb0 con 0x556b145ca380 2024-04-27T03:18:07.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dabe57640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b145ca380 0x556b145cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dabe57640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b145ca380 0x556b145cc840 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145cd850 con 0x556b145c60c0 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f7d9400e640 con 0x556b145c60c0 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 20 searching for the correct realm 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b145cf5c0 0x556b145d19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145cf5c0 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dab656640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b145cf5c0 0x556b145d19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dab656640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b145cf5c0 0x556b145d19b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.209+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dabe57640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7da4034020 0x7f7da40137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x7f7da4034020 2024-04-27T03:18:07.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7da4034020 0x7f7da40137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7da4034020 0x7f7da40137e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9400e640 con 0x7f7da4034020 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9400e640 con 0x556b145c60c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145cf5c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x7f7da4034020 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x7f7da4034020 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.213+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x7f7da4034020 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f7d94029040 con 0x7f7da4034020 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145cf5c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145cf5c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145cf5c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x7f7da4034020 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x7f7da4034020 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.217+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145c60c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x7f7da4034020 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x7f7da4034020 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145cf5c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x7f7da4034020 2024-04-27T03:18:07.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d94029040 con 0x7f7da4034020 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145cf5c0 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b145d1fc0 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.221+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145ce1d0 con 0x556b145c60c0 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7d94029040 con 0x556b145c60c0 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145ce650 con 0x556b145ca380 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145ceab0 con 0x556b145c60c0 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 zone default found 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 4 Realm: () 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:07.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:07.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 zonegroup default 2024-04-27T03:18:07.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145d2440 con 0x7f7da4034020 2024-04-27T03:18:07.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x7f7da4034020 2024-04-27T03:18:07.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145cde00 con 0x556b145cf5c0 2024-04-27T03:18:07.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145d43e0 con 0x556b145cf5c0 2024-04-27T03:18:07.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 started sync module instance, tier type = 2024-04-27T03:18:07.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:07.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.225+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d3760 con 0x556b145ca380 2024-04-27T03:18:07.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93918391515984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d4cf0 con 0x556b145ca380 2024-04-27T03:18:07.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d5540 con 0x556b145c60c0 2024-04-27T03:18:07.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93918391520496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d5700 con 0x556b145c60c0 2024-04-27T03:18:07.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d6220 con 0x556b145ca380 2024-04-27T03:18:07.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93918391520496] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93918391515984] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93918391519296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d61e0 con 0x556b145ca380 2024-04-27T03:18:07.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.229+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d6ac0 con 0x7f7da4034020 2024-04-27T03:18:07.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x7f7da4034020 2024-04-27T03:18:07.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93918391526928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d61e0 con 0x7f7da4034020 2024-04-27T03:18:07.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d7520 con 0x556b145ca380 2024-04-27T03:18:07.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93918391519296] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93918391526928] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x7f7da4034020 2024-04-27T03:18:07.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93918391527984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d7520 con 0x556b145ca380 2024-04-27T03:18:07.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d7940 con 0x556b145cf5c0 2024-04-27T03:18:07.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93918391524592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d8190 con 0x556b145cf5c0 2024-04-27T03:18:07.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.233+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d8b50 con 0x556b145cf5c0 2024-04-27T03:18:07.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93918391527984] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93918391529552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d8b50 con 0x556b145cf5c0 2024-04-27T03:18:07.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d9600 con 0x7f7da4034020 2024-04-27T03:18:07.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 93918391524592] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93918391529552] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'42 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x7f7da4034020 2024-04-27T03:18:07.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93918391532768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d86a0 con 0x7f7da4034020 2024-04-27T03:18:07.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=0 2024-04-27T03:18:07.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=1 2024-04-27T03:18:07.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=2 2024-04-27T03:18:07.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=3 2024-04-27T03:18:07.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=4 2024-04-27T03:18:07.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=5 2024-04-27T03:18:07.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=6 2024-04-27T03:18:07.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93918391532768] v27'43 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x7f7da4034020 2024-04-27T03:18:07.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 20 add_watcher() i=7 2024-04-27T03:18:07.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:07.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145d86a0 con 0x556b145c60c0 2024-04-27T03:18:07.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.237+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93918391537888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d86a0 con 0x556b145c60c0 2024-04-27T03:18:07.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93918391537888] v27'68 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7c61fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 5 note: GC not initialized 2024-04-27T03:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7c60fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:07.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:07.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 20 Filter name: none 2024-04-27T03:18:07.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7bda7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:07.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b145ff880 con 0x556b145c60c0 2024-04-27T03:18:07.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv3 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:07.265 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:07.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7c61fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.241+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93918391537888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145fe670 con 0x556b145c60c0 2024-04-27T03:18:07.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 93918391537888] v27'69 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93918391515984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d90e0 con 0x556b145ca380 2024-04-27T03:18:07.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 93918391515984] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7dadb3b980 20 remove_watcher() i=0 2024-04-27T03:18:07.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7dadb3b980 2 removed watcher, disabling cache 2024-04-27T03:18:07.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93918391520496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d90e0 con 0x556b145c60c0 2024-04-27T03:18:07.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 93918391520496] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x556b145c60c0 2024-04-27T03:18:07.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.245+0000 7f7dadb3b980 20 remove_watcher() i=1 2024-04-27T03:18:07.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.249+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93918391519296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d4cf0 con 0x556b145ca380 2024-04-27T03:18:07.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.249+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 93918391519296] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.249+0000 7f7dadb3b980 20 remove_watcher() i=2 2024-04-27T03:18:07.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.249+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93918391526928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d90e0 con 0x7f7da4034020 2024-04-27T03:18:07.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.249+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 93918391526928] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x7f7da4034020 2024-04-27T03:18:07.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.249+0000 7f7dadb3b980 20 remove_watcher() i=3 2024-04-27T03:18:07.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.249+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93918391527984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d4840 con 0x556b145ca380 2024-04-27T03:18:07.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dabe57640 1 -- 172.21.15.110:0/3335086422 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 93918391527984] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7da4041050 con 0x556b145ca380 2024-04-27T03:18:07.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 20 remove_watcher() i=4 2024-04-27T03:18:07.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93918391524592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d90e0 con 0x556b145cf5c0 2024-04-27T03:18:07.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 93918391524592] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 20 remove_watcher() i=5 2024-04-27T03:18:07.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93918391529552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d4840 con 0x556b145cf5c0 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dab656640 1 -- 172.21.15.110:0/3335086422 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 93918391529552] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9c0d1070 con 0x556b145cf5c0 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 20 remove_watcher() i=6 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93918391532768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b145d90e0 con 0x7f7da4034020 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 93918391532768] v27'44 uv42 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7d9403c1c0 con 0x7f7da4034020 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 20 remove_watcher() i=7 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b145ca380 msgr2=0x556b145cc840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b145ca380 0x556b145cc840 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b145c60c0 msgr2=0x556b145c8560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b145c60c0 0x556b145c8560 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7da4034020 msgr2=0x7f7da40137e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7da4034020 0x7f7da40137e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b145cf5c0 msgr2=0x556b145d19b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.253+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b145cf5c0 0x556b145d19b0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c74034650 msgr2=0x7f7c74036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c74034650 0x7f7c74036b10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c0c8fd0 tx=0x7f7d9c0df000 comp rx=0 tx=0).stop 2024-04-27T03:18:07.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 msgr2=0x556b145baa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7daae55640 1 -- 172.21.15.110:0/3335086422 reap_dead start 2024-04-27T03:18:07.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145baa40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x556b145affc0 tx=0x7f7da40498d0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 shutdown_connections 2024-04-27T03:18:07.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/3335086422 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145b3580 0x556b145baa40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 >> 172.21.15.110:0/3335086422 conn(0x556b145af8c0 msgr2=0x556b145acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 shutdown_connections 2024-04-27T03:18:07.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/3335086422 wait complete. 2024-04-27T03:18:07.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c84034650 msgr2=0x7f7c84036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c84034650 0x7f7c84036b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x556b145967b0 tx=0x7f7d9c0c83c0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145a63a0 msgr2=0x556b145a67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145a63a0 0x556b145a67c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7da400aab0 tx=0x7f7da400af70 comp rx=0 tx=0).stop 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 shutdown_connections 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c84034650 0x7f7c84036b10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/1531904121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b145a63a0 0x556b145a67c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 >> 172.21.15.110:0/1531904121 conn(0x556b1459b1b0 msgr2=0x556b1459bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 shutdown_connections 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/1531904121 wait complete. 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b14593760 msgr2=0x556b14595b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b14593760 0x556b14595b50 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b1449c880 msgr2=0x556b14592420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b1449c880 0x556b14592420 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c900346a0 msgr2=0x7f7c90036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c900346a0 0x7f7c90036b60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7d94001da0 tx=0x7f7d94001300 comp rx=0 tx=0).stop 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 msgr2=0x556b1458b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1458b540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c018c00 tx=0x7f7d9c019810 comp rx=0 tx=0).stop 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 shutdown_connections 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b14593760 0x556b14595b50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b1449c880 0x556b14592420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7c900346a0 0x7f7c90036b60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 --2- 172.21.15.110:0/76020137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b1440b850 0x556b1458b540 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 >> 172.21.15.110:0/76020137 conn(0x556b14491a20 msgr2=0x556b1457ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 shutdown_connections 2024-04-27T03:18:07.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.257+0000 7f7dadb3b980 1 -- 172.21.15.110:0/76020137 wait complete. 2024-04-27T03:18:07.282 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:07.283 INFO:tasks.util.rgw:rgwadmin: client.0 : ['key', 'create', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'] 2024-04-27T03:18:07.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'key', 'create', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'] 2024-04-27T03:18:07.283 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph key create --uid foo --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh 2024-04-27T03:18:07.353 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:07.354 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:07.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.345+0000 7fb5ba7da980 1 Processor -- start 2024-04-27T03:18:07.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.345+0000 7fb5ba7da980 1 -- start start 2024-04-27T03:18:07.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.345+0000 7fb5ba7da980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55af1e970650 0x55af1e972a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.345+0000 7fb5ba7da980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55af1e82c0d0 con 0x55af1e96c0f0 2024-04-27T03:18:07.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.345+0000 7fb5ba7da980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55af1e821430 con 0x55af1e970650 2024-04-27T03:18:07.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b7af4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55af1e970650 0x55af1e972a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b7af4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55af1e970650 0x55af1e972a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50676/0 (socket says 172.21.15.110:50676) 2024-04-27T03:18:07.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b82f5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55af1e96c0f0 0x55af1e96c4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34646/0 (socket says 172.21.15.110:34646) 2024-04-27T03:18:07.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b7af4640 1 -- 172.21.15.110:0/3865104056 learned_addr learned my addr 172.21.15.110:0/3865104056 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b7af4640 1 -- 172.21.15.110:0/3865104056 >> v1:172.21.15.77:6789/0 conn(0x55af1e96c0f0 legacy=0x55af1e96c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b7af4640 1 -- 172.21.15.110:0/3865104056 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1e81a510 con 0x55af1e970650 2024-04-27T03:18:07.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b7af4640 1 --2- 172.21.15.110:0/3865104056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e970650 0x55af1e972a40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00a4e0 tx=0x7fb5ac01b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1725d5350e880 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b72f3640 1 -- 172.21.15.110:0/3865104056 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5ac0aa540 con 0x55af1e970650 2024-04-27T03:18:07.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b72f3640 1 -- 172.21.15.110:0/3865104056 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ac0aa6e0 con 0x55af1e970650 2024-04-27T03:18:07.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5b72f3640 1 -- 172.21.15.110:0/3865104056 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5ac0aaa40 con 0x55af1e970650 2024-04-27T03:18:07.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5ba7da980 1 -- 172.21.15.110:0/3865104056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e970650 msgr2=0x55af1e972a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/3865104056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e970650 0x55af1e972a40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00a4e0 tx=0x7fb5ac01b090 comp rx=0 tx=0).stop 2024-04-27T03:18:07.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5ba7da980 1 -- 172.21.15.110:0/3865104056 shutdown_connections 2024-04-27T03:18:07.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/3865104056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e970650 0x55af1e972a40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5ba7da980 1 -- 172.21.15.110:0/3865104056 >> 172.21.15.110:0/3865104056 conn(0x55af1e9679c0 msgr2=0x55af1e969de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5ba7da980 1 -- 172.21.15.110:0/3865104056 shutdown_connections 2024-04-27T03:18:07.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.349+0000 7fb5ba7da980 1 -- 172.21.15.110:0/3865104056 wait complete. 2024-04-27T03:18:07.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 Processor -- start 2024-04-27T03:18:07.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- start start 2024-04-27T03:18:07.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1e8e0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55af1e8702d0 con 0x55af1e96c0f0 2024-04-27T03:18:07.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1e8e0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1e8e0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50678/0 (socket says 172.21.15.110:50678) 2024-04-27T03:18:07.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 -- 172.21.15.110:0/1135564585 learned_addr learned my addr 172.21.15.110:0/1135564585 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 -- 172.21.15.110:0/1135564585 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1e7f6ae0 con 0x55af1e96c0f0 2024-04-27T03:18:07.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/1135564585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1e8e0f00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800b240 tx=0x7fb5a800b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d7cb83e98825d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb4a660c640 1 -- 172.21.15.110:0/1135564585 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5a800e700 con 0x55af1e96c0f0 2024-04-27T03:18:07.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb4a660c640 1 -- 172.21.15.110:0/1135564585 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a800e8a0 con 0x55af1e96c0f0 2024-04-27T03:18:07.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb4a660c640 1 -- 172.21.15.110:0/1135564585 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5a800eb80 con 0x55af1e96c0f0 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1135564585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 msgr2=0x55af1e8e0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1135564585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1e8e0f00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800b240 tx=0x7fb5a800b700 comp rx=0 tx=0).stop 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1135564585 shutdown_connections 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1135564585 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1e8e0f00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1135564585 >> 172.21.15.110:0/1135564585 conn(0x55af1e9679c0 msgr2=0x55af1ea52ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1135564585 shutdown_connections 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1135564585 wait complete. 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 Processor -- start 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- start start 2024-04-27T03:18:07.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1ea58920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5ba7da980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55af1e975a70 con 0x55af1e96c0f0 2024-04-27T03:18:07.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1ea58920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1ea58920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50686/0 (socket says 172.21.15.110:50686) 2024-04-27T03:18:07.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.365+0000 7fb5b82f5640 1 -- 172.21.15.110:0/642795564 learned_addr learned my addr 172.21.15.110:0/642795564 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b82f5640 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1ea58e60 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1ea58920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8000c10 tx=0x7fb5a8000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb4a4e09640 1 -- 172.21.15.110:0/642795564 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5a800fa90 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55af1ea571c0 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb4a4e09640 1 -- 172.21.15.110:0/642795564 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a800fc50 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af1ea576b0 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb4a4e09640 1 -- 172.21.15.110:0/642795564 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5a802c060 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb4a4e09640 1 -- 172.21.15.110:0/642795564 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb5a8017070 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb4a4e09640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4a0034650 0x7fb4a0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb4a4e09640 1 -- 172.21.15.110:0/642795564 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb5a8015020 con 0x55af1e96c0f0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b7af4640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4a0034650 0x7fb4a0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1e970650 0x55af1ea68aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea68fe0 con 0x55af1e970650 2024-04-27T03:18:07.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1e970650 0x55af1ea68aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b7af4640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4a0034650 0x7fb4a0036b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0ab000 tx=0x7fb5ac0a5000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1e970650 0x55af1ea68aa0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b8af6640 1 -- 172.21.15.110:0/642795564 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb5b00083f0 con 0x55af1e970650 2024-04-27T03:18:07.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1ea69890 0x55af1ea6bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea6c470 con 0x55af1ea69890 2024-04-27T03:18:07.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.369+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1ea69890 0x55af1ea6bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1ea69890 0x55af1ea6bd00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b82f5640 1 -- 172.21.15.110:0/642795564 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1ea69890 2024-04-27T03:18:07.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea6dd20 con 0x55af1e970650 2024-04-27T03:18:07.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b8af6640 1 -- 172.21.15.110:0/642795564 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fb5b00083f0 con 0x55af1e970650 2024-04-27T03:18:07.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea6cf50 con 0x55af1ea69890 2024-04-27T03:18:07.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b82f5640 1 -- 172.21.15.110:0/642795564 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1ea69890 2024-04-27T03:18:07.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea6d340 con 0x55af1e970650 2024-04-27T03:18:07.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b8af6640 1 -- 172.21.15.110:0/642795564 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fb5b00083f0 con 0x55af1e970650 2024-04-27T03:18:07.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5ba7da980 1 Processor -- start 2024-04-27T03:18:07.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5ba7da980 1 -- start start 2024-04-27T03:18:07.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5ba7da980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea7c650 0x55af1ea7ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5ba7da980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55af1e874310 con 0x55af1ea7c650 2024-04-27T03:18:07.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b8af6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea7c650 0x55af1ea7ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b8af6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea7c650 0x55af1ea7ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50700/0 (socket says 172.21.15.110:50700) 2024-04-27T03:18:07.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b8af6640 1 -- 172.21.15.110:0/1409766732 learned_addr learned my addr 172.21.15.110:0/1409766732 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.373+0000 7fb5b8af6640 1 -- 172.21.15.110:0/1409766732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1ea7cfb0 con 0x55af1ea7c650 2024-04-27T03:18:07.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea7c650 0x55af1ea7ca70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000aab0 tx=0x7fb5b000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb49dffb640 1 -- 172.21.15.110:0/1409766732 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0007d30 con 0x55af1ea7c650 2024-04-27T03:18:07.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb49dffb640 1 -- 172.21.15.110:0/1409766732 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b001d070 con 0x55af1ea7c650 2024-04-27T03:18:07.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb49dffb640 1 -- 172.21.15.110:0/1409766732 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0018440 con 0x55af1ea7c650 2024-04-27T03:18:07.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55af1ea7da50 con 0x55af1ea7c650 2024-04-27T03:18:07.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af1ea7df20 con 0x55af1ea7c650 2024-04-27T03:18:07.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb49dffb640 1 -- 172.21.15.110:0/1409766732 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb5b0013070 con 0x55af1ea7c650 2024-04-27T03:18:07.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb49dffb640 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4900346a0 0x7fb490036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb49dffb640 1 -- 172.21.15.110:0/1409766732 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb5b004b490 con 0x55af1ea7c650 2024-04-27T03:18:07.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4900346a0 0x7fb490036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5ba7da980 1 Processor -- start 2024-04-27T03:18:07.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5ba7da980 1 -- start start 2024-04-27T03:18:07.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4900346a0 0x7fb490036b60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55af1ea6ca70 tx=0x7fb5a800ff30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5ba7da980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea89c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5ba7da980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55af1e8e1650 con 0x55af1ea89820 2024-04-27T03:18:07.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b8af6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea89c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b8af6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea89c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50710/0 (socket says 172.21.15.110:50710) 2024-04-27T03:18:07.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b8af6640 1 -- 172.21.15.110:0/1043426005 learned_addr learned my addr 172.21.15.110:0/1043426005 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b8af6640 1 -- 172.21.15.110:0/1043426005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1ea8a210 con 0x55af1ea89820 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.377+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/1043426005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea89c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00498a0 tx=0x7fb5b0018930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6860784b8a4b8b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48f7fe640 1 -- 172.21.15.110:0/1043426005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0047070 con 0x55af1ea89820 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48f7fe640 1 -- 172.21.15.110:0/1043426005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b0018af0 con 0x55af1ea89820 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48f7fe640 1 -- 172.21.15.110:0/1043426005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5b004b0d0 con 0x55af1ea89820 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1043426005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 msgr2=0x55af1ea89c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1043426005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea89c40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00498a0 tx=0x7fb5b0018930 comp rx=0 tx=0).stop 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1043426005 shutdown_connections 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1043426005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea89c40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1043426005 >> 172.21.15.110:0/1043426005 conn(0x55af1ea85b60 msgr2=0x55af1ea83bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1043426005 shutdown_connections 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1043426005 wait complete. 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 Processor -- start 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- start start 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea90d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55af1ea8a970 con 0x55af1ea89820 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5b8af6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea90d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5b8af6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea90d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50714/0 (socket says 172.21.15.110:50714) 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 learned_addr learned my addr 172.21.15.110:0/2862512246 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1ea91280 con 0x55af1ea89820 2024-04-27T03:18:07.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea90d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0022770 tx=0x7fb5b0049830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48dffb640 1 -- 172.21.15.110:0/2862512246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0031070 con 0x55af1ea89820 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55af1ea91510 con 0x55af1ea89820 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48dffb640 1 -- 172.21.15.110:0/2862512246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b0016d60 con 0x55af1ea89820 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af1ea941f0 con 0x55af1ea89820 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48dffb640 1 -- 172.21.15.110:0/2862512246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0020820 con 0x55af1ea89820 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48dffb640 1 -- 172.21.15.110:0/2862512246 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb5b0034070 con 0x55af1ea89820 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.381+0000 7fb48dffb640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb480034650 0x7fb480036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb48dffb640 1 -- 172.21.15.110:0/2862512246 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb5b002e3b0 con 0x55af1ea89820 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb480034650 0x7fb480036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1ea9c390 0x55af1ea9e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea9efa0 con 0x55af1ea9c390 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b7af4640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1ea9c390 0x55af1ea9e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb480034650 0x7fb480036b10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800f700 tx=0x7fb5a803c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b7af4640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1ea9c390 0x55af1ea9e850 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 realm 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea9f370 con 0x55af1ea9c390 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ea9fc40 con 0x55af1ea9c390 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1eaa0660 0x55af1eaa2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa3200 con 0x55af1eaa0660 2024-04-27T03:18:07.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1eaa0660 0x55af1eaa2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1eaa0660 0x55af1eaa2a90 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa3a50 con 0x55af1ea9c390 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 20 searching for the correct realm 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55af1eaa57c0 0x55af1eaa7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa57c0 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.385+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55af1eaa57c0 0x55af1eaa7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b82f5640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55af1eaa57c0 0x55af1eaa7bb0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b8af6640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb5b0036070 0x7fb5b0017a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x7fb5b0036070 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb5b0036070 0x7fb5b0017a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb5b0036070 0x7fb5b0017a40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa57c0 2024-04-27T03:18:07.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.389+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x7fb5b0036070 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x7fb5b0036070 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa57c0 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa57c0 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.393+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa57c0 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x7fb5b0036070 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1ea9c390 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x7fb5b0036070 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa57c0 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x7fb5b0036070 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa57c0 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.397+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa81e0 con 0x55af1eaa0660 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa4400 con 0x55af1ea9c390 2024-04-27T03:18:07.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa4ca0 con 0x55af1eaa0660 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa4040 con 0x55af1ea9c390 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 zone default found 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 4 Realm: () 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:07.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 zonegroup default 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa8b10 con 0x7fb5b0036070 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaaadc0 con 0x55af1eaa57c0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaab160 con 0x55af1eaa57c0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 started sync module instance, tier type = 2024-04-27T03:18:07.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:07.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.401+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaaa250 con 0x55af1eaa0660 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94210622137968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaa9eb0 con 0x55af1eaa0660 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaabef0 con 0x55af1ea9c390 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94210622142976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaabe20 con 0x55af1ea9c390 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaacdf0 con 0x55af1eaa0660 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94210622142976] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.405+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94210622137968] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94210622141776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaac900 con 0x55af1eaa0660 2024-04-27T03:18:07.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaad690 con 0x7fb5b0036070 2024-04-27T03:18:07.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94210622149088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaac900 con 0x7fb5b0036070 2024-04-27T03:18:07.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaae130 con 0x55af1eaa0660 2024-04-27T03:18:07.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94210622149088] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94210622141776] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94210622145248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaae0f0 con 0x55af1eaa0660 2024-04-27T03:18:07.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.409+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaae520 con 0x55af1eaa57c0 2024-04-27T03:18:07.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94210622150144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaaedc0 con 0x55af1eaa57c0 2024-04-27T03:18:07.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaaf7c0 con 0x55af1eaa57c0 2024-04-27T03:18:07.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94210622145248] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94210622150144] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94210622151712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eaafbb0 con 0x55af1eaa57c0 2024-04-27T03:18:07.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eab0400 con 0x7fb5b0036070 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94210622151712] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'45 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94210622159792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eab07f0 con 0x7fb5b0036070 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=0 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=1 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=2 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=3 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=4 2024-04-27T03:18:07.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=5 2024-04-27T03:18:07.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=6 2024-04-27T03:18:07.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94210622159792] v27'46 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 20 add_watcher() i=7 2024-04-27T03:18:07.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:07.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.413+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1eab3820 con 0x55af1ea9c390 2024-04-27T03:18:07.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94210622161312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eab0cf0 con 0x55af1ea9c390 2024-04-27T03:18:07.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94210622161312] v27'70 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb46e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 5 note: GC not initialized 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb46d7e2640 20 reqs_thread_entry: start 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 Filter name: none 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb3deffd640 20 reqs_thread_entry: start 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead7180 con 0x55af1ea9c390 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv3 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:07.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead7bf0 con 0x55af1eaa57c0 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 p5YnriCv1nAtykxBrupQ [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 206+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2024-04-27T03:18:07.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.417+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~502 in=502b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eadd5e0 con 0x55af1ea9c390 2024-04-27T03:18:07.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(64 foo [delete,create,call,call,writefull 0~502] v27'4 uv4 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5ba7da980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5ba7da980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5ba7da980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify cookie 94210622345296 in=690b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead8cf0 con 0x55af1eaa0660 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94210622145248 notify 115964116995 ret 0) ==== 720+0+0 (crc 0 0 0) 0x55af1ea941f0 con 0x55af1eaa0660 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb48f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94210622145248 notifier 4331 bl.length()=678 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb48f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb48f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb48f7fe640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fb47c003eb0 con 0x55af1eaa0660 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(65 notify.4 [notify cookie 94210622345296 out=8b] v0'0 uv45 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(66 notify.4 [notify-ack] v0'0 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94210622345296 notify 115964116995 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55af1ea91510 con 0x55af1eaa0660 2024-04-27T03:18:07.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.421+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1eadd050 con 0x55af1eaa57c0 2024-04-27T03:18:07.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~7] v27'1 uv1 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5ba7da980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5ba7da980 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5ba7da980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify cookie 94210622345296 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead8840 con 0x55af1eaa0660 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94210622141776 notify 115964116996 ret 0) ==== 219+0+0 (crc 0 0 0) 0x55af1ea91510 con 0x55af1eaa0660 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb48effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94210622141776 notifier 4331 bl.length()=177 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb48effd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb48effd640 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb48effd640 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fb3ec003410 con 0x55af1eaa0660 2024-04-27T03:18:07.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(68 notify.2 [notify cookie 94210622345296 out=8b] v0'0 uv45 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== watch-notify(notify_complete (2) cookie 94210622345296 notify 115964116996 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55af1ea91280 con 0x55af1eaa0660 2024-04-27T03:18:07.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(69 notify.2 [notify-ack] v0'0 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.434 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"},{"user":"foo","access_key":"p5YnriCv1nAtykxBrupQ","secret_key":"Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh","active":true,"create_date":"2024-04-27T03:18:07.421373Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:07.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb46e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.425+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94210622161312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead8c70 con 0x55af1ea9c390 2024-04-27T03:18:07.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.429+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94210622161312] v27'71 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.429+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94210622137968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x55af1eaa0660 2024-04-27T03:18:07.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.429+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94210622137968] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.429+0000 7fb5ba7da980 20 remove_watcher() i=0 2024-04-27T03:18:07.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.429+0000 7fb5ba7da980 2 removed watcher, disabling cache 2024-04-27T03:18:07.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.429+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94210622142976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x55af1ea9c390 2024-04-27T03:18:07.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94210622142976] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x55af1ea9c390 2024-04-27T03:18:07.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5ba7da980 20 remove_watcher() i=1 2024-04-27T03:18:07.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94210622141776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x55af1eaa0660 2024-04-27T03:18:07.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94210622141776] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5ba7da980 20 remove_watcher() i=2 2024-04-27T03:18:07.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94210622149088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x7fb5b0036070 2024-04-27T03:18:07.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94210622149088] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5ba7da980 20 remove_watcher() i=3 2024-04-27T03:18:07.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.433+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94210622145248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x55af1eaa0660 2024-04-27T03:18:07.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5b8af6640 1 -- 172.21.15.110:0/2862512246 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94210622145248] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5b0033030 con 0x55af1eaa0660 2024-04-27T03:18:07.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5ba7da980 20 remove_watcher() i=4 2024-04-27T03:18:07.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94210622150144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x55af1eaa57c0 2024-04-27T03:18:07.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94210622150144] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5ba7da980 20 remove_watcher() i=5 2024-04-27T03:18:07.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94210622151712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x55af1eaa57c0 2024-04-27T03:18:07.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5b82f5640 1 -- 172.21.15.110:0/2862512246 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94210622151712] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5a802e070 con 0x55af1eaa57c0 2024-04-27T03:18:07.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5ba7da980 20 remove_watcher() i=6 2024-04-27T03:18:07.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.437+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94210622159792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55af1ead4e20 con 0x7fb5b0036070 2024-04-27T03:18:07.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94210622159792] v27'47 uv45 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb5ac0aaad0 con 0x7fb5b0036070 2024-04-27T03:18:07.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 20 remove_watcher() i=7 2024-04-27T03:18:07.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1eaa0660 msgr2=0x55af1eaa2a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1eaa0660 0x55af1eaa2a90 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1ea9c390 msgr2=0x55af1ea9e850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1ea9c390 0x55af1ea9e850 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb5b0036070 msgr2=0x7fb5b0017a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb5b0036070 0x7fb5b0017a40 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55af1eaa57c0 msgr2=0x55af1eaa7bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55af1eaa57c0 0x55af1eaa7bb0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb480034650 msgr2=0x7fb480036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb480034650 0x7fb480036b10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800f700 tx=0x7fb5a803c000 comp rx=0 tx=0).stop 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 msgr2=0x55af1ea90d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/2862512246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea89820 0x55af1ea90d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0022770 tx=0x7fb5b0049830 comp rx=0 tx=0).stop 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 reap_dead start 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5b7af4640 1 -- 172.21.15.110:0/2862512246 reap_dead start 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 shutdown_connections 2024-04-27T03:18:07.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 >> 172.21.15.110:0/2862512246 conn(0x55af1ea85b60 msgr2=0x55af1ea831c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 shutdown_connections 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/2862512246 wait complete. 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4900346a0 msgr2=0x7fb490036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4900346a0 0x7fb490036b60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x55af1ea6ca70 tx=0x7fb5a800ff30 comp rx=0 tx=0).stop 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea7c650 msgr2=0x55af1ea7ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea7c650 0x55af1ea7ca70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000aab0 tx=0x7fb5b000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 shutdown_connections 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4900346a0 0x7fb490036b60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/1409766732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1ea7c650 0x55af1ea7ca70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 >> 172.21.15.110:0/1409766732 conn(0x55af1ea71460 msgr2=0x55af1ea71f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 shutdown_connections 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/1409766732 wait complete. 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1ea69890 msgr2=0x55af1ea6bd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1ea69890 0x55af1ea6bd00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1e970650 msgr2=0x55af1ea68aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1e970650 0x55af1ea68aa0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4a0034650 msgr2=0x7fb4a0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4a0034650 0x7fb4a0036b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0ab000 tx=0x7fb5ac0a5000 comp rx=0 tx=0).stop 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 msgr2=0x55af1ea58920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1ea58920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8000c10 tx=0x7fb5a8000ee0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 shutdown_connections 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55af1ea69890 0x55af1ea6bd00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55af1e970650 0x55af1ea68aa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb4a0034650 0x7fb4a0036b10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 --2- 172.21.15.110:0/642795564 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55af1e96c0f0 0x55af1ea58920 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 >> 172.21.15.110:0/642795564 conn(0x55af1e9679c0 msgr2=0x55af1ea51a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 shutdown_connections 2024-04-27T03:18:07.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.441+0000 7fb5ba7da980 1 -- 172.21.15.110:0/642795564 wait complete. 2024-04-27T03:18:07.458 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True, 'create_date': '2024-04-27T03:18:07.421373Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:07.459 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:07.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:07.459 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:07.524 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:07.525 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:07.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd362e980 1 Processor -- start 2024-04-27T03:18:07.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd362e980 1 -- start start 2024-04-27T03:18:07.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd362e980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564acd1f16d0 0x564acd1f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd362e980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x564acd0ad0d0 con 0x564acd1ed170 2024-04-27T03:18:07.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd362e980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x564acd0a2430 con 0x564acd1f16d0 2024-04-27T03:18:07.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd0948640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564acd1f16d0 0x564acd1f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd1149640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x564acd1ed170 0x564acd1ed550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34656/0 (socket says 172.21.15.110:34656) 2024-04-27T03:18:07.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd1149640 1 -- 172.21.15.110:0/2616108305 learned_addr learned my addr 172.21.15.110:0/2616108305 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:07.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bc3fff640 1 -- 172.21.15.110:0/2616108305 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661347922 0 0) 0x564acd0ad0d0 con 0x564acd1ed170 2024-04-27T03:18:07.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bc3fff640 1 -- 172.21.15.110:0/2616108305 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bbc001e20 con 0x564acd1ed170 2024-04-27T03:18:07.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd0948640 1 -- 172.21.15.110:0/2616108305 >> v1:172.21.15.77:6789/0 conn(0x564acd1ed170 legacy=0x564acd1ed550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.517+0000 7f0bd0948640 1 -- 172.21.15.110:0/2616108305 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acd09b510 con 0x564acd1f16d0 2024-04-27T03:18:07.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd0948640 1 --2- 172.21.15.110:0/2616108305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1f16d0 0x564acd1f3ac0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800a480 tx=0x7f0bc801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d07599ce6fd06e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bc3fff640 1 -- 172.21.15.110:0/2616108305 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc80b0070 con 0x564acd1f16d0 2024-04-27T03:18:07.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bc3fff640 1 -- 172.21.15.110:0/2616108305 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc80ab540 con 0x564acd1f16d0 2024-04-27T03:18:07.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bc3fff640 1 -- 172.21.15.110:0/2616108305 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc80ab820 con 0x564acd1f16d0 2024-04-27T03:18:07.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd362e980 1 -- 172.21.15.110:0/2616108305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1f16d0 msgr2=0x564acd1f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd362e980 1 --2- 172.21.15.110:0/2616108305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1f16d0 0x564acd1f3ac0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800a480 tx=0x7f0bc801b280 comp rx=0 tx=0).stop 2024-04-27T03:18:07.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd362e980 1 -- 172.21.15.110:0/2616108305 shutdown_connections 2024-04-27T03:18:07.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd362e980 1 --2- 172.21.15.110:0/2616108305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1f16d0 0x564acd1f3ac0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd362e980 1 -- 172.21.15.110:0/2616108305 >> 172.21.15.110:0/2616108305 conn(0x564acd1e8a40 msgr2=0x564acd1eae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd362e980 1 -- 172.21.15.110:0/2616108305 shutdown_connections 2024-04-27T03:18:07.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.521+0000 7f0bd362e980 1 -- 172.21.15.110:0/2616108305 wait complete. 2024-04-27T03:18:07.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.533+0000 7f0bd362e980 1 Processor -- start 2024-04-27T03:18:07.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.533+0000 7f0bd362e980 1 -- start start 2024-04-27T03:18:07.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.533+0000 7f0bd362e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd1616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.533+0000 7f0bd362e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564acd0f12d0 con 0x564acd1ed170 2024-04-27T03:18:07.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.533+0000 7f0bd1149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd1616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.533+0000 7f0bd1149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd1616d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50728/0 (socket says 172.21.15.110:50728) 2024-04-27T03:18:07.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.533+0000 7f0bd1149640 1 -- 172.21.15.110:0/253731088 learned_addr learned my addr 172.21.15.110:0/253731088 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd1149640 1 -- 172.21.15.110:0/253731088 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acd077ae0 con 0x564acd1ed170 2024-04-27T03:18:07.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd1149640 1 --2- 172.21.15.110:0/253731088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd1616d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc006fc0 tx=0x7f0bcc001af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f14fef71a8b701d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0abf606640 1 -- 172.21.15.110:0/253731088 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bcc004530 con 0x564acd1ed170 2024-04-27T03:18:07.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0abf606640 1 -- 172.21.15.110:0/253731088 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bcc0046d0 con 0x564acd1ed170 2024-04-27T03:18:07.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0abf606640 1 -- 172.21.15.110:0/253731088 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bcc004a70 con 0x564acd1ed170 2024-04-27T03:18:07.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- 172.21.15.110:0/253731088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 msgr2=0x564acd1616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 --2- 172.21.15.110:0/253731088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd1616d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc006fc0 tx=0x7f0bcc001af0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- 172.21.15.110:0/253731088 shutdown_connections 2024-04-27T03:18:07.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 --2- 172.21.15.110:0/253731088 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd1616d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- 172.21.15.110:0/253731088 >> 172.21.15.110:0/253731088 conn(0x564acd1e8a40 msgr2=0x564acd2d3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- 172.21.15.110:0/253731088 shutdown_connections 2024-04-27T03:18:07.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- 172.21.15.110:0/253731088 wait complete. 2024-04-27T03:18:07.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 Processor -- start 2024-04-27T03:18:07.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- start start 2024-04-27T03:18:07.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd2e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564acd1f6af0 con 0x564acd1ed170 2024-04-27T03:18:07.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd1149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd2e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd1149640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd2e02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50730/0 (socket says 172.21.15.110:50730) 2024-04-27T03:18:07.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd1149640 1 -- 172.21.15.110:0/1596720692 learned_addr learned my addr 172.21.15.110:0/1596720692 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd1149640 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acd2e07e0 con 0x564acd1ed170 2024-04-27T03:18:07.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd1149640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd2e02a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc002260 tx=0x7f0bcc000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0abde03640 1 -- 172.21.15.110:0/1596720692 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bcc013760 con 0x564acd1ed170 2024-04-27T03:18:07.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0abde03640 1 -- 172.21.15.110:0/1596720692 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bcc013900 con 0x564acd1ed170 2024-04-27T03:18:07.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564acd2e0a70 con 0x564acd1ed170 2024-04-27T03:18:07.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0abde03640 1 -- 172.21.15.110:0/1596720692 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bcc013c00 con 0x564acd1ed170 2024-04-27T03:18:07.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.537+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564acd2e0f00 con 0x564acd1ed170 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0abde03640 1 -- 172.21.15.110:0/1596720692 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0bcc010070 con 0x564acd1ed170 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0abde03640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0ab8034650 0x7f0ab8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0abde03640 1 -- 172.21.15.110:0/1596720692 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f0bcc018070 con 0x564acd1ed170 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd0948640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0ab8034650 0x7f0ab8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd1f16d0 0x564acd2e7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd2e7770 con 0x564acd1f16d0 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd194a640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd1f16d0 0x564acd2e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd0948640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0ab8034650 0x7f0ab8036b10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80ad000 tx=0x7f0bc80a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd194a640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd1f16d0 0x564acd2e7170 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd194a640 1 -- 172.21.15.110:0/1596720692 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0bc40083f0 con 0x564acd1f16d0 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd2e8560 0x564acd2ea950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd2eaf80 con 0x564acd2e8560 2024-04-27T03:18:07.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd1149640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd2e8560 0x564acd2ea950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd1149640 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd2e8560 0x564acd2ea950 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd1149640 1 -- 172.21.15.110:0/1596720692 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f0bcc01b030 con 0x564acd2e8560 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.541+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd2ec830 con 0x564acd1f16d0 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd194a640 1 -- 172.21.15.110:0/1596720692 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f0bc40083f0 con 0x564acd1f16d0 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd2eba60 con 0x564acd2e8560 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd1149640 1 -- 172.21.15.110:0/1596720692 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f0bcc01b030 con 0x564acd2e8560 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd2ebe50 con 0x564acd1f16d0 2024-04-27T03:18:07.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd194a640 1 -- 172.21.15.110:0/1596720692 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f0bc40083f0 con 0x564acd1f16d0 2024-04-27T03:18:07.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 Processor -- start 2024-04-27T03:18:07.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 -- start start 2024-04-27T03:18:07.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd2d9b50 0x564acd2d7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564acd0f5310 con 0x564acd2d9b50 2024-04-27T03:18:07.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd194a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd2d9b50 0x564acd2d7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd194a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd2d9b50 0x564acd2d7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50738/0 (socket says 172.21.15.110:50738) 2024-04-27T03:18:07.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd194a640 1 -- 172.21.15.110:0/763778122 learned_addr learned my addr 172.21.15.110:0/763778122 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd194a640 1 -- 172.21.15.110:0/763778122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acd2d8530 con 0x564acd2d9b50 2024-04-27T03:18:07.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd194a640 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd2d9b50 0x564acd2d7ff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400aab0 tx=0x7f0bc400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0ab6ffd640 1 -- 172.21.15.110:0/763778122 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc4007d30 con 0x564acd2d9b50 2024-04-27T03:18:07.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564acd2d8820 con 0x564acd2d9b50 2024-04-27T03:18:07.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0ab6ffd640 1 -- 172.21.15.110:0/763778122 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc401d070 con 0x564acd2d9b50 2024-04-27T03:18:07.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0ab6ffd640 1 -- 172.21.15.110:0/763778122 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc4018440 con 0x564acd2d9b50 2024-04-27T03:18:07.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.545+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564acd2d8cf0 con 0x564acd2d9b50 2024-04-27T03:18:07.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0ab6ffd640 1 -- 172.21.15.110:0/763778122 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0bc4013070 con 0x564acd2d9b50 2024-04-27T03:18:07.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0ab6ffd640 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0aac034650 0x7f0aac036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0ab6ffd640 1 -- 172.21.15.110:0/763778122 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f0bc404b520 con 0x564acd2d9b50 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd1149640 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0aac034650 0x7f0aac036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 Processor -- start 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- start start 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd1149640 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0aac034650 0x7f0aac036b10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x564acd2eb580 tx=0x7f0bcc010380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd30a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564acd161e20 con 0x564acd30a530 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd30a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd30a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50750/0 (socket says 172.21.15.110:50750) 2024-04-27T03:18:07.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 -- 172.21.15.110:0/2122272777 learned_addr learned my addr 172.21.15.110:0/2122272777 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 -- 172.21.15.110:0/2122272777 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acd30af20 con 0x564acd30a530 2024-04-27T03:18:07.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 --2- 172.21.15.110:0/2122272777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd30a950 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4049930 tx=0x7f0bc4018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8382f1e905a67acb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0ab4ff9640 1 -- 172.21.15.110:0/2122272777 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc4047070 con 0x564acd30a530 2024-04-27T03:18:07.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0ab4ff9640 1 -- 172.21.15.110:0/2122272777 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc4018bc0 con 0x564acd30a530 2024-04-27T03:18:07.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0ab4ff9640 1 -- 172.21.15.110:0/2122272777 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc4020360 con 0x564acd30a530 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- 172.21.15.110:0/2122272777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 msgr2=0x564acd30a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 --2- 172.21.15.110:0/2122272777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd30a950 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4049930 tx=0x7f0bc4018950 comp rx=0 tx=0).stop 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- 172.21.15.110:0/2122272777 shutdown_connections 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 --2- 172.21.15.110:0/2122272777 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd30a950 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- 172.21.15.110:0/2122272777 >> 172.21.15.110:0/2122272777 conn(0x564acd306870 msgr2=0x564acd3048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- 172.21.15.110:0/2122272777 shutdown_connections 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- 172.21.15.110:0/2122272777 wait complete. 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 Processor -- start 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- start start 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd311a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd362e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564acd30b6a0 con 0x564acd30a530 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd311a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd311a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50754/0 (socket says 172.21.15.110:50754) 2024-04-27T03:18:07.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.549+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 learned_addr learned my addr 172.21.15.110:0/395173363 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564acd311f50 con 0x564acd30a530 2024-04-27T03:18:07.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd194a640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd311a10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40168b0 tx=0x7f0bc40498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0aa2ffd640 1 -- 172.21.15.110:0/395173363 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc4031070 con 0x564acd30a530 2024-04-27T03:18:07.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564acd3121e0 con 0x564acd30a530 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0aa2ffd640 1 -- 172.21.15.110:0/395173363 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc4020900 con 0x564acd30a530 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564acd314ec0 con 0x564acd30a530 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0aa2ffd640 1 -- 172.21.15.110:0/395173363 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bc4049a70 con 0x564acd30a530 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0aa2ffd640 1 -- 172.21.15.110:0/395173363 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0bc4034070 con 0x564acd30a530 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0aa2ffd640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0a9c034650 0x7f0a9c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0aa2ffd640 1 -- 172.21.15.110:0/395173363 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f0bc400c0b0 con 0x564acd30a530 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd1149640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0a9c034650 0x7f0a9c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd31d090 0x564acd31f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd31fca0 con 0x564acd31d090 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd0948640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd31d090 0x564acd31f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd1149640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0a9c034650 0x7f0a9c036b10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc0106e0 tx=0x7f0bcc034000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd0948640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd31d090 0x564acd31f550 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 realm 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd320070 con 0x564acd31d090 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd320910 con 0x564acd31d090 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd321330 0x564acd323720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd323e90 con 0x564acd321330 2024-04-27T03:18:07.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.553+0000 7f0bd194a640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd321330 0x564acd323720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd194a640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd321330 0x564acd323720 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd362e980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd324730 con 0x564acd31d090 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd362e980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd362e980 20 searching for the correct realm 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564acd3264a0 0x564acd328890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd3264a0 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd1149640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564acd3264a0 0x564acd328890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd1149640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564acd3264a0 0x564acd328890 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd194a640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0bc4036070 0x7f0bc4021700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x7f0bc4036070 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd0948640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0bc4036070 0x7f0bc4021700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd0948640 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0bc4036070 0x7f0bc4021700 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.557+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd3264a0 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x7f0bc4036070 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x7f0bc4036070 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.561+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd3264a0 2024-04-27T03:18:07.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd3264a0 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd3264a0 2024-04-27T03:18:07.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x7f0bc4036070 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd31d090 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x7f0bc4036070 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd3264a0 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x7f0bc4036070 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.565+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd3264a0 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564acd328e20 con 0x564acd321330 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd3250e0 con 0x564acd31d090 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd325560 con 0x564acd321330 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd3259c0 con 0x564acd31d090 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 zone default found 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 4 Realm: () 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 zonegroup default 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.569+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd329750 con 0x7f0bc4036070 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd32b160 con 0x564acd3264a0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd32b630 con 0x564acd3264a0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 20 started sync module instance, tier type = 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:07.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32a570 con 0x564acd321330 2024-04-27T03:18:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94879270215872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32ba90 con 0x564acd321330 2024-04-27T03:18:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32c2e0 con 0x564acd31d090 2024-04-27T03:18:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94879270222480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32c500 con 0x564acd31d090 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.573+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32d020 con 0x564acd321330 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94879270222480] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94879270215872] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94879270218080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32cfe0 con 0x564acd321330 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32d8c0 con 0x7f0bc4036070 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94879270229008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32cfe0 con 0x7f0bc4036070 2024-04-27T03:18:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32e370 con 0x564acd321330 2024-04-27T03:18:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94879270229008] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.577+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94879270218080] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94879270226672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32e370 con 0x564acd321330 2024-04-27T03:18:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32e760 con 0x564acd3264a0 2024-04-27T03:18:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94879270230064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32ec10 con 0x564acd3264a0 2024-04-27T03:18:07.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32f610 con 0x564acd3264a0 2024-04-27T03:18:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94879270230064] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94879270233952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32f5d0 con 0x564acd3264a0 2024-04-27T03:18:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94879270226672] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd330010 con 0x7f0bc4036070 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94879270233952] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'48 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94879270236448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32fb60 con 0x7f0bc4036070 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 20 add_watcher() i=0 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 20 add_watcher() i=1 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 20 add_watcher() i=2 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 20 add_watcher() i=3 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 20 add_watcher() i=4 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 20 add_watcher() i=5 2024-04-27T03:18:07.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.581+0000 7f0bd362e980 20 add_watcher() i=6 2024-04-27T03:18:07.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94879270236448] v27'49 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 20 add_watcher() i=7 2024-04-27T03:18:07.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:07.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd332f90 con 0x564acd31d090 2024-04-27T03:18:07.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94879270231680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd3304e0 con 0x564acd31d090 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94879270231680] v27'72 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0a877e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 5 note: GC not initialized 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0a867e4640 20 reqs_thread_entry: start 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 20 Filter name: none 2024-04-27T03:18:07.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f09f7fff640 20 reqs_thread_entry: start 2024-04-27T03:18:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.585+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564acd356290 con 0x564acd31d090 2024-04-27T03:18:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.589+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~502 out=502b,getxattrs out=75b] v0'0 uv4 ondisk = 0) ==== 231+0+625 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.589+0000 7f0bd362e980 20 rados_obj.operate() r=0 bl.length=502 2024-04-27T03:18:07.596 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"},{"user":"foo","access_key":"p5YnriCv1nAtykxBrupQ","secret_key":"Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh","active":true,"create_date":"2024-04-27T03:18:07.421373Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.589+0000 7f0a877e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.589+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94879270231680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd354ce0 con 0x564acd31d090 2024-04-27T03:18:07.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.589+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94879270231680] v27'73 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.589+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94879270215872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32de80 con 0x564acd321330 2024-04-27T03:18:07.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94879270215872] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd362e980 20 remove_watcher() i=0 2024-04-27T03:18:07.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd362e980 2 removed watcher, disabling cache 2024-04-27T03:18:07.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94879270222480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32a0c0 con 0x564acd31d090 2024-04-27T03:18:07.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94879270222480] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x564acd31d090 2024-04-27T03:18:07.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd362e980 20 remove_watcher() i=1 2024-04-27T03:18:07.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94879270218080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32a0c0 con 0x564acd321330 2024-04-27T03:18:07.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94879270218080] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd362e980 20 remove_watcher() i=2 2024-04-27T03:18:07.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.593+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94879270229008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32a0c0 con 0x7f0bc4036070 2024-04-27T03:18:07.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.597+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94879270229008] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.597+0000 7f0bd362e980 20 remove_watcher() i=3 2024-04-27T03:18:07.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.597+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94879270226672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32d410 con 0x564acd321330 2024-04-27T03:18:07.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.597+0000 7f0bd194a640 1 -- 172.21.15.110:0/395173363 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94879270226672] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc4033030 con 0x564acd321330 2024-04-27T03:18:07.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.597+0000 7f0bd362e980 20 remove_watcher() i=4 2024-04-27T03:18:07.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.597+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94879270230064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32a0c0 con 0x564acd3264a0 2024-04-27T03:18:07.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94879270230064] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 20 remove_watcher() i=5 2024-04-27T03:18:07.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94879270233952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32caf0 con 0x564acd3264a0 2024-04-27T03:18:07.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd1149640 1 -- 172.21.15.110:0/395173363 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94879270233952] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bcc03d070 con 0x564acd3264a0 2024-04-27T03:18:07.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 20 remove_watcher() i=6 2024-04-27T03:18:07.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94879270236448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564acd32a0c0 con 0x7f0bc4036070 2024-04-27T03:18:07.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94879270236448] v27'50 uv48 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0bc80b3040 con 0x7f0bc4036070 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 20 remove_watcher() i=7 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd321330 msgr2=0x564acd323720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd321330 0x564acd323720 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd31d090 msgr2=0x564acd31f550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd31d090 0x564acd31f550 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0bc4036070 msgr2=0x7f0bc4021700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0bc4036070 0x7f0bc4021700 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564acd3264a0 msgr2=0x564acd328890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564acd3264a0 0x564acd328890 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0a9c034650 msgr2=0x7f0a9c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0a9c034650 0x7f0a9c036b10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc0106e0 tx=0x7f0bcc034000 comp rx=0 tx=0).stop 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 msgr2=0x564acd311a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd0948640 1 -- 172.21.15.110:0/395173363 reap_dead start 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd311a10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40168b0 tx=0x7f0bc40498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 shutdown_connections 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/395173363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd30a530 0x564acd311a10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 >> 172.21.15.110:0/395173363 conn(0x564acd306870 msgr2=0x564acd303ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 shutdown_connections 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/395173363 wait complete. 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0aac034650 msgr2=0x7f0aac036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0aac034650 0x7f0aac036b10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x564acd2eb580 tx=0x7f0bcc010380 comp rx=0 tx=0).stop 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd2d9b50 msgr2=0x564acd2d7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd2d9b50 0x564acd2d7ff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400aab0 tx=0x7f0bc400af70 comp rx=0 tx=0).stop 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 shutdown_connections 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0aac034650 0x7f0aac036b10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/763778122 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd2d9b50 0x564acd2d7ff0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 >> 172.21.15.110:0/763778122 conn(0x564acd2eff60 msgr2=0x564acd2f0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 shutdown_connections 2024-04-27T03:18:07.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/763778122 wait complete. 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd2e8560 msgr2=0x564acd2ea950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd2e8560 0x564acd2ea950 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd1f16d0 msgr2=0x564acd2e7170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd1f16d0 0x564acd2e7170 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0ab8034650 msgr2=0x7f0ab8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0ab8034650 0x7f0ab8036b10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80ad000 tx=0x7f0bc80a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 msgr2=0x564acd2e02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd2e02a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc002260 tx=0x7f0bcc000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 shutdown_connections 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564acd2e8560 0x564acd2ea950 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564acd1f16d0 0x564acd2e7170 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0ab8034650 0x7f0ab8036b10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 --2- 172.21.15.110:0/1596720692 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564acd1ed170 0x564acd2e02a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 >> 172.21.15.110:0/1596720692 conn(0x564acd1e8a40 msgr2=0x564acd2d27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 shutdown_connections 2024-04-27T03:18:07.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.601+0000 7f0bd362e980 1 -- 172.21.15.110:0/1596720692 wait complete. 2024-04-27T03:18:07.619 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True, 'create_date': '2024-04-27T03:18:07.421373Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:07.619 INFO:tasks.util.rgw:rgwadmin: client.0 : ['key', 'rm', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ'] 2024-04-27T03:18:07.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'key', 'rm', '--uid', 'foo', '--access-key', 'p5YnriCv1nAtykxBrupQ'] 2024-04-27T03:18:07.620 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph key rm --uid foo --access-key p5YnriCv1nAtykxBrupQ 2024-04-27T03:18:07.692 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:07.692 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:07.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14c18f7980 1 Processor -- start 2024-04-27T03:18:07.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14c18f7980 1 -- start start 2024-04-27T03:18:07.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14c18f7980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56165f71b160 0x56165f71b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14c18f7980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x56165f5db0d0 con 0x56165f71ba80 2024-04-27T03:18:07.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14c18f7980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x56165f5d0430 con 0x56165f71b160 2024-04-27T03:18:07.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14bf412640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56165f71b160 0x56165f71b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14bec11640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x56165f71ba80 0x56165f71c490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34662/0 (socket says 172.21.15.110:34662) 2024-04-27T03:18:07.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14bec11640 1 -- 172.21.15.110:0/1823812156 learned_addr learned my addr 172.21.15.110:0/1823812156 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:07.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14bf412640 1 -- 172.21.15.110:0/1823812156 >> v1:172.21.15.77:6789/0 conn(0x56165f71ba80 legacy=0x56165f71c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.685+0000 7f14bf412640 1 -- 172.21.15.110:0/1823812156 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165f5c9510 con 0x56165f71b160 2024-04-27T03:18:07.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14bf412640 1 --2- 172.21.15.110:0/1823812156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f71b540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f14b000a480 tx=0x7f14b001b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9737f37057c156ad server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14be410640 1 -- 172.21.15.110:0/1823812156 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00b0070 con 0x56165f71b160 2024-04-27T03:18:07.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14be410640 1 -- 172.21.15.110:0/1823812156 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b00ab540 con 0x56165f71b160 2024-04-27T03:18:07.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14be410640 1 -- 172.21.15.110:0/1823812156 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00ab840 con 0x56165f71b160 2024-04-27T03:18:07.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14c18f7980 1 -- 172.21.15.110:0/1823812156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 msgr2=0x56165f71b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14c18f7980 1 --2- 172.21.15.110:0/1823812156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f71b540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f14b000a480 tx=0x7f14b001b280 comp rx=0 tx=0).stop 2024-04-27T03:18:07.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14c18f7980 1 -- 172.21.15.110:0/1823812156 shutdown_connections 2024-04-27T03:18:07.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14c18f7980 1 --2- 172.21.15.110:0/1823812156 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f71b540 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14c18f7980 1 -- 172.21.15.110:0/1823812156 >> 172.21.15.110:0/1823812156 conn(0x56165f716a30 msgr2=0x56165f718e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14c18f7980 1 -- 172.21.15.110:0/1823812156 shutdown_connections 2024-04-27T03:18:07.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.689+0000 7f14c18f7980 1 -- 172.21.15.110:0/1823812156 wait complete. 2024-04-27T03:18:07.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.701+0000 7f14c18f7980 1 Processor -- start 2024-04-27T03:18:07.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.701+0000 7f14c18f7980 1 -- start start 2024-04-27T03:18:07.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.701+0000 7f14c18f7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f806ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.701+0000 7f14c18f7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56165f61f2d0 con 0x56165f71b160 2024-04-27T03:18:07.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.701+0000 7f14bf412640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f806ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f806ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50768/0 (socket says 172.21.15.110:50768) 2024-04-27T03:18:07.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 -- 172.21.15.110:0/909498779 learned_addr learned my addr 172.21.15.110:0/909498779 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 -- 172.21.15.110:0/909498779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165f5a5ae0 con 0x56165f71b160 2024-04-27T03:18:07.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 --2- 172.21.15.110:0/909498779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f806ae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f14b00ae040 tx=0x7f14b0003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb50f805d4a094 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f13ad60a640 1 -- 172.21.15.110:0/909498779 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00b0040 con 0x56165f71b160 2024-04-27T03:18:07.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f13ad60a640 1 -- 172.21.15.110:0/909498779 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b00ab9d0 con 0x56165f71b160 2024-04-27T03:18:07.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f13ad60a640 1 -- 172.21.15.110:0/909498779 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00abcf0 con 0x56165f71b160 2024-04-27T03:18:07.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- 172.21.15.110:0/909498779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 msgr2=0x56165f806ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 --2- 172.21.15.110:0/909498779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f806ae0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f14b00ae040 tx=0x7f14b0003030 comp rx=0 tx=0).stop 2024-04-27T03:18:07.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- 172.21.15.110:0/909498779 shutdown_connections 2024-04-27T03:18:07.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 --2- 172.21.15.110:0/909498779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f806ae0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- 172.21.15.110:0/909498779 >> 172.21.15.110:0/909498779 conn(0x56165f716a30 msgr2=0x56165f802670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- 172.21.15.110:0/909498779 shutdown_connections 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- 172.21.15.110:0/909498779 wait complete. 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 Processor -- start 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- start start 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f690520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56165f725200 con 0x56165f71b160 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f690520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f690520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50782/0 (socket says 172.21.15.110:50782) 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 -- 172.21.15.110:0/877574324 learned_addr learned my addr 172.21.15.110:0/877574324 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165f690a60 con 0x56165f71b160 2024-04-27T03:18:07.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14bf412640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f690520 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f14b0018eb0 tx=0x7f14b0018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f13abe07640 1 -- 172.21.15.110:0/877574324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00b0070 con 0x56165f71b160 2024-04-27T03:18:07.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f13abe07640 1 -- 172.21.15.110:0/877574324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b00b68f0 con 0x56165f71b160 2024-04-27T03:18:07.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56165f811920 con 0x56165f71b160 2024-04-27T03:18:07.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f13abe07640 1 -- 172.21.15.110:0/877574324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00b6bd0 con 0x56165f71b160 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.705+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165f811cd0 con 0x56165f71b160 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f13abe07640 1 -- 172.21.15.110:0/877574324 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f14b00bf3f0 con 0x56165f71b160 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f13abe07640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f13a4034650 0x7f13a4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f13abe07640 1 -- 172.21.15.110:0/877574324 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f14b00bc070 con 0x56165f71b160 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bec11640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f13a4034650 0x7f13a4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f71ba80 0x56165f815940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f815fb0 con 0x56165f71ba80 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bfc13640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f71ba80 0x56165f815940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bec11640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f13a4034650 0x7f13a4036b10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000f70 tx=0x7f14b40012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bfc13640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f71ba80 0x56165f815940 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bfc13640 1 -- 172.21.15.110:0/877574324 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f14b800a4d0 con 0x56165f71ba80 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f816da0 0x56165f819190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f8197d0 con 0x56165f816da0 2024-04-27T03:18:07.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bf412640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f816da0 0x56165f819190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bf412640 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f816da0 0x56165f819190 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14bf412640 1 -- 172.21.15.110:0/877574324 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f816da0 2024-04-27T03:18:07.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.709+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f81b080 con 0x56165f71ba80 2024-04-27T03:18:07.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bfc13640 1 -- 172.21.15.110:0/877574324 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f14b800a4d0 con 0x56165f71ba80 2024-04-27T03:18:07.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f81a2b0 con 0x56165f816da0 2024-04-27T03:18:07.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bf412640 1 -- 172.21.15.110:0/877574324 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f816da0 2024-04-27T03:18:07.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f81a6a0 con 0x56165f71ba80 2024-04-27T03:18:07.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bfc13640 1 -- 172.21.15.110:0/877574324 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f14b800a4d0 con 0x56165f71ba80 2024-04-27T03:18:07.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 Processor -- start 2024-04-27T03:18:07.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 -- start start 2024-04-27T03:18:07.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f829980 0x56165f829da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56165f623310 con 0x56165f829980 2024-04-27T03:18:07.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bfc13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f829980 0x56165f829da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bfc13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f829980 0x56165f829da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50784/0 (socket says 172.21.15.110:50784) 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bfc13640 1 -- 172.21.15.110:0/2797047100 learned_addr learned my addr 172.21.15.110:0/2797047100 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bfc13640 1 -- 172.21.15.110:0/2797047100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165f82a2e0 con 0x56165f829980 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14bfc13640 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f829980 0x56165f829da0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f14b800dfc0 tx=0x7f14b80178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f13a9602640 1 -- 172.21.15.110:0/2797047100 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b80157d0 con 0x56165f829980 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56165f82ad80 con 0x56165f829980 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f13a9602640 1 -- 172.21.15.110:0/2797047100 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b8015970 con 0x56165f829980 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f13a9602640 1 -- 172.21.15.110:0/2797047100 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b8015c70 con 0x56165f829980 2024-04-27T03:18:07.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.713+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165f82b250 con 0x56165f829980 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f13a9602640 1 -- 172.21.15.110:0/2797047100 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f14b8029440 con 0x56165f829980 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f13a9602640 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1398034650 0x7f1398036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f13a9602640 1 -- 172.21.15.110:0/2797047100 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f14b800e040 con 0x56165f829980 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bf412640 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1398034650 0x7f1398036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 Processor -- start 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- start start 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bf412640 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1398034650 0x7f1398036b10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x56165f819dd0 tx=0x7f14b00e2d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f836f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56165f807170 con 0x56165f836b30 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f836f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f836f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50800/0 (socket says 172.21.15.110:50800) 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 -- 172.21.15.110:0/3461118324 learned_addr learned my addr 172.21.15.110:0/3461118324 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 -- 172.21.15.110:0/3461118324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165f837520 con 0x56165f836b30 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 --2- 172.21.15.110:0/3461118324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f836f50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14b8015f50 tx=0x7f14b803c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aadc752633e256 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f13a2ffd640 1 -- 172.21.15.110:0/3461118324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b8039e60 con 0x56165f836b30 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f13a2ffd640 1 -- 172.21.15.110:0/3461118324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b804c110 con 0x56165f836b30 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f13a2ffd640 1 -- 172.21.15.110:0/3461118324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b80283a0 con 0x56165f836b30 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- 172.21.15.110:0/3461118324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 msgr2=0x56165f836f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 --2- 172.21.15.110:0/3461118324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f836f50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f14b8015f50 tx=0x7f14b803c2d0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- 172.21.15.110:0/3461118324 shutdown_connections 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 --2- 172.21.15.110:0/3461118324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f836f50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- 172.21.15.110:0/3461118324 >> 172.21.15.110:0/3461118324 conn(0x56165f832ec0 msgr2=0x56165f830f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- 172.21.15.110:0/3461118324 shutdown_connections 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- 172.21.15.110:0/3461118324 wait complete. 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 Processor -- start 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- start start 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f83e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14c18f7980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56165f837c90 con 0x56165f836b30 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f83e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f83e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50802/0 (socket says 172.21.15.110:50802) 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.717+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 learned_addr learned my addr 172.21.15.110:0/154438755 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56165f83e5a0 con 0x56165f836b30 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bfc13640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f83e060 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f14b803c310 tx=0x7f14b804a8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f13a17fa640 1 -- 172.21.15.110:0/154438755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b80493c0 con 0x56165f836b30 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56165f83e830 con 0x56165f836b30 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56165f841510 con 0x56165f836b30 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f13a17fa640 1 -- 172.21.15.110:0/154438755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b8049560 con 0x56165f836b30 2024-04-27T03:18:07.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f13a17fa640 1 -- 172.21.15.110:0/154438755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b8051060 con 0x56165f836b30 2024-04-27T03:18:07.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f13a17fa640 1 -- 172.21.15.110:0/154438755 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f14b8044070 con 0x56165f836b30 2024-04-27T03:18:07.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f13a17fa640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1388034650 0x7f1388036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f13a17fa640 1 -- 172.21.15.110:0/154438755 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f14b8041050 con 0x56165f836b30 2024-04-27T03:18:07.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bf412640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1388034650 0x7f1388036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f8496f0 0x56165f84bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f84c300 con 0x56165f8496f0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bec11640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f8496f0 0x56165f84bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bf412640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1388034650 0x7f1388036b10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006500 tx=0x7f14b00bfa40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bec11640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f8496f0 0x56165f84bbb0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f14b400e5a0 con 0x56165f8496f0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 realm 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f848b80 con 0x56165f8496f0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f14b400e5a0 con 0x56165f8496f0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f84ce00 con 0x56165f8496f0 2024-04-27T03:18:07.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f14b400e5a0 con 0x56165f8496f0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f84dbe0 0x56165f84ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f850660 con 0x56165f84dbe0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.721+0000 7f14bfc13640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f84dbe0 0x56165f84ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bfc13640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f84dbe0 0x56165f84ffd0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14c18f7980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f850eb0 con 0x56165f8496f0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f14b400e5a0 con 0x56165f8496f0 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14c18f7980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14c18f7980 20 searching for the correct realm 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56165f852c20 0x56165f855010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f852c20 2024-04-27T03:18:07.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bf412640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56165f852c20 0x56165f855010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bf412640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56165f852c20 0x56165f855010 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bfc13640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f14b8014030 0x7f14b801f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x7f14b8014030 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bec11640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f14b8014030 0x7f14b801f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bec11640 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f14b8014030 0x7f14b801f720 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b400e5a0 con 0x7f14b8014030 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.725+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b400e5a0 con 0x56165f8496f0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f852c20 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x7f14b8014030 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x7f14b8014030 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x7f14b8014030 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f14b4029040 con 0x7f14b8014030 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.729+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f852c20 2024-04-27T03:18:07.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f852c20 2024-04-27T03:18:07.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f852c20 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x7f14b8014030 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x7f14b8014030 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f8496f0 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x7f14b8014030 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x7f14b8014030 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f852c20 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x7f14b8014030 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b4029040 con 0x7f14b8014030 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.733+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f852c20 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56165f855690 con 0x56165f84dbe0 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f851830 con 0x56165f8496f0 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f14b4029040 con 0x56165f8496f0 2024-04-27T03:18:07.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f851440 con 0x56165f84dbe0 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f852120 con 0x56165f8496f0 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 zone default found 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 4 Realm: () 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 zonegroup default 2024-04-27T03:18:07.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.737+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f855b10 con 0x7f14b8014030 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x7f14b8014030 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f857930 con 0x56165f852c20 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f857d00 con 0x56165f852c20 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 20 started sync module instance, tier type = 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f856dd0 con 0x56165f84dbe0 2024-04-27T03:18:07.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94654091850448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f856dd0 con 0x56165f84dbe0 2024-04-27T03:18:07.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f858760 con 0x56165f8496f0 2024-04-27T03:18:07.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94654091854512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f8586d0 con 0x56165f8496f0 2024-04-27T03:18:07.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.741+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f859660 con 0x56165f84dbe0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94654091850448] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94654091854512] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94654091856720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f859170 con 0x56165f84dbe0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f859f30 con 0x7f14b8014030 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x7f14b8014030 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94654091860608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f859170 con 0x7f14b8014030 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85a9b0 con 0x56165f84dbe0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94654091860608] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x7f14b8014030 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.745+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94654091856720] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94654091847968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85a4c0 con 0x56165f84dbe0 2024-04-27T03:18:07.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85b250 con 0x56165f852c20 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94654091865504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85a4c0 con 0x56165f852c20 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85c0c0 con 0x56165f852c20 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94654091847968] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94654091865504] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94654091861664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85c0c0 con 0x56165f852c20 2024-04-27T03:18:07.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85c910 con 0x7f14b8014030 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94654091861664] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'51 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x7f14b8014030 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94654091869712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85c7e0 con 0x7f14b8014030 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 20 add_watcher() i=0 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 20 add_watcher() i=1 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 20 add_watcher() i=2 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 20 add_watcher() i=3 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 20 add_watcher() i=4 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 20 add_watcher() i=5 2024-04-27T03:18:07.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.749+0000 7f14c18f7980 20 add_watcher() i=6 2024-04-27T03:18:07.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94654091869712] v27'52 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x7f14b8014030 2024-04-27T03:18:07.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 20 add_watcher() i=7 2024-04-27T03:18:07.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:07.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f85fdc0 con 0x56165f8496f0 2024-04-27T03:18:07.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94654091870832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f85d280 con 0x56165f8496f0 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94654091870832] v27'74 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f1375fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 5 note: GC not initialized 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f1374fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 20 Filter name: none 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f12ee7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:07.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.753+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f885880 con 0x56165f8496f0 2024-04-27T03:18:07.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~502 out=502b,getxattrs out=75b] v0'0 uv4 ondisk = 0) ==== 231+0+625 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 20 rados_obj.operate() r=0 bl.length=502 2024-04-27T03:18:07.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:07.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:07.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f88bf10 con 0x56165f8496f0 2024-04-27T03:18:07.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [delete,create,call,call,writefull 0~391] v27'5 uv5 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94654092065152 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f8875e0 con 0x56165f84dbe0 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94654091847968 notify 115964116997 ret 0) ==== 609+0+0 (crc 0 0 0) 0x56165f841510 con 0x56165f84dbe0 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f13a2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94654091847968 notifier 4345 bl.length()=567 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94654092065152 out=8b] v0'0 uv51 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f13a2ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f13a2ffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f13a2ffd640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f1384003eb0 con 0x56165f84dbe0 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== watch-notify(notify_complete (2) cookie 94654092065152 notify 115964116997 ret 0) ==== 42+0+48 (crc 0 0 0) 0x56165f83e830 con 0x56165f84dbe0 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:66 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f88b980 con 0x56165f852c20 2024-04-27T03:18:07.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(66 p5YnriCv1nAtykxBrupQ [delete] v27'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.757+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [notify cookie 94654092065152 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56165f887130 con 0x56165f84dbe0 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94654091856720 notify 115964116998 ret 0) ==== 212+0+0 (crc 0 0 0) 0x56165f83e830 con 0x56165f84dbe0 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f13a2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116998 cookie 94654091856720 notifier 4345 bl.length()=170 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(67 notify.2 [notify cookie 94654092065152 out=8b] v0'0 uv51 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f13a2ffd640 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f1384003eb0 con 0x56165f84dbe0 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== watch-notify(notify_complete (2) cookie 94654092065152 notify 115964116998 ret 0) ==== 42+0+48 (crc 0 0 0) 0x56165f83e5a0 con 0x56165f84dbe0 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(68 notify.2 [notify-ack] v0'0 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.769 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:07.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f1375fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.761+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94654091870832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f8875e0 con 0x56165f8496f0 2024-04-27T03:18:07.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(69 data_loggenerations_metadata [watch unwatch cookie 94654091870832] v27'75 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94654091850448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f887130 con 0x56165f84dbe0 2024-04-27T03:18:07.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(70 notify.0 [watch unwatch cookie 94654091850448] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14c18f7980 20 remove_watcher() i=0 2024-04-27T03:18:07.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14c18f7980 2 removed watcher, disabling cache 2024-04-27T03:18:07.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:71 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94654091854512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f8572d0 con 0x56165f8496f0 2024-04-27T03:18:07.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(71 notify.1 [watch unwatch cookie 94654091854512] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x56165f8496f0 2024-04-27T03:18:07.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14c18f7980 20 remove_watcher() i=1 2024-04-27T03:18:07.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.765+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94654091856720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f8572d0 con 0x56165f84dbe0 2024-04-27T03:18:07.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.769+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(72 notify.2 [watch unwatch cookie 94654091856720] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.769+0000 7f14c18f7980 20 remove_watcher() i=2 2024-04-27T03:18:07.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.769+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94654091860608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f8572d0 con 0x7f14b8014030 2024-04-27T03:18:07.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.769+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(73 notify.3 [watch unwatch cookie 94654091860608] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x7f14b8014030 2024-04-27T03:18:07.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.769+0000 7f14c18f7980 20 remove_watcher() i=3 2024-04-27T03:18:07.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.769+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94654091847968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f859a80 con 0x56165f84dbe0 2024-04-27T03:18:07.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14bfc13640 1 -- 172.21.15.110:0/154438755 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(74 notify.4 [watch unwatch cookie 94654091847968] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b8038020 con 0x56165f84dbe0 2024-04-27T03:18:07.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 20 remove_watcher() i=4 2024-04-27T03:18:07.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94654091865504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f8572d0 con 0x56165f852c20 2024-04-27T03:18:07.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(75 notify.5 [watch unwatch cookie 94654091865504] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 20 remove_watcher() i=5 2024-04-27T03:18:07.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94654091861664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f856920 con 0x56165f852c20 2024-04-27T03:18:07.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14bf412640 1 -- 172.21.15.110:0/154438755 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(76 notify.6 [watch unwatch cookie 94654091861664] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b00b3040 con 0x56165f852c20 2024-04-27T03:18:07.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 20 remove_watcher() i=6 2024-04-27T03:18:07.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:77 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94654091869712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56165f8572d0 con 0x7f14b8014030 2024-04-27T03:18:07.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(77 notify.7 [watch unwatch cookie 94654091869712] v27'53 uv51 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f14b403c1c0 con 0x7f14b8014030 2024-04-27T03:18:07.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 20 remove_watcher() i=7 2024-04-27T03:18:07.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f84dbe0 msgr2=0x56165f84ffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f84dbe0 0x56165f84ffd0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f8496f0 msgr2=0x56165f84bbb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f8496f0 0x56165f84bbb0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f14b8014030 msgr2=0x7f14b801f720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f14b8014030 0x7f14b801f720 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56165f852c20 msgr2=0x56165f855010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56165f852c20 0x56165f855010 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1388034650 msgr2=0x7f1388036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1388034650 0x7f1388036b10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f14b0006500 tx=0x7f14b00bfa40 comp rx=0 tx=0).stop 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 msgr2=0x56165f83e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/154438755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f836b30 0x56165f83e060 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f14b803c310 tx=0x7f14b804a8a0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 reap_dead start 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14bec11640 1 -- 172.21.15.110:0/154438755 reap_dead start 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 shutdown_connections 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 >> 172.21.15.110:0/154438755 conn(0x56165f832ec0 msgr2=0x56165f8383b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 shutdown_connections 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/154438755 wait complete. 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1398034650 msgr2=0x7f1398036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1398034650 0x7f1398036b10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x56165f819dd0 tx=0x7f14b00e2d30 comp rx=0 tx=0).stop 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f829980 msgr2=0x56165f829da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f829980 0x56165f829da0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f14b800dfc0 tx=0x7f14b80178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 shutdown_connections 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1398034650 0x7f1398036b10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 --2- 172.21.15.110:0/2797047100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f829980 0x56165f829da0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 >> 172.21.15.110:0/2797047100 conn(0x56165f81e790 msgr2=0x56165f81f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 shutdown_connections 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.773+0000 7f14c18f7980 1 -- 172.21.15.110:0/2797047100 wait complete. 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f816da0 msgr2=0x56165f819190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f816da0 0x56165f819190 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f71ba80 msgr2=0x56165f815940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f71ba80 0x56165f815940 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f13a4034650 msgr2=0x7f13a4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f13a4034650 0x7f13a4036b10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000f70 tx=0x7f14b40012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 msgr2=0x56165f690520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f690520 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f14b0018eb0 tx=0x7f14b0018020 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 shutdown_connections 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56165f816da0 0x56165f819190 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56165f71ba80 0x56165f815940 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f13a4034650 0x7f13a4036b10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 --2- 172.21.15.110:0/877574324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56165f71b160 0x56165f690520 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 >> 172.21.15.110:0/877574324 conn(0x56165f716a30 msgr2=0x56165f801180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 shutdown_connections 2024-04-27T03:18:07.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.777+0000 7f14c18f7980 1 -- 172.21.15.110:0/877574324 wait complete. 2024-04-27T03:18:07.792 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:07.793 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'create', '--subuser', 'foo:foo1', '--access', 'full'] 2024-04-27T03:18:07.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'create', '--subuser', 'foo:foo1', '--access', 'full'] 2024-04-27T03:18:07.793 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser create --subuser foo:foo1 --access full 2024-04-27T03:18:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dcf2c9980 1 Processor -- start 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dcf2c9980 1 -- start start 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dcf2c9980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556324cca180 0x556324cca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dcf2c9980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x556324b8a0d0 con 0x556324ccaaa0 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dcf2c9980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x556324b7f430 con 0x556324cca180 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dccde4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556324cca180 0x556324cca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dccde4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556324cca180 0x556324cca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50810/0 (socket says 172.21.15.110:50810) 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dccde4640 1 -- 172.21.15.110:0/1855255821 learned_addr learned my addr 172.21.15.110:0/1855255821 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.857+0000 7f3dcc5e3640 1 --1- 172.21.15.110:0/1855255821 >> v1:172.21.15.77:6789/0 conn(0x556324ccaaa0 0x556324ccb4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34666/0 (socket says 172.21.15.110:34666) 2024-04-27T03:18:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dccde4640 1 -- 172.21.15.110:0/1855255821 >> v1:172.21.15.77:6789/0 conn(0x556324ccaaa0 legacy=0x556324ccb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dccde4640 1 -- 172.21.15.110:0/1855255821 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x556324b78510 con 0x556324cca180 2024-04-27T03:18:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dccde4640 1 --2- 172.21.15.110:0/1855255821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324cca560 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00a5c0 tx=0x7f3dbc01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840352bcdc4eb2f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcbde2640 1 -- 172.21.15.110:0/1855255821 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc0a6070 con 0x556324cca180 2024-04-27T03:18:07.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcbde2640 1 -- 172.21.15.110:0/1855255821 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0af070 con 0x556324cca180 2024-04-27T03:18:07.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcbde2640 1 -- 172.21.15.110:0/1855255821 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc0aa5c0 con 0x556324cca180 2024-04-27T03:18:07.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/1855255821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 msgr2=0x556324cca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/1855255821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324cca560 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00a5c0 tx=0x7f3dbc01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:07.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/1855255821 shutdown_connections 2024-04-27T03:18:07.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/1855255821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324cca560 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/1855255821 >> 172.21.15.110:0/1855255821 conn(0x556324cc5a50 msgr2=0x556324cc7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/1855255821 shutdown_connections 2024-04-27T03:18:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.861+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/1855255821 wait complete. 2024-04-27T03:18:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 Processor -- start 2024-04-27T03:18:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- start start 2024-04-27T03:18:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324db6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556324bce2d0 con 0x556324cca180 2024-04-27T03:18:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324db6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324db6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50822/0 (socket says 172.21.15.110:50822) 2024-04-27T03:18:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 -- 172.21.15.110:0/148729036 learned_addr learned my addr 172.21.15.110:0/148729036 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 -- 172.21.15.110:0/148729036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556324b54ae0 con 0x556324cca180 2024-04-27T03:18:07.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 --2- 172.21.15.110:0/148729036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324db6b30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0199f0 tx=0x7f3dbc003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477f1e5ab74a313 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3cbae0d640 1 -- 172.21.15.110:0/148729036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc0aaa00 con 0x556324cca180 2024-04-27T03:18:07.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3cbae0d640 1 -- 172.21.15.110:0/148729036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0aaba0 con 0x556324cca180 2024-04-27T03:18:07.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3cbae0d640 1 -- 172.21.15.110:0/148729036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc0bd3f0 con 0x556324cca180 2024-04-27T03:18:07.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/148729036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 msgr2=0x556324db6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/148729036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324db6b30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0199f0 tx=0x7f3dbc003030 comp rx=0 tx=0).stop 2024-04-27T03:18:07.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/148729036 shutdown_connections 2024-04-27T03:18:07.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/148729036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324db6b30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/148729036 >> 172.21.15.110:0/148729036 conn(0x556324cc5a50 msgr2=0x556324c3e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/148729036 shutdown_connections 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/148729036 wait complete. 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 Processor -- start 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- start start 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324dbdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dcf2c9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556324cd4220 con 0x556324cca180 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324dbdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324dbdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50838/0 (socket says 172.21.15.110:50838) 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.877+0000 7f3dccde4640 1 -- 172.21.15.110:0/2521046460 learned_addr learned my addr 172.21.15.110:0/2521046460 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dccde4640 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556324dbe030 con 0x556324cca180 2024-04-27T03:18:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dccde4640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324dbdaf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc01b380 tx=0x7f3dbc0180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3cb960a640 1 -- 172.21.15.110:0/2521046460 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc0bd7c0 con 0x556324cca180 2024-04-27T03:18:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556324dbe2c0 con 0x556324cca180 2024-04-27T03:18:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3cb960a640 1 -- 172.21.15.110:0/2521046460 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0bd960 con 0x556324cca180 2024-04-27T03:18:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3cb960a640 1 -- 172.21.15.110:0/2521046460 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dbc0bdc40 con 0x556324cca180 2024-04-27T03:18:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556324dc0f40 con 0x556324cca180 2024-04-27T03:18:07.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3cb960a640 1 -- 172.21.15.110:0/2521046460 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f3dbc0ce020 con 0x556324cca180 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3cb960a640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3cb4034650 0x7f3cb4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3cb960a640 1 -- 172.21.15.110:0/2521046460 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f3dbc0cc020 con 0x556324cca180 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcc5e3640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3cb4034650 0x7f3cb4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324ccaaa0 0x556324dc4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324dc4ff0 con 0x556324ccaaa0 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324ccaaa0 0x556324dc4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcc5e3640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3cb4034650 0x7f3cb4036b10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000f70 tx=0x7f3dc0001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324ccaaa0 0x556324dc4a20 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/2521046460 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3dc400a4d0 con 0x556324ccaaa0 2024-04-27T03:18:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dc5de0 0x556324dc81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324dc8800 con 0x556324dc5de0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.881+0000 7f3dccde4640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dc5de0 0x556324dc81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dccde4640 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dc5de0 0x556324dc81d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dccde4640 1 -- 172.21.15.110:0/2521046460 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f3dbc0c6cd0 con 0x556324dc5de0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324dca0b0 con 0x556324ccaaa0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/2521046460 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f3dc400a4d0 con 0x556324ccaaa0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324dc92e0 con 0x556324dc5de0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dccde4640 1 -- 172.21.15.110:0/2521046460 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f3dbc0c6cd0 con 0x556324dc5de0 2024-04-27T03:18:07.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324dc96d0 con 0x556324ccaaa0 2024-04-27T03:18:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/2521046460 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f3dc400a4d0 con 0x556324ccaaa0 2024-04-27T03:18:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcf2c9980 1 Processor -- start 2024-04-27T03:18:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcf2c9980 1 -- start start 2024-04-27T03:18:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcf2c9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324dd89b0 0x556324dd8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcf2c9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556324bd2310 con 0x556324dd89b0 2024-04-27T03:18:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcd5e5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324dd89b0 0x556324dd8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcd5e5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324dd89b0 0x556324dd8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50846/0 (socket says 172.21.15.110:50846) 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/609084480 learned_addr learned my addr 172.21.15.110:0/609084480 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.885+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/609084480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556324dd9310 con 0x556324dd89b0 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324dd89b0 0x556324dd8dd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400dfc0 tx=0x7f3dc40178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3cb27fc640 1 -- 172.21.15.110:0/609084480 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc40157d0 con 0x556324dd89b0 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556324dd9db0 con 0x556324dd89b0 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3cb27fc640 1 -- 172.21.15.110:0/609084480 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc4015970 con 0x556324dd89b0 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3cb27fc640 1 -- 172.21.15.110:0/609084480 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc4015c70 con 0x556324dd89b0 2024-04-27T03:18:07.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556324dda280 con 0x556324dd89b0 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3cb27fc640 1 -- 172.21.15.110:0/609084480 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f3dc4029440 con 0x556324dd89b0 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3cb27fc640 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3ca4034650 0x7f3ca4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3cb27fc640 1 -- 172.21.15.110:0/609084480 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f3dc400e040 con 0x556324dd89b0 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dccde4640 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3ca4034650 0x7f3ca4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcf2c9980 1 Processor -- start 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcf2c9980 1 -- start start 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dccde4640 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3ca4034650 0x7f3ca4036b10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x556324dc8e00 tx=0x7f3dbc0e3690 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcf2c9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324db2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcf2c9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556324db71c0 con 0x556324db2840 2024-04-27T03:18:07.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcd5e5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324db2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcd5e5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324db2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50860/0 (socket says 172.21.15.110:50860) 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/522232864 learned_addr learned my addr 172.21.15.110:0/522232864 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/522232864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556324db1070 con 0x556324db2840 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/522232864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324db2c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4015f50 tx=0x7f3dc401e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaedbd7c90b33e9a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3ca3fff640 1 -- 172.21.15.110:0/522232864 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc403b070 con 0x556324db2840 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3ca3fff640 1 -- 172.21.15.110:0/522232864 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc4039e50 con 0x556324db2840 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.889+0000 7f3ca3fff640 1 -- 172.21.15.110:0/522232864 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc4011cc0 con 0x556324db2840 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/522232864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 msgr2=0x556324db2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/522232864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324db2c60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4015f50 tx=0x7f3dc401e4b0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/522232864 shutdown_connections 2024-04-27T03:18:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/522232864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324db2c60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/522232864 >> 172.21.15.110:0/522232864 conn(0x556324de1ef0 msgr2=0x556324ddff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/522232864 shutdown_connections 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/522232864 wait complete. 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 Processor -- start 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- start start 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324def210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556324db1510 con 0x556324db2840 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcd5e5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324def210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcd5e5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324def210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50862/0 (socket says 172.21.15.110:50862) 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 learned_addr learned my addr 172.21.15.110:0/3667793943 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556324def750 con 0x556324db2840 2024-04-27T03:18:07.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324def210 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4049780 tx=0x7f3dc404a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3ca27fc640 1 -- 172.21.15.110:0/3667793943 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc404c110 con 0x556324db2840 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3ca27fc640 1 -- 172.21.15.110:0/3667793943 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc40133b0 con 0x556324db2840 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3ca27fc640 1 -- 172.21.15.110:0/3667793943 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3dc404c2f0 con 0x556324db2840 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556324def9e0 con 0x556324db2840 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556324df26c0 con 0x556324db2840 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3ca27fc640 1 -- 172.21.15.110:0/3667793943 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f3dc4044070 con 0x556324db2840 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3ca27fc640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3c980346a0 0x7f3c98036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3ca27fc640 1 -- 172.21.15.110:0/3667793943 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f3dc4041050 con 0x556324db2840 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dccde4640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3c980346a0 0x7f3c98036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324dfa8b0 0x556324dfcd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.893+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324dfd4c0 con 0x556324dfa8b0 2024-04-27T03:18:07.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcc5e3640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324dfa8b0 0x556324dfcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dccde4640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3c980346a0 0x7f3c98036b60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005270 tx=0x7f3dbc0ee000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcc5e3640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324dfa8b0 0x556324dfcd70 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3dc000f620 con 0x556324dfa8b0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 realm 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324df9d00 con 0x556324dfa8b0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3dc000f620 con 0x556324dfa8b0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324dfdfc0 con 0x556324dfa8b0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3dc000f620 con 0x556324dfa8b0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dfeda0 0x556324e01190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e01810 con 0x556324dfeda0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dfeda0 0x556324e01190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dfeda0 0x556324e01190 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e02060 con 0x556324dfa8b0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f3dc000f620 con 0x556324dfa8b0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 20 searching for the correct realm 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556324e03dd0 0x556324e061c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324e03dd0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dccde4640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556324e03dd0 0x556324e061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dccde4640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556324e03dd0 0x556324e061c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.897+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcd5e5640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3dc4014030 0x7f3dc402ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:07.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x7f3dc4014030 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3dc4014030 0x7f3dc402ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3dc4014030 0x7f3dc402ba20 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc000f620 con 0x7f3dc4014030 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc000f620 con 0x556324dfa8b0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324e03dd0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.901+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x7f3dc4014030 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x7f3dc4014030 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x7f3dc4014030 2024-04-27T03:18:07.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f3dc002b040 con 0x7f3dc4014030 2024-04-27T03:18:07.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324e03dd0 2024-04-27T03:18:07.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324e03dd0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324e03dd0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.905+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x7f3dc4014030 2024-04-27T03:18:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x7f3dc4014030 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfa8b0 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc002b040 con 0x556324dfa8b0 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x7f3dc4014030 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324e03dd0 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x7f3dc4014030 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324e03dd0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556324e06840 con 0x556324dfeda0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcf2c9980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcf2c9980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcf2c9980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcf2c9980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcf2c9980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.909+0000 7f3dcf2c9980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e02a10 con 0x556324dfa8b0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e02650 con 0x556324dfeda0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e03370 con 0x556324dfa8b0 2024-04-27T03:18:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 zone default found 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 4 Realm: () 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 zonegroup default 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e07150 con 0x7f3dc4014030 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e08f40 con 0x556324e03dd0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e09330 con 0x556324e03dd0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 started sync module instance, tier type = 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e083e0 con 0x556324dfeda0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93884308817872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e08040 con 0x556324dfeda0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.913+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0a130 con 0x556324dfa8b0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93884308823104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0a060 con 0x556324dfa8b0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0b050 con 0x556324dfeda0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93884308817872] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93884308823104] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93884308825376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0b010 con 0x556324dfeda0 2024-04-27T03:18:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0b8f0 con 0x7f3dc4014030 2024-04-27T03:18:07.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93884308829248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0b010 con 0x7f3dc4014030 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.917+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0c370 con 0x556324dfeda0 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93884308829248] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93884308825376] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93884308826944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0be80 con 0x556324dfeda0 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0cc40 con 0x556324e03dd0 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93884308834192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0be80 con 0x556324e03dd0 2024-04-27T03:18:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0dab0 con 0x556324e03dd0 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93884308826944] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93884308830304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0dab0 con 0x556324e03dd0 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0e4b0 con 0x7f3dc4014030 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.921+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 93884308834192] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'54 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93884308830304] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93884308836320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0d600 con 0x7f3dc4014030 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=0 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=1 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=2 2024-04-27T03:18:07.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=3 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=4 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=5 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=6 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93884308836320] v27'55 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 20 add_watcher() i=7 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e0d600 con 0x556324dfa8b0 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.925+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93884308840384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e0d600 con 0x556324dfa8b0 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93884308840384] v27'76 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3c82fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 5 note: GC not initialized 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3c81fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 Filter name: none 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3bf37fe640 20 reqs_thread_entry: start 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e34bd0 con 0x556324dfa8b0 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv5 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 cache get: name=default.rgw.meta+users.swift+foo:foo1 : miss 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.8 8:17d68d04:users.swift::foo%3afoo1:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e35640 con 0x556324dfa8b0 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo:foo1 [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 194+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 cache put: name=default.rgw.meta+users.swift+foo:foo1 info.flags=0x0 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 adding default.rgw.meta+users.swift+foo:foo1 to cache LRU end 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 cache get: name=default.rgw.meta+users.swift+foo:foo1 : hit (negative entry) 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~556 in=556b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e3b010 con 0x556324dfa8b0 2024-04-27T03:18:07.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(64 foo [delete,create,call,call,writefull 0~556] v27'6 uv6 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:07.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.929+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify cookie 93884309023360 in=744b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e36650 con 0x556324dfeda0 2024-04-27T03:18:07.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 93884308826944 notify 115964116999 ret 0) ==== 774+0+0 (crc 0 0 0) 0x556324df26c0 con 0x556324dfeda0 2024-04-27T03:18:07.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca3fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116999 cookie 93884308826944 notifier 4352 bl.length()=732 2024-04-27T03:18:07.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca3fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:07.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca3fff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:07.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca3fff640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f3c90003eb0 con 0x556324dfeda0 2024-04-27T03:18:07.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(65 notify.4 [notify cookie 93884309023360 out=8b] v0'0 uv54 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(66 notify.4 [notify-ack] v0'0 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 93884309023360 notify 115964116999 ret 0) ==== 42+0+48 (crc 0 0 0) 0x556324def9e0 con 0x556324dfeda0 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 8.8 8:17d68d04:users.swift::foo%3afoo1:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e3aa80 con 0x556324dfa8b0 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(67 foo:foo1 [delete,create,writefull 0~7] v27'1 uv1 ondisk = 0) ==== 236+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcf2c9980 10 cache put: name=default.rgw.meta+users.swift+foo:foo1 info.flags=0x7 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcf2c9980 10 moving default.rgw.meta+users.swift+foo:foo1 to cache LRU end 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcf2c9980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.swift:foo:foo1, ofs0, ns] 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.1f 7:f95f44c2:::notify.0:head [notify cookie 93884309023360 in=178b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556324e37380 con 0x556324dfeda0 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 93884308817872 notify 115964117000 ret 0) ==== 208+0+0 (crc 0 0 0) 0x556324def9e0 con 0x556324dfeda0 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117000 cookie 93884308817872 notifier 4352 bl.length()=166 2024-04-27T03:18:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca37fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.swift+foo:foo1 info.flags=0x7 2024-04-27T03:18:07.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca37fe640 10 rgw watcher librados: moving default.rgw.meta+users.swift+foo:foo1 to cache LRU end 2024-04-27T03:18:07.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3ca37fe640 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f3c00003410 con 0x556324dfeda0 2024-04-27T03:18:07.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.933+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(68 notify.0 [notify cookie 93884309023360 out=8b] v0'0 uv54 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.937+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== watch-notify(notify_complete (2) cookie 93884309023360 notify 115964117000 ret 0) ==== 42+0+48 (crc 0 0 0) 0x556324def750 con 0x556324dfeda0 2024-04-27T03:18:07.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.937+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(69 notify.0 [notify-ack] v0'0 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.948 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo1","permissions":"full-control"}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo1","secret_key":"ADujP0qQeqPH3P3EGvx8nXXv2M9j7UQYbxpEhBAB","active":true,"create_date":"2024-04-27T03:18:07.931578Z"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.937+0000 7f3c82fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.937+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93884308840384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x556324dfa8b0 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.937+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 93884308840384] v27'77 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.937+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93884308817872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x556324dfeda0 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.941+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 93884308817872] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.941+0000 7f3dcf2c9980 20 remove_watcher() i=0 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.941+0000 7f3dcf2c9980 2 removed watcher, disabling cache 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.941+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93884308823104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x556324dfa8b0 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.941+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 93884308823104] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x556324dfa8b0 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.941+0000 7f3dcf2c9980 20 remove_watcher() i=1 2024-04-27T03:18:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.941+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93884308825376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x556324dfeda0 2024-04-27T03:18:07.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 93884308825376] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcf2c9980 20 remove_watcher() i=2 2024-04-27T03:18:07.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93884308829248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x7f3dc4014030 2024-04-27T03:18:07.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 93884308829248] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcf2c9980 20 remove_watcher() i=3 2024-04-27T03:18:07.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93884308826944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x556324dfeda0 2024-04-27T03:18:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcd5e5640 1 -- 172.21.15.110:0/3667793943 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 93884308826944] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc4038020 con 0x556324dfeda0 2024-04-27T03:18:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcf2c9980 20 remove_watcher() i=4 2024-04-27T03:18:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.945+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93884308834192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x556324e03dd0 2024-04-27T03:18:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 93884308834192] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 20 remove_watcher() i=5 2024-04-27T03:18:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93884308830304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x556324e03dd0 2024-04-27T03:18:07.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dccde4640 1 -- 172.21.15.110:0/3667793943 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 93884308830304] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dbc0ee070 con 0x556324e03dd0 2024-04-27T03:18:07.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 20 remove_watcher() i=6 2024-04-27T03:18:07.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93884308836320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556324e32900 con 0x7f3dc4014030 2024-04-27T03:18:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 93884308836320] v27'56 uv54 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3dc003e040 con 0x7f3dc4014030 2024-04-27T03:18:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 20 remove_watcher() i=7 2024-04-27T03:18:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dfeda0 msgr2=0x556324e01190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dfeda0 0x556324e01190 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324dfa8b0 msgr2=0x556324dfcd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324dfa8b0 0x556324dfcd70 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3dc4014030 msgr2=0x7f3dc402ba20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3dc4014030 0x7f3dc402ba20 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556324e03dd0 msgr2=0x556324e061c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556324e03dd0 0x556324e061c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3c980346a0 msgr2=0x7f3c98036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3c980346a0 0x7f3c98036b60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc005270 tx=0x7f3dbc0ee000 comp rx=0 tx=0).stop 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 msgr2=0x556324def210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcc5e3640 1 -- 172.21.15.110:0/3667793943 reap_dead start 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324def210 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4049780 tx=0x7f3dc404a890 comp rx=0 tx=0).stop 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 shutdown_connections 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/3667793943 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324db2840 0x556324def210 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 >> 172.21.15.110:0/3667793943 conn(0x556324de1ef0 msgr2=0x556324ddf550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 shutdown_connections 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/3667793943 wait complete. 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3ca4034650 msgr2=0x7f3ca4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3ca4034650 0x7f3ca4036b10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x556324dc8e00 tx=0x7f3dbc0e3690 comp rx=0 tx=0).stop 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324dd89b0 msgr2=0x556324dd8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324dd89b0 0x556324dd8dd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400dfc0 tx=0x7f3dc40178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 shutdown_connections 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3ca4034650 0x7f3ca4036b10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/609084480 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324dd89b0 0x556324dd8dd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 >> 172.21.15.110:0/609084480 conn(0x556324dcd7c0 msgr2=0x556324dce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 shutdown_connections 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.949+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/609084480 wait complete. 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dc5de0 msgr2=0x556324dc81d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dc5de0 0x556324dc81d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324ccaaa0 msgr2=0x556324dc4a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324ccaaa0 0x556324dc4a20 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3cb4034650 msgr2=0x7f3cb4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3cb4034650 0x7f3cb4036b10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0000f70 tx=0x7f3dc0001380 comp rx=0 tx=0).stop 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 msgr2=0x556324dbdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324dbdaf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc01b380 tx=0x7f3dbc0180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 shutdown_connections 2024-04-27T03:18:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556324dc5de0 0x556324dc81d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556324ccaaa0 0x556324dc4a20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3cb4034650 0x7f3cb4036b10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 --2- 172.21.15.110:0/2521046460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556324cca180 0x556324dbdaf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 >> 172.21.15.110:0/2521046460 conn(0x556324cc5a50 msgr2=0x556324db00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 shutdown_connections 2024-04-27T03:18:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:07.953+0000 7f3dcf2c9980 1 -- 172.21.15.110:0/2521046460 wait complete. 2024-04-27T03:18:07.969 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'ADujP0qQeqPH3P3EGvx8nXXv2M9j7UQYbxpEhBAB', 'active': True, 'create_date': '2024-04-27T03:18:07.931578Z'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:07.969 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'modify', '--subuser', 'foo:foo1', '--secret', 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', '--key-type', 'swift'] 2024-04-27T03:18:07.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'modify', '--subuser', 'foo:foo1', '--secret', 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', '--key-type', 'swift'] 2024-04-27T03:18:07.970 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser modify --subuser foo:foo1 --secret gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL --key-type swift 2024-04-27T03:18:08.050 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:08.050 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.041+0000 7f2e7146c980 1 Processor -- start 2024-04-27T03:18:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.041+0000 7f2e7146c980 1 -- start start 2024-04-27T03:18:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.041+0000 7f2e7146c980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55707e95ca00 0x55707e95edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e7146c980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55707e8150d0 con 0x55707e8ca7f0 2024-04-27T03:18:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e7146c980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55707e80a430 con 0x55707e95ca00 2024-04-27T03:18:08.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6e786640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55707e95ca00 0x55707e95edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6e786640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55707e95ca00 0x55707e95edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50878/0 (socket says 172.21.15.110:50878) 2024-04-27T03:18:08.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6ef87640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55707e8ca7f0 0x55707e95b2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34672/0 (socket says 172.21.15.110:34672) 2024-04-27T03:18:08.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6e786640 1 -- 172.21.15.110:0/1242215354 learned_addr learned my addr 172.21.15.110:0/1242215354 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6e786640 1 -- 172.21.15.110:0/1242215354 >> v1:172.21.15.77:6789/0 conn(0x55707e8ca7f0 legacy=0x55707e95b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6e786640 1 -- 172.21.15.110:0/1242215354 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707e803510 con 0x55707e95ca00 2024-04-27T03:18:08.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6e786640 1 --2- 172.21.15.110:0/1242215354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e95ca00 0x55707e95edf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400a5c0 tx=0x7f2e6401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa57a3c3b6267595 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6df85640 1 -- 172.21.15.110:0/1242215354 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e640a6070 con 0x55707e95ca00 2024-04-27T03:18:08.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6df85640 1 -- 172.21.15.110:0/1242215354 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e640af070 con 0x55707e95ca00 2024-04-27T03:18:08.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.045+0000 7f2e6df85640 1 -- 172.21.15.110:0/1242215354 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e640aa5c0 con 0x55707e95ca00 2024-04-27T03:18:08.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.049+0000 7f2e7146c980 1 -- 172.21.15.110:0/1242215354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e95ca00 msgr2=0x55707e95edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.049+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1242215354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e95ca00 0x55707e95edf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2e6400a5c0 tx=0x7f2e6401b340 comp rx=0 tx=0).stop 2024-04-27T03:18:08.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.049+0000 7f2e7146c980 1 -- 172.21.15.110:0/1242215354 shutdown_connections 2024-04-27T03:18:08.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.049+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1242215354 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e95ca00 0x55707e95edf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.049+0000 7f2e7146c980 1 -- 172.21.15.110:0/1242215354 >> 172.21.15.110:0/1242215354 conn(0x55707e9509c0 msgr2=0x55707e952de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.049+0000 7f2e7146c980 1 -- 172.21.15.110:0/1242215354 shutdown_connections 2024-04-27T03:18:08.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.049+0000 7f2e7146c980 1 -- 172.21.15.110:0/1242215354 wait complete. 2024-04-27T03:18:08.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 Processor -- start 2024-04-27T03:18:08.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 -- start start 2024-04-27T03:18:08.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea43620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55707e8592d0 con 0x55707e8ca7f0 2024-04-27T03:18:08.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e6ef87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea43620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e6ef87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea43620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50894/0 (socket says 172.21.15.110:50894) 2024-04-27T03:18:08.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e6ef87640 1 -- 172.21.15.110:0/3071011729 learned_addr learned my addr 172.21.15.110:0/3071011729 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e6ef87640 1 -- 172.21.15.110:0/3071011729 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707e7dfae0 con 0x55707e8ca7f0 2024-04-27T03:18:08.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/3071011729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea43620 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000b240 tx=0x7f2e6000b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332f62d48b4606ba server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2d5ce09640 1 -- 172.21.15.110:0/3071011729 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e6000e700 con 0x55707e8ca7f0 2024-04-27T03:18:08.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2d5ce09640 1 -- 172.21.15.110:0/3071011729 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6000e8a0 con 0x55707e8ca7f0 2024-04-27T03:18:08.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2d5ce09640 1 -- 172.21.15.110:0/3071011729 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e6000eb80 con 0x55707e8ca7f0 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 -- 172.21.15.110:0/3071011729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 msgr2=0x55707ea43620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3071011729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea43620 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000b240 tx=0x7f2e6000b700 comp rx=0 tx=0).stop 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 -- 172.21.15.110:0/3071011729 shutdown_connections 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3071011729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea43620 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 -- 172.21.15.110:0/3071011729 >> 172.21.15.110:0/3071011729 conn(0x55707e9509c0 msgr2=0x55707ea3f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 -- 172.21.15.110:0/3071011729 shutdown_connections 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.061+0000 7f2e7146c980 1 -- 172.21.15.110:0/3071011729 wait complete. 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 Processor -- start 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 -- start start 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea4a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55707e961d90 con 0x55707e8ca7f0 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6ef87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea4a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6ef87640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea4a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50896/0 (socket says 172.21.15.110:50896) 2024-04-27T03:18:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6ef87640 1 -- 172.21.15.110:0/3418851769 learned_addr learned my addr 172.21.15.110:0/3418851769 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6ef87640 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707ea4ac60 con 0x55707e8ca7f0 2024-04-27T03:18:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea4a720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2e60000c10 tx=0x7f2e60000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2d5b606640 1 -- 172.21.15.110:0/3418851769 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e6000fa90 con 0x55707e8ca7f0 2024-04-27T03:18:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55707ea4aef0 con 0x55707e8ca7f0 2024-04-27T03:18:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2d5b606640 1 -- 172.21.15.110:0/3418851769 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6000fc50 con 0x55707e8ca7f0 2024-04-27T03:18:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2d5b606640 1 -- 172.21.15.110:0/3418851769 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e6002c060 con 0x55707e8ca7f0 2024-04-27T03:18:08.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707ea4dad0 con 0x55707e8ca7f0 2024-04-27T03:18:08.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2d5b606640 1 -- 172.21.15.110:0/3418851769 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2e60017070 con 0x55707e8ca7f0 2024-04-27T03:18:08.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2d5b606640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d54034650 0x7f2d54036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2d5b606640 1 -- 172.21.15.110:0/3418851769 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2e60015020 con 0x55707e8ca7f0 2024-04-27T03:18:08.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6e786640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d54034650 0x7f2d54036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707e95ca00 0x55707ea515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea51c40 con 0x55707e95ca00 2024-04-27T03:18:08.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6f788640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707e95ca00 0x55707ea515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.065+0000 7f2e6e786640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d54034650 0x7f2d54036b10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2e640ae000 tx=0x7f2e640a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6f788640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707e95ca00 0x55707ea515d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6f788640 1 -- 172.21.15.110:0/3418851769 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2e6800a4d0 con 0x55707e95ca00 2024-04-27T03:18:08.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea52a60 0x55707ea54e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea55490 con 0x55707ea52a60 2024-04-27T03:18:08.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea52a60 0x55707ea54e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea52a60 0x55707ea54e50 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6ef87640 1 -- 172.21.15.110:0/3418851769 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea52a60 2024-04-27T03:18:08.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea56d40 con 0x55707e95ca00 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6f788640 1 -- 172.21.15.110:0/3418851769 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f2e6800a4d0 con 0x55707e95ca00 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea55f70 con 0x55707ea52a60 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6ef87640 1 -- 172.21.15.110:0/3418851769 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea52a60 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea56360 con 0x55707e95ca00 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6f788640 1 -- 172.21.15.110:0/3418851769 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f2e6800a4d0 con 0x55707e95ca00 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 Processor -- start 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 -- start start 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea65680 0x55707ea65aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e7146c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55707e85d310 con 0x55707ea65680 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.069+0000 7f2e6f788640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea65680 0x55707ea65aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6f788640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea65680 0x55707ea65aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50906/0 (socket says 172.21.15.110:50906) 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6f788640 1 -- 172.21.15.110:0/253885145 learned_addr learned my addr 172.21.15.110:0/253885145 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6f788640 1 -- 172.21.15.110:0/253885145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707ea65fe0 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6f788640 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea65680 0x55707ea65aa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e680056d0 tx=0x7f2e68017820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2d58e01640 1 -- 172.21.15.110:0/253885145 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68012840 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2d58e01640 1 -- 172.21.15.110:0/253885145 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e680129e0 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55707ea66a80 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2d58e01640 1 -- 172.21.15.110:0/253885145 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68012ce0 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707ea66f50 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2d58e01640 1 -- 172.21.15.110:0/253885145 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2e68020440 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2d58e01640 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d480346f0 0x7f2d48036bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2d58e01640 1 -- 172.21.15.110:0/253885145 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2e6800e040 con 0x55707ea65680 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d480346f0 0x7f2d48036bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e7146c980 1 Processor -- start 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e7146c980 1 -- start start 2024-04-27T03:18:08.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d480346f0 0x7f2d48036bb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55707ea55a90 tx=0x7f2e6000ff30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e7146c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea72c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e7146c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55707ea43d70 con 0x55707ea72810 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6f788640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea72c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6f788640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea72c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50918/0 (socket says 172.21.15.110:50918) 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.073+0000 7f2e6f788640 1 -- 172.21.15.110:0/498655391 learned_addr learned my addr 172.21.15.110:0/498655391 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e6f788640 1 -- 172.21.15.110:0/498655391 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707ea73200 con 0x55707ea72810 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e6f788640 1 --2- 172.21.15.110:0/498655391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea72c30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e6801e870 tx=0x7f2e68014610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a436caa10189ab2c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2d527fc640 1 -- 172.21.15.110:0/498655391 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e6803c070 con 0x55707ea72810 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2d527fc640 1 -- 172.21.15.110:0/498655391 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6801f3a0 con 0x55707ea72810 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2d527fc640 1 -- 172.21.15.110:0/498655391 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68020730 con 0x55707ea72810 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- 172.21.15.110:0/498655391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 msgr2=0x55707ea72c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 --2- 172.21.15.110:0/498655391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea72c30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e6801e870 tx=0x7f2e68014610 comp rx=0 tx=0).stop 2024-04-27T03:18:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- 172.21.15.110:0/498655391 shutdown_connections 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 --2- 172.21.15.110:0/498655391 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea72c30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- 172.21.15.110:0/498655391 >> 172.21.15.110:0/498655391 conn(0x55707ea6eb50 msgr2=0x55707ea6cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- 172.21.15.110:0/498655391 shutdown_connections 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- 172.21.15.110:0/498655391 wait complete. 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 Processor -- start 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- start start 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea79d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55707ea73980 con 0x55707ea72810 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e6f788640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea79d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e6f788640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea79d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50932/0 (socket says 172.21.15.110:50932) 2024-04-27T03:18:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 learned_addr learned my addr 172.21.15.110:0/1246393875 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707ea7a270 con 0x55707ea72810 2024-04-27T03:18:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e6f788640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea79d30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2e6801e870 tx=0x7f2e68014f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2d50ff9640 1 -- 172.21.15.110:0/1246393875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e68013b80 con 0x55707ea72810 2024-04-27T03:18:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2d50ff9640 1 -- 172.21.15.110:0/1246393875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e68013d20 con 0x55707ea72810 2024-04-27T03:18:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55707ea7a500 con 0x55707ea72810 2024-04-27T03:18:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2d50ff9640 1 -- 172.21.15.110:0/1246393875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e680134e0 con 0x55707ea72810 2024-04-27T03:18:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.077+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707ea7d1e0 con 0x55707ea72810 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2d50ff9640 1 -- 172.21.15.110:0/1246393875 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2e68030070 con 0x55707ea72810 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2d50ff9640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d380346a0 0x7f2d38036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2d50ff9640 1 -- 172.21.15.110:0/1246393875 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2e6804fc70 con 0x55707ea72810 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707ea85400 0x55707ea878a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d380346a0 0x7f2d38036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea87ff0 con 0x55707ea85400 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6e786640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707ea85400 0x55707ea878a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d380346a0 0x7f2d38036b60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000e870 tx=0x7f2e6003c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6e786640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707ea85400 0x55707ea878a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 realm 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea88420 con 0x55707ea85400 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea88cc0 con 0x55707ea85400 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea896e0 0x55707ea8bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea8c2a0 con 0x55707ea896e0 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6f788640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea896e0 0x55707ea8bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6f788640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea896e0 0x55707ea8bb30 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea8caf0 con 0x55707ea85400 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.081+0000 7f2e7146c980 20 searching for the correct realm 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55707ea8e860 0x55707ea90c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea8e860 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55707ea8e860 0x55707ea90c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6ef87640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55707ea8e860 0x55707ea90c50 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6f788640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2e6805a040 0x7f2e680115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x7f2e6805a040 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2e6805a040 0x7f2e680115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2e6805a040 0x7f2e680115b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea8e860 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.085+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x7f2e6805a040 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x7f2e6805a040 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea8e860 2024-04-27T03:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.089+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea8e860 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea8e860 2024-04-27T03:18:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x7f2e6805a040 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea85400 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x7f2e6805a040 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea8e860 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x7f2e6805a040 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea8e860 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.093+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91280 con 0x55707ea896e0 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea8d470 con 0x55707ea85400 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea8dd10 con 0x55707ea896e0 2024-04-27T03:18:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea8d0b0 con 0x55707ea85400 2024-04-27T03:18:08.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:08.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 zone default found 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 4 Realm: () 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 zonegroup default 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea91bb0 con 0x7f2e6805a040 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea93e60 con 0x55707ea8e860 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea94200 con 0x55707ea8e860 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 started sync module instance, tier type = 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.097+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea932a0 con 0x55707ea896e0 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93941649716880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea93ab0 con 0x55707ea896e0 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea95070 con 0x55707ea85400 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93941649722576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea95700 con 0x55707ea85400 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea96220 con 0x55707ea896e0 2024-04-27T03:18:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93941649716880] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93941649722576] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.101+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93941649726704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea961e0 con 0x55707ea896e0 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea96ac0 con 0x7f2e6805a040 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93941649729040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea961e0 con 0x7f2e6805a040 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea97520 con 0x55707ea896e0 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93941649729040] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93941649726704] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93941649720736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea97520 con 0x55707ea896e0 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea97940 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93941649730096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea98190 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.105+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea98b50 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93941649720736] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93941649730096] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93941649737376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea98f40 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea997f0 con 0x7f2e6805a040 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93941649737376] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'57 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93941649731664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea993f0 con 0x7f2e6805a040 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=0 2024-04-27T03:18:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=1 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=2 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=3 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=4 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=5 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=6 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93941649731664] v27'58 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 20 add_watcher() i=7 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707ea9cc70 con 0x55707ea85400 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.109+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93941649740048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707ea9a190 con 0x55707ea85400 2024-04-27T03:18:08.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93941649740048] v27'78 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2d257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 5 note: GC not initialized 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2c9ffff640 20 reqs_thread_entry: start 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 20 Filter name: none 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2c9dffb640 20 reqs_thread_entry: start 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707eac04c0 con 0x55707ea85400 2024-04-27T03:18:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~556 out=556b,getxattrs out=75b] v0'0 uv6 ondisk = 0) ==== 231+0+679 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 20 rados_obj.operate() r=0 bl.length=556 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~556 in=556b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eac6c60 con 0x55707ea85400 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [delete,create,call,call,writefull 0~556] v27'7 uv7 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.113+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 93941649925840 in=744b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55707eac2390 con 0x55707ea896e0 2024-04-27T03:18:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 93941649720736 notify 115964117001 ret 0) ==== 774+0+0 (crc 0 0 0) 0x55707ea7d1e0 con 0x55707ea896e0 2024-04-27T03:18:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2d527fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117001 cookie 93941649720736 notifier 4359 bl.length()=732 2024-04-27T03:18:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2d527fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2d527fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2d527fc640 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f2d34003eb0 con 0x55707ea896e0 2024-04-27T03:18:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 93941649925840 out=8b] v0'0 uv57 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 93941649925840 notify 115964117001 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55707ea7a500 con 0x55707ea896e0 2024-04-27T03:18:08.129 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo1","permissions":"full-control"}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo1","secret_key":"gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL","active":true,"create_date":"0.000000"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2d257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.117+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93941649740048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eac5710 con 0x55707ea85400 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 93941649740048] v27'79 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93941649716880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x55707ea896e0 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 93941649716880] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e7146c980 20 remove_watcher() i=0 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e7146c980 2 removed watcher, disabling cache 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93941649722576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x55707ea85400 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 93941649722576] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x55707ea85400 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e7146c980 20 remove_watcher() i=1 2024-04-27T03:18:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.121+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93941649726704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x55707ea896e0 2024-04-27T03:18:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.125+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 93941649726704] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.125+0000 7f2e7146c980 20 remove_watcher() i=2 2024-04-27T03:18:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.125+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93941649729040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x7f2e6805a040 2024-04-27T03:18:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.125+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 93941649729040] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.125+0000 7f2e7146c980 20 remove_watcher() i=3 2024-04-27T03:18:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.125+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93941649720736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x55707ea896e0 2024-04-27T03:18:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e6f788640 1 -- 172.21.15.110:0/1246393875 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 93941649720736] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e68057040 con 0x55707ea896e0 2024-04-27T03:18:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e7146c980 20 remove_watcher() i=4 2024-04-27T03:18:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93941649730096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x55707ea8e860 2024-04-27T03:18:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 93941649730096] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e7146c980 20 remove_watcher() i=5 2024-04-27T03:18:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93941649737376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x55707ea8e860 2024-04-27T03:18:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e6ef87640 1 -- 172.21.15.110:0/1246393875 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 93941649737376] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e6002e070 con 0x55707ea8e860 2024-04-27T03:18:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e7146c980 20 remove_watcher() i=6 2024-04-27T03:18:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.129+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93941649731664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55707eabe1b0 con 0x7f2e6805a040 2024-04-27T03:18:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 93941649731664] v27'59 uv57 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2e640aaa10 con 0x7f2e6805a040 2024-04-27T03:18:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 20 remove_watcher() i=7 2024-04-27T03:18:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea896e0 msgr2=0x55707ea8bb30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea896e0 0x55707ea8bb30 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707ea85400 msgr2=0x55707ea878a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707ea85400 0x55707ea878a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2e6805a040 msgr2=0x7f2e680115b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2e6805a040 0x7f2e680115b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55707ea8e860 msgr2=0x55707ea90c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55707ea8e860 0x55707ea90c50 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d380346a0 msgr2=0x7f2d38036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d380346a0 0x7f2d38036b60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000e870 tx=0x7f2e6003c000 comp rx=0 tx=0).stop 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 msgr2=0x55707ea79d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/1246393875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea72810 0x55707ea79d30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2e6801e870 tx=0x7f2e68014f70 comp rx=0 tx=0).stop 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 reap_dead start 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e6e786640 1 -- 172.21.15.110:0/1246393875 reap_dead start 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 shutdown_connections 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 >> 172.21.15.110:0/1246393875 conn(0x55707ea6eb50 msgr2=0x55707ea6c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 shutdown_connections 2024-04-27T03:18:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/1246393875 wait complete. 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d480346f0 msgr2=0x7f2d48036bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d480346f0 0x7f2d48036bb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x55707ea55a90 tx=0x7f2e6000ff30 comp rx=0 tx=0).stop 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea65680 msgr2=0x55707ea65aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea65680 0x55707ea65aa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2e680056d0 tx=0x7f2e68017820 comp rx=0 tx=0).stop 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 shutdown_connections 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d480346f0 0x7f2d48036bb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/253885145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707ea65680 0x55707ea65aa0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 >> 172.21.15.110:0/253885145 conn(0x55707ea5a490 msgr2=0x55707ea5afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 shutdown_connections 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/253885145 wait complete. 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea52a60 msgr2=0x55707ea54e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea52a60 0x55707ea54e50 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707e95ca00 msgr2=0x55707ea515d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707e95ca00 0x55707ea515d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d54034650 msgr2=0x7f2d54036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d54034650 0x7f2d54036b10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2e640ae000 tx=0x7f2e640a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 msgr2=0x55707ea4a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea4a720 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2e60000c10 tx=0x7f2e60000ee0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 shutdown_connections 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55707ea52a60 0x55707ea54e50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55707e95ca00 0x55707ea515d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2d54034650 0x7f2d54036b10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 --2- 172.21.15.110:0/3418851769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55707e8ca7f0 0x55707ea4a720 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 >> 172.21.15.110:0/3418851769 conn(0x55707e9509c0 msgr2=0x55707ea3dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 shutdown_connections 2024-04-27T03:18:08.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.133+0000 7f2e7146c980 1 -- 172.21.15.110:0/3418851769 wait complete. 2024-04-27T03:18:08.150 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'active': True, 'create_date': '0.000000'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:08.150 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:08.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:08.151 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6886915980 1 Processor -- start 2024-04-27T03:18:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6886915980 1 -- start start 2024-04-27T03:18:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6886915980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55b818bc8a60 0x55b818bcae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6886915980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55b818a810d0 con 0x55b818b36850 2024-04-27T03:18:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6886915980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55b818a76430 con 0x55b818bc8a60 2024-04-27T03:18:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6883c2f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55b818bc8a60 0x55b818bcae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6884430640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55b818b36850 0x55b818bc7340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34684/0 (socket says 172.21.15.110:34684) 2024-04-27T03:18:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6884430640 1 -- 172.21.15.110:0/1704166520 learned_addr learned my addr 172.21.15.110:0/1704166520 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6883c2f640 1 -- 172.21.15.110:0/1704166520 >> v1:172.21.15.77:6789/0 conn(0x55b818b36850 legacy=0x55b818bc7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.209+0000 7f6883c2f640 1 -- 172.21.15.110:0/1704166520 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b818a6f510 con 0x55b818bc8a60 2024-04-27T03:18:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6883c2f640 1 --2- 172.21.15.110:0/1704166520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818bc8a60 0x55b818bcae50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f687800a480 tx=0x7f687801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b55a03077de49d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f688342e640 1 -- 172.21.15.110:0/1704166520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68780b0070 con 0x55b818bc8a60 2024-04-27T03:18:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f688342e640 1 -- 172.21.15.110:0/1704166520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68780ab540 con 0x55b818bc8a60 2024-04-27T03:18:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f688342e640 1 -- 172.21.15.110:0/1704166520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68780ab840 con 0x55b818bc8a60 2024-04-27T03:18:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6886915980 1 -- 172.21.15.110:0/1704166520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818bc8a60 msgr2=0x55b818bcae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6886915980 1 --2- 172.21.15.110:0/1704166520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818bc8a60 0x55b818bcae50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f687800a480 tx=0x7f687801b280 comp rx=0 tx=0).stop 2024-04-27T03:18:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6886915980 1 -- 172.21.15.110:0/1704166520 shutdown_connections 2024-04-27T03:18:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6886915980 1 --2- 172.21.15.110:0/1704166520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818bc8a60 0x55b818bcae50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6886915980 1 -- 172.21.15.110:0/1704166520 >> 172.21.15.110:0/1704166520 conn(0x55b818bbca20 msgr2=0x55b818bbee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6886915980 1 -- 172.21.15.110:0/1704166520 shutdown_connections 2024-04-27T03:18:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.213+0000 7f6886915980 1 -- 172.21.15.110:0/1704166520 wait complete. 2024-04-27T03:18:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 Processor -- start 2024-04-27T03:18:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- start start 2024-04-27T03:18:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818caf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b818ac52d0 con 0x55b818b36850 2024-04-27T03:18:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818caf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818caf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50946/0 (socket says 172.21.15.110:50946) 2024-04-27T03:18:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 -- 172.21.15.110:0/2425263411 learned_addr learned my addr 172.21.15.110:0/2425263411 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 -- 172.21.15.110:0/2425263411 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b818a4bae0 con 0x55b818b36850 2024-04-27T03:18:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 --2- 172.21.15.110:0/2425263411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818caf5f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6874001d90 tx=0x7f6874001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47d28b738077877 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f677260c640 1 -- 172.21.15.110:0/2425263411 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6874013030 con 0x55b818b36850 2024-04-27T03:18:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f677260c640 1 -- 172.21.15.110:0/2425263411 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687400e690 con 0x55b818b36850 2024-04-27T03:18:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f677260c640 1 -- 172.21.15.110:0/2425263411 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f687400ea10 con 0x55b818b36850 2024-04-27T03:18:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- 172.21.15.110:0/2425263411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 msgr2=0x55b818caf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 --2- 172.21.15.110:0/2425263411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818caf5f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6874001d90 tx=0x7f6874001e80 comp rx=0 tx=0).stop 2024-04-27T03:18:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- 172.21.15.110:0/2425263411 shutdown_connections 2024-04-27T03:18:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 --2- 172.21.15.110:0/2425263411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818caf5f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- 172.21.15.110:0/2425263411 >> 172.21.15.110:0/2425263411 conn(0x55b818bbca20 msgr2=0x55b818cab180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- 172.21.15.110:0/2425263411 shutdown_connections 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- 172.21.15.110:0/2425263411 wait complete. 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 Processor -- start 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- start start 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818cb6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6886915980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b818bcddf0 con 0x55b818b36850 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818cb6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818cb6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50956/0 (socket says 172.21.15.110:50956) 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.229+0000 7f6884430640 1 -- 172.21.15.110:0/116090826 learned_addr learned my addr 172.21.15.110:0/116090826 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6884430640 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b818cb6b70 con 0x55b818b36850 2024-04-27T03:18:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6884430640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818cb6630 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55b818cb0480 tx=0x7f6874000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6770e09640 1 -- 172.21.15.110:0/116090826 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6874027070 con 0x55b818b36850 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b818cb6e00 con 0x55b818b36850 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6770e09640 1 -- 172.21.15.110:0/116090826 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6874019900 con 0x55b818b36850 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b818cb9a80 con 0x55b818b36850 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6770e09640 1 -- 172.21.15.110:0/116090826 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6874019be0 con 0x55b818b36850 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6770e09640 1 -- 172.21.15.110:0/116090826 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f687402a070 con 0x55b818b36850 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6770e09640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f676c034650 0x7f676c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6770e09640 1 -- 172.21.15.110:0/116090826 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f687401f070 con 0x55b818b36850 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6883c2f640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f676c034650 0x7f676c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818bc8a60 0x55b818cbd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cbdb80 con 0x55b818bc8a60 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6884c31640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818bc8a60 0x55b818cbd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6883c2f640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f676c034650 0x7f676c036b10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f68780ad000 tx=0x7f68780a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6884c31640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818bc8a60 0x55b818cbd560 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6884c31640 1 -- 172.21.15.110:0/116090826 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f687c00a4d0 con 0x55b818bc8a60 2024-04-27T03:18:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cbe950 0x55b818cc0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cc1370 con 0x55b818cbe950 2024-04-27T03:18:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.233+0000 7f6884430640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cbe950 0x55b818cc0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884430640 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cbe950 0x55b818cc0d40 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884430640 1 -- 172.21.15.110:0/116090826 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f6874029030 con 0x55b818cbe950 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cc2c20 con 0x55b818bc8a60 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884c31640 1 -- 172.21.15.110:0/116090826 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f687c00a4d0 con 0x55b818bc8a60 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cc1e50 con 0x55b818cbe950 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884430640 1 -- 172.21.15.110:0/116090826 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f6874029030 con 0x55b818cbe950 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cc2240 con 0x55b818bc8a60 2024-04-27T03:18:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884c31640 1 -- 172.21.15.110:0/116090826 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f687c00a4d0 con 0x55b818bc8a60 2024-04-27T03:18:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 Processor -- start 2024-04-27T03:18:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 -- start start 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cd1580 0x55b818cd19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b818ac9310 con 0x55b818cd1580 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884c31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cd1580 0x55b818cd19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884c31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cd1580 0x55b818cd19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50958/0 (socket says 172.21.15.110:50958) 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884c31640 1 -- 172.21.15.110:0/2637136163 learned_addr learned my addr 172.21.15.110:0/2637136163 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884c31640 1 -- 172.21.15.110:0/2637136163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b818cd1ee0 con 0x55b818cd1580 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6884c31640 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cd1580 0x55b818cd19a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f687c00dfc0 tx=0x7f687c0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6769ffb640 1 -- 172.21.15.110:0/2637136163 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f687c0157d0 con 0x55b818cd1580 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6769ffb640 1 -- 172.21.15.110:0/2637136163 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687c015970 con 0x55b818cd1580 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.237+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b818cd2980 con 0x55b818cd1580 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6769ffb640 1 -- 172.21.15.110:0/2637136163 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f687c015c70 con 0x55b818cd1580 2024-04-27T03:18:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b818cd2e50 con 0x55b818cd1580 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6769ffb640 1 -- 172.21.15.110:0/2637136163 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f687c029440 con 0x55b818cd1580 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6769ffb640 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f675c034650 0x7f675c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6769ffb640 1 -- 172.21.15.110:0/2637136163 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f687c00e040 con 0x55b818cd1580 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6884430640 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f675c034650 0x7f675c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6886915980 1 Processor -- start 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6886915980 1 -- start start 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6884430640 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f675c034650 0x7f675c036b10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55b818cc1970 tx=0x7f687402ae50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6886915980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818cdeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6886915980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b818cafd40 con 0x55b818cde710 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6884c31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818cdeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6884c31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818cdeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50960/0 (socket says 172.21.15.110:50960) 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6884c31640 1 -- 172.21.15.110:0/2243335644 learned_addr learned my addr 172.21.15.110:0/2243335644 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6884c31640 1 -- 172.21.15.110:0/2243335644 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b818cdf100 con 0x55b818cde710 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f6884c31640 1 --2- 172.21.15.110:0/2243335644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818cdeb30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f687c015f40 tx=0x7f687c04aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4d40652249f12 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f67637fe640 1 -- 172.21.15.110:0/2243335644 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f687c0283a0 con 0x55b818cde710 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f67637fe640 1 -- 172.21.15.110:0/2243335644 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687c028540 con 0x55b818cde710 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.241+0000 7f67637fe640 1 -- 172.21.15.110:0/2243335644 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f687c04c110 con 0x55b818cde710 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/2243335644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 msgr2=0x55b818cdeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 --2- 172.21.15.110:0/2243335644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818cdeb30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f687c015f40 tx=0x7f687c04aa00 comp rx=0 tx=0).stop 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/2243335644 shutdown_connections 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 --2- 172.21.15.110:0/2243335644 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818cdeb30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/2243335644 >> 172.21.15.110:0/2243335644 conn(0x55b818cdaaa0 msgr2=0x55b818cd8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/2243335644 shutdown_connections 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/2243335644 wait complete. 2024-04-27T03:18:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 Processor -- start 2024-04-27T03:18:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- start start 2024-04-27T03:18:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818ce5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b818cdf850 con 0x55b818cde710 2024-04-27T03:18:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6884c31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818ce5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6884c31640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818ce5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50976/0 (socket says 172.21.15.110:50976) 2024-04-27T03:18:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 learned_addr learned my addr 172.21.15.110:0/1494152387 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b818ce60e0 con 0x55b818cde710 2024-04-27T03:18:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6884c31640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818ce5ba0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f687c015f40 tx=0x7f687c04a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6761ffb640 1 -- 172.21.15.110:0/1494152387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f687c036070 con 0x55b818cde710 2024-04-27T03:18:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b818ce6370 con 0x55b818cde710 2024-04-27T03:18:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b818ce9050 con 0x55b818cde710 2024-04-27T03:18:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6761ffb640 1 -- 172.21.15.110:0/1494152387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687c049460 con 0x55b818cde710 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6761ffb640 1 -- 172.21.15.110:0/1494152387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f687c041040 con 0x55b818cde710 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6761ffb640 1 -- 172.21.15.110:0/1494152387 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f687c039070 con 0x55b818cde710 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6761ffb640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6754034650 0x7f6754036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6761ffb640 1 -- 172.21.15.110:0/1494152387 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f687c0453c0 con 0x55b818cde710 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6884430640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6754034650 0x7f6754036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818cf11e0 0x55b818cf3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf3dd0 con 0x55b818cf11e0 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.245+0000 7f6883c2f640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818cf11e0 0x55b818cf3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884430640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6754034650 0x7f6754036b10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6874042160 tx=0x7f6874037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6883c2f640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818cf11e0 0x55b818cf3680 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 realm 2024-04-27T03:18:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf4200 con 0x55b818cf11e0 2024-04-27T03:18:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf4aa0 con 0x55b818cf11e0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cf54c0 0x55b818cf78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf8040 con 0x55b818cf54c0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884c31640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cf54c0 0x55b818cf78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884c31640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cf54c0 0x55b818cf78d0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf88e0 con 0x55b818cf11e0 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 20 searching for the correct realm 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b818cfa650 0x55b818cfca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cfa650 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884430640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b818cfa650 0x55b818cfca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884430640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b818cfa650 0x55b818cfca40 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.249+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf54c0 2024-04-27T03:18:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884c31640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f687c014030 0x7f687c013a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x7f687c014030 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f687c014030 0x7f687c013a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f687c014030 0x7f687c013a50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf11e0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf11e0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cfa650 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf11e0 2024-04-27T03:18:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf54c0 2024-04-27T03:18:08.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf54c0 2024-04-27T03:18:08.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf11e0 2024-04-27T03:18:08.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.253+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x7f687c014030 2024-04-27T03:18:08.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf11e0 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x7f687c014030 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfcfd0 con 0x55b818cf54c0 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cfa650 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf11e0 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf54c0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cfa650 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf11e0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf11e0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf11e0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cfa650 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x7f687c014030 2024-04-27T03:18:08.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf11e0 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.257+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x7f687c014030 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cfa650 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x7f687c014030 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf54c0 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf54c0 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cfa650 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf54c0 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f687c064040 con 0x55b818cf54c0 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf9290 con 0x55b818cf11e0 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.261+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf9710 con 0x55b818cf54c0 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cf9b70 con 0x55b818cf11e0 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 zone default found 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 4 Realm: () 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:08.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 zonegroup default 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfd900 con 0x7f687c014030 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cff340 con 0x55b818cfa650 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818cff830 con 0x55b818cfa650 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 started sync module instance, tier type = 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfe770 con 0x55b818cf54c0 2024-04-27T03:18:08.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94249178623088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfe770 con 0x55b818cf54c0 2024-04-27T03:18:08.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.265+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d00360 con 0x55b818cf11e0 2024-04-27T03:18:08.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94249178627632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d00320 con 0x55b818cf11e0 2024-04-27T03:18:08.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d01260 con 0x55b818cf54c0 2024-04-27T03:18:08.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94249178623088] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94249178627632] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94249178629968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d01260 con 0x55b818cf54c0 2024-04-27T03:18:08.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d01b00 con 0x7f687c014030 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.269+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94249178633808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d01260 con 0x7f687c014030 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d025d0 con 0x55b818cf54c0 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94249178629968] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94249178633808] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94249178634864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d02590 con 0x55b818cf54c0 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d029c0 con 0x55b818cfa650 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94249178631536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d029c0 con 0x55b818cfa650 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d03880 con 0x55b818cfa650 2024-04-27T03:18:08.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94249178634864] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94249178631536] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94249178639792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d033d0 con 0x55b818cfa650 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.273+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d04120 con 0x7f687c014030 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94249178639792] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'60 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94249178636480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d033d0 con 0x7f687c014030 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=0 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=1 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=2 2024-04-27T03:18:08.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=3 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=4 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=5 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=6 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94249178636480] v27'61 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 20 add_watcher() i=7 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818d07210 con 0x55b818cf11e0 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.277+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94249178643568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d04730 con 0x55b818cf11e0 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94249178643568] v27'80 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f673a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:08.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 5 note: GC not initialized 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f67397e2640 20 reqs_thread_entry: start 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 20 Filter name: none 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f66aaffd640 20 reqs_thread_entry: start 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b818d2a570 con 0x55b818cf11e0 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~556 out=556b,getxattrs out=75b] v0'0 uv7 ondisk = 0) ==== 231+0+679 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 20 rados_obj.operate() r=0 bl.length=556 2024-04-27T03:18:08.293 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo1","permissions":"full-control"}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo1","secret_key":"gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL","active":true,"create_date":"0.000000"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:08.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f673a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.281+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94249178643568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d28fc0 con 0x55b818cf11e0 2024-04-27T03:18:08.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94249178643568] v27'81 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94249178623088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d03c70 con 0x55b818cf54c0 2024-04-27T03:18:08.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94249178623088] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6886915980 20 remove_watcher() i=0 2024-04-27T03:18:08.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6886915980 2 removed watcher, disabling cache 2024-04-27T03:18:08.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94249178627632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfec70 con 0x55b818cf11e0 2024-04-27T03:18:08.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94249178627632] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x55b818cf11e0 2024-04-27T03:18:08.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6886915980 20 remove_watcher() i=1 2024-04-27T03:18:08.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.285+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94249178629968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfec70 con 0x55b818cf54c0 2024-04-27T03:18:08.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.289+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94249178629968] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.289+0000 7f6886915980 20 remove_watcher() i=2 2024-04-27T03:18:08.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.289+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94249178633808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfec70 con 0x7f687c014030 2024-04-27T03:18:08.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.289+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94249178633808] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.289+0000 7f6886915980 20 remove_watcher() i=3 2024-04-27T03:18:08.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.289+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94249178634864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d01650 con 0x55b818cf54c0 2024-04-27T03:18:08.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6884c31640 1 -- 172.21.15.110:0/1494152387 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94249178634864] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f687c038030 con 0x55b818cf54c0 2024-04-27T03:18:08.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6886915980 20 remove_watcher() i=4 2024-04-27T03:18:08.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94249178631536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfec70 con 0x55b818cfa650 2024-04-27T03:18:08.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94249178631536] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6886915980 20 remove_watcher() i=5 2024-04-27T03:18:08.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94249178639792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818d00d70 con 0x55b818cfa650 2024-04-27T03:18:08.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6884430640 1 -- 172.21.15.110:0/1494152387 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94249178639792] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6874029030 con 0x55b818cfa650 2024-04-27T03:18:08.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6886915980 20 remove_watcher() i=6 2024-04-27T03:18:08.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.293+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94249178636480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b818cfec70 con 0x7f687c014030 2024-04-27T03:18:08.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94249178636480] v27'62 uv60 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68780b3040 con 0x7f687c014030 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 20 remove_watcher() i=7 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cf54c0 msgr2=0x55b818cf78d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cf54c0 0x55b818cf78d0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818cf11e0 msgr2=0x55b818cf3680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818cf11e0 0x55b818cf3680 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f687c014030 msgr2=0x7f687c013a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f687c014030 0x7f687c013a50 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b818cfa650 msgr2=0x55b818cfca40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b818cfa650 0x55b818cfca40 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6754034650 msgr2=0x7f6754036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6754034650 0x7f6754036b10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6874042160 tx=0x7f6874037000 comp rx=0 tx=0).stop 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 msgr2=0x55b818ce5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818ce5ba0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f687c015f40 tx=0x7f687c04a850 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 shutdown_connections 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b818cfa650 0x55b818cfca40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cf54c0 0x55b818cf78d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6883c2f640 1 -- 172.21.15.110:0/1494152387 reap_dead start 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818cf11e0 0x55b818cf3680 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6754034650 0x7f6754036b10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f687c014030 0x7f687c013a50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/1494152387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cde710 0x55b818ce5ba0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 >> 172.21.15.110:0/1494152387 conn(0x55b818cdaaa0 msgr2=0x55b818cdff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 shutdown_connections 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/1494152387 wait complete. 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f675c034650 msgr2=0x7f675c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f675c034650 0x7f675c036b10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55b818cc1970 tx=0x7f687402ae50 comp rx=0 tx=0).stop 2024-04-27T03:18:08.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cd1580 msgr2=0x55b818cd19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cd1580 0x55b818cd19a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f687c00dfc0 tx=0x7f687c0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 shutdown_connections 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f675c034650 0x7f675c036b10 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/2637136163 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818cd1580 0x55b818cd19a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 >> 172.21.15.110:0/2637136163 conn(0x55b818cc6390 msgr2=0x55b818cc6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 shutdown_connections 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/2637136163 wait complete. 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cbe950 msgr2=0x55b818cc0d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cbe950 0x55b818cc0d40 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818bc8a60 msgr2=0x55b818cbd560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818bc8a60 0x55b818cbd560 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f676c034650 msgr2=0x7f676c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f676c034650 0x7f676c036b10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f68780ad000 tx=0x7f68780a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:08.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 msgr2=0x55b818cb6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818cb6630 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55b818cb0480 tx=0x7f6874000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 shutdown_connections 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b818cbe950 0x55b818cc0d40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b818bc8a60 0x55b818cbd560 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f676c034650 0x7f676c036b10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 --2- 172.21.15.110:0/116090826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b818b36850 0x55b818cb6630 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 >> 172.21.15.110:0/116090826 conn(0x55b818bbca20 msgr2=0x55b818ca9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 shutdown_connections 2024-04-27T03:18:08.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.297+0000 7f6886915980 1 -- 172.21.15.110:0/116090826 wait complete. 2024-04-27T03:18:08.314 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'active': True, 'create_date': '0.000000'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:08.314 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:08.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:08.315 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:08.382 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:08.382 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.373+0000 7f8765169980 1 Processor -- start 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.373+0000 7f8765169980 1 -- start start 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.373+0000 7f8765169980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5599569bd4b0 0x5599569bd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.373+0000 7f8765169980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55995687a0d0 con 0x5599569bddd0 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.373+0000 7f8765169980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55995686f430 con 0x5599569bd4b0 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8762c84640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5599569bd4b0 0x5599569bd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8762483640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5599569bddd0 0x5599569be790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34686/0 (socket says 172.21.15.110:34686) 2024-04-27T03:18:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8762483640 1 -- 172.21.15.110:0/1928172935 learned_addr learned my addr 172.21.15.110:0/1928172935 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:08.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8762c84640 1 -- 172.21.15.110:0/1928172935 >> v1:172.21.15.77:6789/0 conn(0x5599569bddd0 legacy=0x5599569be790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8762c84640 1 -- 172.21.15.110:0/1928172935 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x559956868510 con 0x5599569bd4b0 2024-04-27T03:18:08.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8762c84640 1 --2- 172.21.15.110:0/1928172935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x5599569bd890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f875c00a420 tx=0x7f875c01afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731961b3c2571857 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8761c82640 1 -- 172.21.15.110:0/1928172935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f875c0af070 con 0x5599569bd4b0 2024-04-27T03:18:08.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8761c82640 1 -- 172.21.15.110:0/1928172935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c0aa530 con 0x5599569bd4b0 2024-04-27T03:18:08.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8761c82640 1 -- 172.21.15.110:0/1928172935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f875c0aa850 con 0x5599569bd4b0 2024-04-27T03:18:08.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8765169980 1 -- 172.21.15.110:0/1928172935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 msgr2=0x5599569bd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.377+0000 7f8765169980 1 --2- 172.21.15.110:0/1928172935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x5599569bd890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f875c00a420 tx=0x7f875c01afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.381+0000 7f8765169980 1 -- 172.21.15.110:0/1928172935 shutdown_connections 2024-04-27T03:18:08.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.381+0000 7f8765169980 1 --2- 172.21.15.110:0/1928172935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x5599569bd890 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.381+0000 7f8765169980 1 -- 172.21.15.110:0/1928172935 >> 172.21.15.110:0/1928172935 conn(0x559956930b90 msgr2=0x559956930fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.381+0000 7f8765169980 1 -- 172.21.15.110:0/1928172935 shutdown_connections 2024-04-27T03:18:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.381+0000 7f8765169980 1 -- 172.21.15.110:0/1928172935 wait complete. 2024-04-27T03:18:08.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 Processor -- start 2024-04-27T03:18:08.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- start start 2024-04-27T03:18:08.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aa8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599568be2d0 con 0x5599569bd4b0 2024-04-27T03:18:08.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8762c84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aa8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8762c84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aa8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50990/0 (socket says 172.21.15.110:50990) 2024-04-27T03:18:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8762c84640 1 -- 172.21.15.110:0/997158334 learned_addr learned my addr 172.21.15.110:0/997158334 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8762c84640 1 -- 172.21.15.110:0/997158334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559956844ae0 con 0x5599569bd4b0 2024-04-27T03:18:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8762c84640 1 --2- 172.21.15.110:0/997158334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aa8d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f875c0ad040 tx=0x7f875c0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228476dec31ede85 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8650e09640 1 -- 172.21.15.110:0/997158334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f875c0af050 con 0x5599569bd4b0 2024-04-27T03:18:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8650e09640 1 -- 172.21.15.110:0/997158334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c0b0040 con 0x5599569bd4b0 2024-04-27T03:18:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8650e09640 1 -- 172.21.15.110:0/997158334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f875c0aaca0 con 0x5599569bd4b0 2024-04-27T03:18:08.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- 172.21.15.110:0/997158334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 msgr2=0x559956aa8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 --2- 172.21.15.110:0/997158334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aa8d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f875c0ad040 tx=0x7f875c0033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- 172.21.15.110:0/997158334 shutdown_connections 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 --2- 172.21.15.110:0/997158334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aa8d70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- 172.21.15.110:0/997158334 >> 172.21.15.110:0/997158334 conn(0x559956930b90 msgr2=0x559956aa4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- 172.21.15.110:0/997158334 shutdown_connections 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- 172.21.15.110:0/997158334 wait complete. 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 Processor -- start 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- start start 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aafcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.393+0000 7f8765169980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599569c7500 con 0x5599569bd4b0 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8762c84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aafcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8762c84640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aafcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50992/0 (socket says 172.21.15.110:50992) 2024-04-27T03:18:08.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8762c84640 1 -- 172.21.15.110:0/1423906318 learned_addr learned my addr 172.21.15.110:0/1423906318 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8762c84640 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559956ab0230 con 0x5599569bd4b0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8762c84640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aafcf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f875c018c00 tx=0x7f875c019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f864f606640 1 -- 172.21.15.110:0/1423906318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f875c0c5070 con 0x5599569bd4b0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f864f606640 1 -- 172.21.15.110:0/1423906318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c0b6d60 con 0x5599569bd4b0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559956ab04c0 con 0x5599569bd4b0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f864f606640 1 -- 172.21.15.110:0/1423906318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f875c0c15f0 con 0x5599569bd4b0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559956ab30a0 con 0x5599569bd4b0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f864f606640 1 -- 172.21.15.110:0/1423906318 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f875c0c8070 con 0x5599569bd4b0 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f864f606640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f86480346f0 0x7f8648036bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f864f606640 1 -- 172.21.15.110:0/1423906318 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f875c0bb070 con 0x5599569bd4b0 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8762483640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f86480346f0 0x7f8648036bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5599569bddd0 0x559956ab6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956ab7110 con 0x5599569bddd0 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8763485640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5599569bddd0 0x559956ab6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8762483640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f86480346f0 0x7f8648036bb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8758000f70 tx=0x7f87580012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8763485640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5599569bddd0 0x559956ab6b80 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.397+0000 7f8763485640 1 -- 172.21.15.110:0/1423906318 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f87540083f0 con 0x5599569bddd0 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956ab7ee0 0x559956aba2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956aba910 con 0x559956ab7ee0 2024-04-27T03:18:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8762c84640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956ab7ee0 0x559956aba2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8762c84640 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956ab7ee0 0x559956aba2d0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8762c84640 1 -- 172.21.15.110:0/1423906318 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f875c0d1070 con 0x559956ab7ee0 2024-04-27T03:18:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956abc1c0 con 0x5599569bddd0 2024-04-27T03:18:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8763485640 1 -- 172.21.15.110:0/1423906318 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f87540083f0 con 0x5599569bddd0 2024-04-27T03:18:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956abb3f0 con 0x559956ab7ee0 2024-04-27T03:18:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8762c84640 1 -- 172.21.15.110:0/1423906318 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f875c0d1070 con 0x559956ab7ee0 2024-04-27T03:18:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956abb7e0 con 0x5599569bddd0 2024-04-27T03:18:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8763485640 1 -- 172.21.15.110:0/1423906318 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f87540083f0 con 0x5599569bddd0 2024-04-27T03:18:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 Processor -- start 2024-04-27T03:18:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 -- start start 2024-04-27T03:18:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956acaac0 0x559956acaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.401+0000 7f8765169980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5599568c2310 con 0x559956acaac0 2024-04-27T03:18:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956acaac0 0x559956acaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956acaac0 0x559956acaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51002/0 (socket says 172.21.15.110:51002) 2024-04-27T03:18:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 -- 172.21.15.110:0/832118872 learned_addr learned my addr 172.21.15.110:0/832118872 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 -- 172.21.15.110:0/832118872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559956acb420 con 0x559956acaac0 2024-04-27T03:18:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956acaac0 0x559956acaee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f875400aab0 tx=0x7f875400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f864ce01640 1 -- 172.21.15.110:0/832118872 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754007d30 con 0x559956acaac0 2024-04-27T03:18:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f864ce01640 1 -- 172.21.15.110:0/832118872 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875401d070 con 0x559956acaac0 2024-04-27T03:18:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559956acbec0 con 0x559956acaac0 2024-04-27T03:18:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f864ce01640 1 -- 172.21.15.110:0/832118872 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754018440 con 0x559956acaac0 2024-04-27T03:18:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559956acc390 con 0x559956acaac0 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f864ce01640 1 -- 172.21.15.110:0/832118872 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8754013070 con 0x559956acaac0 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f864ce01640 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f863c034650 0x7f863c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f864ce01640 1 -- 172.21.15.110:0/832118872 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f875404b520 con 0x559956acaac0 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8762c84640 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f863c034650 0x7f863c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8765169980 1 Processor -- start 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8765169980 1 -- start start 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8762c84640 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f863c034650 0x7f863c036b10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x559956abaf10 tx=0x7f875c0e3610 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8765169980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956ad80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8765169980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559956aa9400 con 0x559956ad7cc0 2024-04-27T03:18:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956ad80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956ad80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51016/0 (socket says 172.21.15.110:51016) 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.405+0000 7f8763485640 1 -- 172.21.15.110:0/3921464904 learned_addr learned my addr 172.21.15.110:0/3921464904 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8763485640 1 -- 172.21.15.110:0/3921464904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559956ad86b0 con 0x559956ad7cc0 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8763485640 1 --2- 172.21.15.110:0/3921464904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956ad80e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8754049930 tx=0x7f8754018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78d2b9e8c2424e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f86467fc640 1 -- 172.21.15.110:0/3921464904 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754047070 con 0x559956ad7cc0 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f86467fc640 1 -- 172.21.15.110:0/3921464904 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8754018bc0 con 0x559956ad7cc0 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f86467fc640 1 -- 172.21.15.110:0/3921464904 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754020360 con 0x559956ad7cc0 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- 172.21.15.110:0/3921464904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 msgr2=0x559956ad80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 --2- 172.21.15.110:0/3921464904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956ad80e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8754049930 tx=0x7f8754018950 comp rx=0 tx=0).stop 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- 172.21.15.110:0/3921464904 shutdown_connections 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 --2- 172.21.15.110:0/3921464904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956ad80e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- 172.21.15.110:0/3921464904 >> 172.21.15.110:0/3921464904 conn(0x559956ad4000 msgr2=0x559956ad2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- 172.21.15.110:0/3921464904 shutdown_connections 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- 172.21.15.110:0/3921464904 wait complete. 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 Processor -- start 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- start start 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956adf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559956ad8e10 con 0x559956ad7cc0 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8763485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956adf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8763485640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956adf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51032/0 (socket says 172.21.15.110:51032) 2024-04-27T03:18:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 learned_addr learned my addr 172.21.15.110:0/4191408131 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559956adf6c0 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8763485640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956adf180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f87540168b0 tx=0x7f87540498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8644ff9640 1 -- 172.21.15.110:0/4191408131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754031070 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559956adf950 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8644ff9640 1 -- 172.21.15.110:0/4191408131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8754020900 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8644ff9640 1 -- 172.21.15.110:0/4191408131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8754049a70 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.409+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559956ae2630 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8644ff9640 1 -- 172.21.15.110:0/4191408131 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8754034070 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8644ff9640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f862c034650 0x7f862c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8644ff9640 1 -- 172.21.15.110:0/4191408131 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f8754037c90 con 0x559956ad7cc0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762c84640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f862c034650 0x7f862c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559956aea800 0x559956aecca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956aed3f0 con 0x559956aea800 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762483640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559956aea800 0x559956aecca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762c84640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f862c034650 0x7f862c036b10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f875c0c88c0 tx=0x7f875c0e0000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762483640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559956aea800 0x559956aecca0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f875800e5a0 con 0x559956aea800 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 realm 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956ae9cd0 con 0x559956aea800 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f875800e5a0 con 0x559956aea800 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956aedef0 con 0x559956aea800 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f875800e5a0 con 0x559956aea800 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956aeecd0 0x559956af10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af1740 con 0x559956aeecd0 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8763485640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956aeecd0 0x559956af10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8763485640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956aeecd0 0x559956af10c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af1fe0 con 0x559956aea800 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f875800e5a0 con 0x559956aea800 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 20 searching for the correct realm 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559956af3d50 0x559956af6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956af3d50 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.413+0000 7f8762c84640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559956af3d50 0x559956af6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762c84640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559956af3d50 0x559956af6140 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8763485640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8754036070 0x7f875401a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x7f8754036070 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8754036070 0x7f875401a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8754036070 0x7f875401a9b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875800e5a0 con 0x7f8754036070 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875800e5a0 con 0x559956aea800 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956af3d50 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.417+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x7f8754036070 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x7f8754036070 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x7f8754036070 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f8758029040 con 0x7f8754036070 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956af3d50 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956af3d50 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.421+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956af3d50 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x7f8754036070 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x7f8754036070 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aea800 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x7f8754036070 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x7f8754036070 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956af3d50 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x7f8754036070 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8758029040 con 0x7f8754036070 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956af3d50 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.425+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559956af6720 con 0x559956aeecd0 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af2990 con 0x559956aea800 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8758029040 con 0x559956aea800 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af2e10 con 0x559956aeecd0 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af3270 con 0x559956aea800 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 zone default found 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 4 Realm: () 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 zonegroup default 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af6ba0 con 0x7f8754036070 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x7f8754036070 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af8a30 con 0x559956af3d50 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956af8dd0 con 0x559956af3d50 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 started sync module instance, tier type = 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.429+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af7e70 con 0x559956aeecd0 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94117072697792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af9170 con 0x559956aeecd0 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af9b30 con 0x559956aea800 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94117072705152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af9f50 con 0x559956aea800 2024-04-27T03:18:08.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afa9a0 con 0x559956aeecd0 2024-04-27T03:18:08.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94117072697792] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94117072705152] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94117072708752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af9aa0 con 0x559956aeecd0 2024-04-27T03:18:08.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.433+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afb240 con 0x7f8754036070 2024-04-27T03:18:08.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x7f8754036070 2024-04-27T03:18:08.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94117072711056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af9aa0 con 0x7f8754036070 2024-04-27T03:18:08.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afbd00 con 0x559956aeecd0 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94117072711056] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x7f8754036070 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94117072708752] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94117072700048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afb1b0 con 0x559956aeecd0 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.437+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afc170 con 0x559956af3d50 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94117072714944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afc9c0 con 0x559956af3d50 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afd380 con 0x559956af3d50 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94117072700048] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94117072714944] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94117072719568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afc9c0 con 0x559956af3d50 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afd350 con 0x7f8754036070 2024-04-27T03:18:08.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94117072719568] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'63 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x7f8754036070 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94117072721776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afc9c0 con 0x7f8754036070 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=0 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=1 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=2 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=3 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=4 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=5 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=6 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94117072721776] v27'64 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x7f8754036070 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 20 add_watcher() i=7 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.441+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956b00ca0 con 0x559956aea800 2024-04-27T03:18:08.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94117072717536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afe1a0 con 0x559956aea800 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94117072717536] v27'82 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f86197e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 5 note: GC not initialized 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8593fff640 20 reqs_thread_entry: start 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 20 Filter name: none 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8591ffb640 20 reqs_thread_entry: start 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559956b24010 con 0x559956aea800 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~556 out=556b,getxattrs out=75b] v0'0 uv7 ondisk = 0) ==== 231+0+679 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 20 rados_obj.operate() r=0 bl.length=556 2024-04-27T03:18:08.458 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo1","permissions":"full-control"}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo1","secret_key":"gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL","active":true,"create_date":"0.000000"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:08.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f86197e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.445+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94117072717536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956b22a60 con 0x559956aea800 2024-04-27T03:18:08.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.449+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94117072717536] v27'83 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.449+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94117072697792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afc6b0 con 0x559956aeecd0 2024-04-27T03:18:08.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.453+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94117072697792] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.453+0000 7f8765169980 20 remove_watcher() i=0 2024-04-27T03:18:08.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.453+0000 7f8765169980 2 removed watcher, disabling cache 2024-04-27T03:18:08.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.453+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94117072705152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af79c0 con 0x559956aea800 2024-04-27T03:18:08.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.453+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94117072705152] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x559956aea800 2024-04-27T03:18:08.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.453+0000 7f8765169980 20 remove_watcher() i=1 2024-04-27T03:18:08.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.453+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94117072708752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af79c0 con 0x559956aeecd0 2024-04-27T03:18:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94117072708752] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8765169980 20 remove_watcher() i=2 2024-04-27T03:18:08.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94117072711056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afa490 con 0x7f8754036070 2024-04-27T03:18:08.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94117072711056] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x7f8754036070 2024-04-27T03:18:08.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8765169980 20 remove_watcher() i=3 2024-04-27T03:18:08.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94117072700048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af79c0 con 0x559956aeecd0 2024-04-27T03:18:08.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8763485640 1 -- 172.21.15.110:0/4191408131 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94117072700048] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8754033030 con 0x559956aeecd0 2024-04-27T03:18:08.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8765169980 20 remove_watcher() i=4 2024-04-27T03:18:08.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.457+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94117072714944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956afad90 con 0x559956af3d50 2024-04-27T03:18:08.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94117072714944] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 20 remove_watcher() i=5 2024-04-27T03:18:08.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94117072719568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af8290 con 0x559956af3d50 2024-04-27T03:18:08.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8762c84640 1 -- 172.21.15.110:0/4191408131 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94117072719568] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875c0d1070 con 0x559956af3d50 2024-04-27T03:18:08.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 20 remove_watcher() i=6 2024-04-27T03:18:08.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94117072721776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559956af79c0 con 0x7f8754036070 2024-04-27T03:18:08.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94117072721776] v27'65 uv63 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f875803c1c0 con 0x7f8754036070 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 20 remove_watcher() i=7 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956aeecd0 msgr2=0x559956af10c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956aeecd0 0x559956af10c0 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559956aea800 msgr2=0x559956aecca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559956aea800 0x559956aecca0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8754036070 msgr2=0x7f875401a9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8754036070 0x7f875401a9b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559956af3d50 msgr2=0x559956af6140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559956af3d50 0x559956af6140 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f862c034650 msgr2=0x7f862c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f862c034650 0x7f862c036b10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f875c0c88c0 tx=0x7f875c0e0000 comp rx=0 tx=0).stop 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 msgr2=0x559956adf180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8762483640 1 -- 172.21.15.110:0/4191408131 reap_dead start 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956adf180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f87540168b0 tx=0x7f87540498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 shutdown_connections 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/4191408131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956ad7cc0 0x559956adf180 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 >> 172.21.15.110:0/4191408131 conn(0x559956ad4000 msgr2=0x559956ad1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 shutdown_connections 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/4191408131 wait complete. 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f863c034650 msgr2=0x7f863c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f863c034650 0x7f863c036b10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x559956abaf10 tx=0x7f875c0e3610 comp rx=0 tx=0).stop 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956acaac0 msgr2=0x559956acaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956acaac0 0x559956acaee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f875400aab0 tx=0x7f875400af70 comp rx=0 tx=0).stop 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 shutdown_connections 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f863c034650 0x7f863c036b10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/832118872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559956acaac0 0x559956acaee0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 >> 172.21.15.110:0/832118872 conn(0x559956abf8d0 msgr2=0x559956ac03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 shutdown_connections 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/832118872 wait complete. 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956ab7ee0 msgr2=0x559956aba2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956ab7ee0 0x559956aba2d0 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5599569bddd0 msgr2=0x559956ab6b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5599569bddd0 0x559956ab6b80 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f86480346f0 msgr2=0x7f8648036bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f86480346f0 0x7f8648036bb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8758000f70 tx=0x7f87580012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 msgr2=0x559956aafcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aafcf0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f875c018c00 tx=0x7f875c019810 comp rx=0 tx=0).stop 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 shutdown_connections 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559956ab7ee0 0x559956aba2d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5599569bddd0 0x559956ab6b80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f86480346f0 0x7f8648036bb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 --2- 172.21.15.110:0/1423906318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5599569bd4b0 0x559956aafcf0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 >> 172.21.15.110:0/1423906318 conn(0x559956930b90 msgr2=0x559956aa3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 shutdown_connections 2024-04-27T03:18:08.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.461+0000 7f8765169980 1 -- 172.21.15.110:0/1423906318 wait complete. 2024-04-27T03:18:08.480 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'active': True, 'create_date': '0.000000'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:08.480 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'create', '--subuser', 'foo:foo2', '--secret', 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', '--key-type', 'swift'] 2024-04-27T03:18:08.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'create', '--subuser', 'foo:foo2', '--secret', 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', '--key-type', 'swift'] 2024-04-27T03:18:08.480 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser create --subuser foo:foo2 --secret ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy --key-type swift 2024-04-27T03:18:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:08.556 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:08.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f6127fd5980 1 Processor -- start 2024-04-27T03:18:08.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f6127fd5980 1 -- start start 2024-04-27T03:18:08.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f6127fd5980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ab42819b60 0x55ab4281bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f6127fd5980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55ab426d20d0 con 0x55ab428138c0 2024-04-27T03:18:08.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f6127fd5980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55ab426c7430 con 0x55ab42819b60 2024-04-27T03:18:08.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f61252ef640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ab42819b60 0x55ab4281bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f6125af0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55ab428138c0 0x55ab42813ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34694/0 (socket says 172.21.15.110:34694) 2024-04-27T03:18:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f6125af0640 1 -- 172.21.15.110:0/288869173 learned_addr learned my addr 172.21.15.110:0/288869173 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f61252ef640 1 -- 172.21.15.110:0/288869173 >> v1:172.21.15.77:6789/0 conn(0x55ab428138c0 legacy=0x55ab42813ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.549+0000 7f61252ef640 1 -- 172.21.15.110:0/288869173 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab426c0510 con 0x55ab42819b60 2024-04-27T03:18:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f61252ef640 1 --2- 172.21.15.110:0/288869173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab42819b60 0x55ab4281bf50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f611c00a480 tx=0x7f611c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708865d6d38c30d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6124aee640 1 -- 172.21.15.110:0/288869173 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f611c0af070 con 0x55ab42819b60 2024-04-27T03:18:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6124aee640 1 -- 172.21.15.110:0/288869173 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f611c0aa530 con 0x55ab42819b60 2024-04-27T03:18:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6124aee640 1 -- 172.21.15.110:0/288869173 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f611c0aa830 con 0x55ab42819b60 2024-04-27T03:18:08.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6127fd5980 1 -- 172.21.15.110:0/288869173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab42819b60 msgr2=0x55ab4281bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6127fd5980 1 --2- 172.21.15.110:0/288869173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab42819b60 0x55ab4281bf50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f611c00a480 tx=0x7f611c01b280 comp rx=0 tx=0).stop 2024-04-27T03:18:08.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6127fd5980 1 -- 172.21.15.110:0/288869173 shutdown_connections 2024-04-27T03:18:08.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6127fd5980 1 --2- 172.21.15.110:0/288869173 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab42819b60 0x55ab4281bf50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6127fd5980 1 -- 172.21.15.110:0/288869173 >> 172.21.15.110:0/288869173 conn(0x55ab4280d160 msgr2=0x55ab4280f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6127fd5980 1 -- 172.21.15.110:0/288869173 shutdown_connections 2024-04-27T03:18:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.553+0000 7f6127fd5980 1 -- 172.21.15.110:0/288869173 wait complete. 2024-04-27T03:18:08.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6127fd5980 1 Processor -- start 2024-04-27T03:18:08.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6127fd5980 1 -- start start 2024-04-27T03:18:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6127fd5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42900750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6127fd5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab427162d0 con 0x55ab428138c0 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6125af0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42900750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6125af0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42900750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51062/0 (socket says 172.21.15.110:51062) 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6125af0640 1 -- 172.21.15.110:0/2106304733 learned_addr learned my addr 172.21.15.110:0/2106304733 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6125af0640 1 -- 172.21.15.110:0/2106304733 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab4269cae0 con 0x55ab428138c0 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6125af0640 1 --2- 172.21.15.110:0/2106304733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42900750 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f61200015f0 tx=0x7f6120001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f013e85b98d59c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6013e07640 1 -- 172.21.15.110:0/2106304733 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6120004530 con 0x55ab428138c0 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6013e07640 1 -- 172.21.15.110:0/2106304733 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61200046f0 con 0x55ab428138c0 2024-04-27T03:18:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.565+0000 7f6013e07640 1 -- 172.21.15.110:0/2106304733 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6120004a90 con 0x55ab428138c0 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- 172.21.15.110:0/2106304733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 msgr2=0x55ab42900750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 --2- 172.21.15.110:0/2106304733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42900750 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f61200015f0 tx=0x7f6120001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- 172.21.15.110:0/2106304733 shutdown_connections 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 --2- 172.21.15.110:0/2106304733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42900750 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- 172.21.15.110:0/2106304733 >> 172.21.15.110:0/2106304733 conn(0x55ab4280d160 msgr2=0x55ab428fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- 172.21.15.110:0/2106304733 shutdown_connections 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- 172.21.15.110:0/2106304733 wait complete. 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 Processor -- start 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- start start 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42907850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab4281ef20 con 0x55ab428138c0 2024-04-27T03:18:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6125af0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42907850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6125af0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42907850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51078/0 (socket says 172.21.15.110:51078) 2024-04-27T03:18:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6125af0640 1 -- 172.21.15.110:0/1902147606 learned_addr learned my addr 172.21.15.110:0/1902147606 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6125af0640 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab42907d90 con 0x55ab428138c0 2024-04-27T03:18:08.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6125af0640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42907850 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55ab429015e0 tx=0x7f6120000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6012604640 1 -- 172.21.15.110:0/1902147606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6120013760 con 0x55ab428138c0 2024-04-27T03:18:08.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ab42908020 con 0x55ab428138c0 2024-04-27T03:18:08.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6012604640 1 -- 172.21.15.110:0/1902147606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6120013900 con 0x55ab428138c0 2024-04-27T03:18:08.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6012604640 1 -- 172.21.15.110:0/1902147606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6120013be0 con 0x55ab428138c0 2024-04-27T03:18:08.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4290ac00 con 0x55ab428138c0 2024-04-27T03:18:08.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6012604640 1 -- 172.21.15.110:0/1902147606 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6120010070 con 0x55ab428138c0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6012604640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f600c034650 0x7f600c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.569+0000 7f6012604640 1 -- 172.21.15.110:0/1902147606 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f6120018070 con 0x55ab428138c0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f61252ef640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f600c034650 0x7f600c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab42819b60 0x55ab4290e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4290ed00 con 0x55ab42819b60 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f61262f1640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab42819b60 0x55ab4290e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f61252ef640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f600c034650 0x7f600c036b10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f611c0ac000 tx=0x7f611c0a5000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f61262f1640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab42819b60 0x55ab4290e6e0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f61262f1640 1 -- 172.21.15.110:0/1902147606 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f61180083f0 con 0x55ab42819b60 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab4290fad0 0x55ab42911ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab42912500 con 0x55ab4290fad0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6125af0640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab4290fad0 0x55ab42911ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6125af0640 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab4290fad0 0x55ab42911ec0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6125af0640 1 -- 172.21.15.110:0/1902147606 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f612001b030 con 0x55ab4290fad0 2024-04-27T03:18:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab42913db0 con 0x55ab42819b60 2024-04-27T03:18:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f61262f1640 1 -- 172.21.15.110:0/1902147606 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f61180083f0 con 0x55ab42819b60 2024-04-27T03:18:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab42912fe0 con 0x55ab4290fad0 2024-04-27T03:18:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6125af0640 1 -- 172.21.15.110:0/1902147606 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f612001b030 con 0x55ab4290fad0 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.573+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab429133d0 con 0x55ab42819b60 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f61262f1640 1 -- 172.21.15.110:0/1902147606 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f61180083f0 con 0x55ab42819b60 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 Processor -- start 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 -- start start 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab429226c0 0x55ab42922ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab4271a310 con 0x55ab429226c0 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f61262f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab429226c0 0x55ab42922ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f61262f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab429226c0 0x55ab42922ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51082/0 (socket says 172.21.15.110:51082) 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f61262f1640 1 -- 172.21.15.110:0/2540323862 learned_addr learned my addr 172.21.15.110:0/2540323862 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f61262f1640 1 -- 172.21.15.110:0/2540323862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab42923020 con 0x55ab429226c0 2024-04-27T03:18:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f61262f1640 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab429226c0 0x55ab42922ae0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f611800aab0 tx=0x7f611800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f600b7fe640 1 -- 172.21.15.110:0/2540323862 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118007d30 con 0x55ab429226c0 2024-04-27T03:18:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f600b7fe640 1 -- 172.21.15.110:0/2540323862 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f611801d070 con 0x55ab429226c0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ab42923ac0 con 0x55ab429226c0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f600b7fe640 1 -- 172.21.15.110:0/2540323862 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118018440 con 0x55ab429226c0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab42923f90 con 0x55ab429226c0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f600b7fe640 1 -- 172.21.15.110:0/2540323862 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6118013070 con 0x55ab429226c0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f600b7fe640 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6000034650 0x7f6000036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f600b7fe640 1 -- 172.21.15.110:0/2540323862 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f611804b520 con 0x55ab429226c0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6125af0640 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6000034650 0x7f6000036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 Processor -- start 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6125af0640 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6000034650 0x7f6000036b10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x55ab42912b00 tx=0x7f6120010540 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 -- start start 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab4292fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.577+0000 7f6127fd5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab42900ea0 con 0x55ab4292f870 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab4292fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab4292fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51098/0 (socket says 172.21.15.110:51098) 2024-04-27T03:18:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 -- 172.21.15.110:0/1391068213 learned_addr learned my addr 172.21.15.110:0/1391068213 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 -- 172.21.15.110:0/1391068213 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab42930260 con 0x55ab4292f870 2024-04-27T03:18:08.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 --2- 172.21.15.110:0/1391068213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab4292fc90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6118049930 tx=0x7f6118018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926d61444c668d0b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f60097fa640 1 -- 172.21.15.110:0/1391068213 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118047070 con 0x55ab4292f870 2024-04-27T03:18:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f60097fa640 1 -- 172.21.15.110:0/1391068213 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6118018bc0 con 0x55ab4292f870 2024-04-27T03:18:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f60097fa640 1 -- 172.21.15.110:0/1391068213 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118020360 con 0x55ab4292f870 2024-04-27T03:18:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- 172.21.15.110:0/1391068213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 msgr2=0x55ab4292fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1391068213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab4292fc90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6118049930 tx=0x7f6118018950 comp rx=0 tx=0).stop 2024-04-27T03:18:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- 172.21.15.110:0/1391068213 shutdown_connections 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1391068213 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab4292fc90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- 172.21.15.110:0/1391068213 >> 172.21.15.110:0/1391068213 conn(0x55ab4292bc00 msgr2=0x55ab42929c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- 172.21.15.110:0/1391068213 shutdown_connections 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- 172.21.15.110:0/1391068213 wait complete. 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 Processor -- start 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- start start 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab42936d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ab429309d0 con 0x55ab4292f870 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab42936d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab42936d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51110/0 (socket says 172.21.15.110:51110) 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 learned_addr learned my addr 172.21.15.110:0/3796586736 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab42937240 con 0x55ab4292f870 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f61262f1640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab42936d00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f61180168b0 tx=0x7f61180498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f5ff77fe640 1 -- 172.21.15.110:0/3796586736 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118031070 con 0x55ab4292f870 2024-04-27T03:18:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ab429374d0 con 0x55ab4292f870 2024-04-27T03:18:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f5ff77fe640 1 -- 172.21.15.110:0/3796586736 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6118020900 con 0x55ab4292f870 2024-04-27T03:18:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab4293a1b0 con 0x55ab4292f870 2024-04-27T03:18:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.581+0000 7f5ff77fe640 1 -- 172.21.15.110:0/3796586736 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6118049a70 con 0x55ab4292f870 2024-04-27T03:18:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f5ff77fe640 1 -- 172.21.15.110:0/3796586736 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6118034070 con 0x55ab4292f870 2024-04-27T03:18:08.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f5ff77fe640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ff0034650 0x7f5ff0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f5ff77fe640 1 -- 172.21.15.110:0/3796586736 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f611803c380 con 0x55ab4292f870 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6125af0640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ff0034650 0x7f5ff0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab429423d0 0x55ab42944870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab42944fc0 con 0x55ab429423d0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61252ef640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab429423d0 0x55ab42944870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6125af0640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ff0034650 0x7f5ff0036b10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6120010650 tx=0x7f6120044000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61252ef640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab429423d0 0x55ab42944870 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 realm 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab429453f0 con 0x55ab429423d0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab42945cc0 con 0x55ab429423d0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab429466e0 0x55ab42948b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab42949280 con 0x55ab429466e0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61262f1640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab429466e0 0x55ab42948b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61262f1640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab429466e0 0x55ab42948b10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.585+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab42949ad0 con 0x55ab429423d0 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6127fd5980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6127fd5980 20 searching for the correct realm 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ab4294b840 0x55ab4294dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab4294b840 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6125af0640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ab4294b840 0x55ab4294dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6125af0640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ab4294b840 0x55ab4294dc30 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61262f1640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6118036070 0x7f611803b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x7f6118036070 2024-04-27T03:18:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6118036070 0x7f611803b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6118036070 0x7f611803b800 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab4294b840 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.589+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x7f6118036070 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x7f6118036070 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab4294b840 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.593+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab4294b840 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab4294b840 2024-04-27T03:18:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x7f6118036070 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429423d0 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x7f6118036070 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab4294b840 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x7f6118036070 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.597+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab4294b840 2024-04-27T03:18:08.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e260 con 0x55ab429466e0 2024-04-27T03:18:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294a450 con 0x55ab429423d0 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294a060 con 0x55ab429466e0 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294ad40 con 0x55ab429423d0 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 zone default found 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 4 Realm: () 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 zonegroup default 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294e6e0 con 0x7f6118036070 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab429504b0 con 0x55ab4294b840 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab429508d0 con 0x55ab4294b840 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 started sync module instance, tier type = 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.601+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab4294f950 con 0x55ab429466e0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.605+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.605+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94194044828992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab429511c0 con 0x55ab429466e0 2024-04-27T03:18:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.605+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42951a10 con 0x55ab429423d0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.605+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.605+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94194044834240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42951b50 con 0x55ab429423d0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.605+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42952650 con 0x55ab429466e0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.605+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94194044828992] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94194044834240] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94194044835296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42952610 con 0x55ab429466e0 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42952f20 con 0x7f6118036070 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94194044840560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42952610 con 0x7f6118036070 2024-04-27T03:18:08.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab429539a0 con 0x55ab429466e0 2024-04-27T03:18:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94194044840560] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.609+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94194044835296] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94194044838208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab429534b0 con 0x55ab429466e0 2024-04-27T03:18:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42954240 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94194044845456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab429534b0 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42954c80 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94194044838208] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94194044845456] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94194044846512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab429547d0 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42955520 con 0x7f6118036070 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.613+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94194044846512] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'66 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94194044841616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab429547d0 con 0x7f6118036070 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=0 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=1 2024-04-27T03:18:08.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=2 2024-04-27T03:18:08.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=3 2024-04-27T03:18:08.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=4 2024-04-27T03:18:08.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=5 2024-04-27T03:18:08.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=6 2024-04-27T03:18:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94194044841616] v27'67 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 add_watcher() i=7 2024-04-27T03:18:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab429585b0 con 0x55ab429423d0 2024-04-27T03:18:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94194044850288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42955ad0 con 0x55ab429423d0 2024-04-27T03:18:08.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94194044850288] v27'84 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f5fd77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:08.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 5 note: GC not initialized 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f5fd67fc640 20 reqs_thread_entry: start 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.617+0000 7f6127fd5980 20 Filter name: none 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f5f47fff640 20 reqs_thread_entry: start 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4297be00 con 0x55ab429423d0 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~556 out=556b,getxattrs out=75b] v0'0 uv7 ondisk = 0) ==== 231+0+679 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 20 rados_obj.operate() r=0 bl.length=556 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 cache get: name=default.rgw.meta+users.swift+foo:foo2 : miss 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.2 8:45671577:users.swift::foo%3afoo2:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4297cb90 con 0x7f6118036070 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 foo:foo2 [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 194+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 cache put: name=default.rgw.meta+users.swift+foo:foo2 info.flags=0x0 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 adding default.rgw.meta+users.swift+foo:foo2 to cache LRU end 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 cache get: name=default.rgw.meta+users.swift+foo:foo2 : hit (negative entry) 2024-04-27T03:18:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~673 in=673b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab429828f0 con 0x55ab429423d0 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(64 foo [delete,create,call,call,writefull 0~673] v27'8 uv8 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify cookie 94194045035392 in=861b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4297f480 con 0x55ab429466e0 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94194044838208 notify 115964117002 ret 0) ==== 891+0+0 (crc 0 0 0) 0x55ab4293a1b0 con 0x55ab429466e0 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f60097fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117002 cookie 94194044838208 notifier 4380 bl.length()=849 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f60097fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f60097fa640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f60097fa640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f5fe8003eb0 con 0x55ab429466e0 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(65 notify.4 [notify cookie 94194045035392 out=8b] v0'0 uv66 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.621+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(66 notify.4 [notify-ack] v0'0 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94194045035392 notify 115964117002 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55ab429374d0 con 0x55ab429466e0 2024-04-27T03:18:08.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 8.2 8:45671577:users.swift::foo%3afoo2:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42982380 con 0x7f6118036070 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(67 foo:foo2 [delete,create,writefull 0~7] v27'1 uv1 ondisk = 0) ==== 236+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6127fd5980 10 cache put: name=default.rgw.meta+users.swift+foo:foo2 info.flags=0x7 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6127fd5980 10 moving default.rgw.meta+users.swift+foo:foo2 to cache LRU end 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6127fd5980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.swift:foo:foo2, ofs0, ns] 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94194045035392 in=178b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ab4297f430 con 0x55ab429466e0 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94194044828992 notify 115964117003 ret 0) ==== 208+0+0 (crc 0 0 0) 0x55ab429374d0 con 0x55ab429466e0 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6008ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117003 cookie 94194044828992 notifier 4380 bl.length()=166 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6008ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.swift+foo:foo2 info.flags=0x7 2024-04-27T03:18:08.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6008ff9640 10 rgw watcher librados: moving default.rgw.meta+users.swift+foo:foo2 to cache LRU end 2024-04-27T03:18:08.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6008ff9640 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f5f50003410 con 0x55ab429466e0 2024-04-27T03:18:08.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(68 notify.0 [notify cookie 94194045035392 out=8b] v0'0 uv66 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(69 notify.0 [notify-ack] v0'0 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== watch-notify(notify_complete (2) cookie 94194045035392 notify 115964117003 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55ab42937240 con 0x55ab429466e0 2024-04-27T03:18:08.635 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo1","permissions":"full-control"},{"id":"foo:foo2","permissions":""}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo1","secret_key":"gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL","active":true,"create_date":"0.000000"},{"user":"foo:foo2","secret_key":"ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy","active":true,"create_date":"2024-04-27T03:18:08.622385Z"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:08.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f5fd77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.625+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94194044850288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x55ab429423d0 2024-04-27T03:18:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.629+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94194044850288] v27'85 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.629+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94194044828992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x55ab429466e0 2024-04-27T03:18:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.629+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94194044828992] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.629+0000 7f6127fd5980 20 remove_watcher() i=0 2024-04-27T03:18:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.629+0000 7f6127fd5980 2 removed watcher, disabling cache 2024-04-27T03:18:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.629+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94194044834240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x55ab429423d0 2024-04-27T03:18:08.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.633+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94194044834240] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x55ab429423d0 2024-04-27T03:18:08.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.633+0000 7f6127fd5980 20 remove_watcher() i=1 2024-04-27T03:18:08.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.633+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94194044835296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x55ab429466e0 2024-04-27T03:18:08.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.633+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94194044835296] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.633+0000 7f6127fd5980 20 remove_watcher() i=2 2024-04-27T03:18:08.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.633+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94194044840560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x7f6118036070 2024-04-27T03:18:08.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94194044840560] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f6127fd5980 20 remove_watcher() i=3 2024-04-27T03:18:08.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94194044838208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x55ab429466e0 2024-04-27T03:18:08.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f61262f1640 1 -- 172.21.15.110:0/3796586736 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94194044838208] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6118033030 con 0x55ab429466e0 2024-04-27T03:18:08.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f6127fd5980 20 remove_watcher() i=4 2024-04-27T03:18:08.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94194044845456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x55ab4294b840 2024-04-27T03:18:08.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94194044845456] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f6127fd5980 20 remove_watcher() i=5 2024-04-27T03:18:08.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.637+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94194044846512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x55ab4294b840 2024-04-27T03:18:08.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6125af0640 1 -- 172.21.15.110:0/3796586736 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94194044846512] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f612004c040 con 0x55ab4294b840 2024-04-27T03:18:08.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 20 remove_watcher() i=6 2024-04-27T03:18:08.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94194044841616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ab42979af0 con 0x7f6118036070 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94194044841616] v27'68 uv66 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f611c0b2040 con 0x7f6118036070 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 20 remove_watcher() i=7 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab429466e0 msgr2=0x55ab42948b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab429466e0 0x55ab42948b10 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab429423d0 msgr2=0x55ab42944870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab429423d0 0x55ab42944870 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6118036070 msgr2=0x7f611803b800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6118036070 0x7f611803b800 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ab4294b840 msgr2=0x55ab4294dc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ab4294b840 0x55ab4294dc30 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ff0034650 msgr2=0x7f5ff0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ff0034650 0x7f5ff0036b10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6120010650 tx=0x7f6120044000 comp rx=0 tx=0).stop 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 msgr2=0x55ab42936d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f61252ef640 1 -- 172.21.15.110:0/3796586736 reap_dead start 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab42936d00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f61180168b0 tx=0x7f61180498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 shutdown_connections 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/3796586736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab4292f870 0x55ab42936d00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 >> 172.21.15.110:0/3796586736 conn(0x55ab4292bc00 msgr2=0x55ab429310f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 shutdown_connections 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/3796586736 wait complete. 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6000034650 msgr2=0x7f6000036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6000034650 0x7f6000036b10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x55ab42912b00 tx=0x7f6120010540 comp rx=0 tx=0).stop 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab429226c0 msgr2=0x55ab42922ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab429226c0 0x55ab42922ae0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f611800aab0 tx=0x7f611800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 shutdown_connections 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6000034650 0x7f6000036b10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/2540323862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab429226c0 0x55ab42922ae0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 >> 172.21.15.110:0/2540323862 conn(0x55ab429174d0 msgr2=0x55ab42917fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 shutdown_connections 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/2540323862 wait complete. 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab4290fad0 msgr2=0x55ab42911ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab4290fad0 0x55ab42911ec0 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab42819b60 msgr2=0x55ab4290e6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab42819b60 0x55ab4290e6e0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f600c034650 msgr2=0x7f600c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f600c034650 0x7f600c036b10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f611c0ac000 tx=0x7f611c0a5000 comp rx=0 tx=0).stop 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 msgr2=0x55ab42907850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42907850 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55ab429015e0 tx=0x7f6120000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 shutdown_connections 2024-04-27T03:18:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ab4290fad0 0x55ab42911ec0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ab42819b60 0x55ab4290e6e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f600c034650 0x7f600c036b10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 --2- 172.21.15.110:0/1902147606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ab428138c0 0x55ab42907850 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 >> 172.21.15.110:0/1902147606 conn(0x55ab4280d160 msgr2=0x55ab428fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 shutdown_connections 2024-04-27T03:18:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.641+0000 7f6127fd5980 1 -- 172.21.15.110:0/1902147606 wait complete. 2024-04-27T03:18:08.659 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'active': True, 'create_date': '0.000000'}, {'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'active': True, 'create_date': '2024-04-27T03:18:08.622385Z'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:08.659 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:18:08.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:18:08.660 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:18:08.734 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7195fd2980 1 Processor -- start 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7195fd2980 1 -- start start 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7195fd2980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55bc0e88c4b0 0x55bc0e88c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7195fd2980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55bc0e7490d0 con 0x55bc0e88cdd0 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7195fd2980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55bc0e73e430 con 0x55bc0e88c4b0 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7193aed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55bc0e88c4b0 0x55bc0e88c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7193aed640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55bc0e88c4b0 0x55bc0e88c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51126/0 (socket says 172.21.15.110:51126) 2024-04-27T03:18:08.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7193aed640 1 -- 172.21.15.110:0/3671674235 learned_addr learned my addr 172.21.15.110:0/3671674235 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7193aed640 1 -- 172.21.15.110:0/3671674235 >> v1:172.21.15.77:6789/0 conn(0x55bc0e88cdd0 legacy=0x55bc0e88d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.725+0000 7f7193aed640 1 -- 172.21.15.110:0/3671674235 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0e737510 con 0x55bc0e88c4b0 2024-04-27T03:18:08.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7193aed640 1 --2- 172.21.15.110:0/3671674235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e88c890 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f717c00a4e0 tx=0x7f717c01b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58556f4ec0a8fb0b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7192aeb640 1 -- 172.21.15.110:0/3671674235 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f717c0aa540 con 0x55bc0e88c4b0 2024-04-27T03:18:08.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7192aeb640 1 -- 172.21.15.110:0/3671674235 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717c0aa6e0 con 0x55bc0e88c4b0 2024-04-27T03:18:08.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7192aeb640 1 -- 172.21.15.110:0/3671674235 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f717c0aaa40 con 0x55bc0e88c4b0 2024-04-27T03:18:08.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7195fd2980 1 -- 172.21.15.110:0/3671674235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 msgr2=0x55bc0e88c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3671674235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e88c890 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f717c00a4e0 tx=0x7f717c01b090 comp rx=0 tx=0).stop 2024-04-27T03:18:08.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7195fd2980 1 -- 172.21.15.110:0/3671674235 shutdown_connections 2024-04-27T03:18:08.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3671674235 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e88c890 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7195fd2980 1 -- 172.21.15.110:0/3671674235 >> 172.21.15.110:0/3671674235 conn(0x55bc0e7ffb90 msgr2=0x55bc0e7fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7195fd2980 1 -- 172.21.15.110:0/3671674235 shutdown_connections 2024-04-27T03:18:08.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.729+0000 7f7195fd2980 1 -- 172.21.15.110:0/3671674235 wait complete. 2024-04-27T03:18:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.741+0000 7f7195fd2980 1 Processor -- start 2024-04-27T03:18:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.741+0000 7f7195fd2980 1 -- start start 2024-04-27T03:18:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.741+0000 7f7195fd2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e977d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.741+0000 7f7195fd2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bc0e78d2d0 con 0x55bc0e88c4b0 2024-04-27T03:18:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.741+0000 7f7193aed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e977d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e977d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51128/0 (socket says 172.21.15.110:51128) 2024-04-27T03:18:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 -- 172.21.15.110:0/3910925290 learned_addr learned my addr 172.21.15.110:0/3910925290 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 -- 172.21.15.110:0/3910925290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0e713ae0 con 0x55bc0e88c4b0 2024-04-27T03:18:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 --2- 172.21.15.110:0/3910925290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e977d10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f717c017ca0 tx=0x7f717c0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec75c97ca05f088 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7081e0b640 1 -- 172.21.15.110:0/3910925290 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f717c0b9030 con 0x55bc0e88c4b0 2024-04-27T03:18:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7081e0b640 1 -- 172.21.15.110:0/3910925290 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717c0aaac0 con 0x55bc0e88c4b0 2024-04-27T03:18:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7081e0b640 1 -- 172.21.15.110:0/3910925290 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f717c0aada0 con 0x55bc0e88c4b0 2024-04-27T03:18:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- 172.21.15.110:0/3910925290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 msgr2=0x55bc0e977d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3910925290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e977d10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f717c017ca0 tx=0x7f717c0033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- 172.21.15.110:0/3910925290 shutdown_connections 2024-04-27T03:18:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3910925290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88c4b0 0x55bc0e977d10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- 172.21.15.110:0/3910925290 >> 172.21.15.110:0/3910925290 conn(0x55bc0e7ffb90 msgr2=0x55bc0e9738a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- 172.21.15.110:0/3910925290 shutdown_connections 2024-04-27T03:18:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- 172.21.15.110:0/3910925290 wait complete. 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 Processor -- start 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- start start 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88cdd0 0x55bc0e97ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bc0e896500 con 0x55bc0e88cdd0 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88cdd0 0x55bc0e97ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88cdd0 0x55bc0e97ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51134/0 (socket says 172.21.15.110:51134) 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 -- 172.21.15.110:0/3263091135 learned_addr learned my addr 172.21.15.110:0/3263091135 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0e97f3c0 con 0x55bc0e88cdd0 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7193aed640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88cdd0 0x55bc0e97ee80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f717c01b0d0 tx=0x7f717c017e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f707bfff640 1 -- 172.21.15.110:0/3263091135 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f717c0b9030 con 0x55bc0e88cdd0 2024-04-27T03:18:08.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bc0e97f650 con 0x55bc0e88cdd0 2024-04-27T03:18:08.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f707bfff640 1 -- 172.21.15.110:0/3263091135 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717c0c3070 con 0x55bc0e88cdd0 2024-04-27T03:18:08.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0e982280 con 0x55bc0e88cdd0 2024-04-27T03:18:08.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.745+0000 7f707bfff640 1 -- 172.21.15.110:0/3263091135 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f717c0cc060 con 0x55bc0e88cdd0 2024-04-27T03:18:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f707bfff640 1 -- 172.21.15.110:0/3263091135 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f717c0c5030 con 0x55bc0e88cdd0 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f707bfff640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f707c034650 0x7f707c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f707bfff640 1 -- 172.21.15.110:0/3263091135 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f717c0ba070 con 0x55bc0e88cdd0 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f71932ec640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f707c034650 0x7f707c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e983d50 0x55bc0e986140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9866c0 con 0x55bc0e983d50 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f71942ee640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e983d50 0x55bc0e986140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f71932ec640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f707c034650 0x7f707c036b10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f718c001e20 tx=0x7f718c0013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f71942ee640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e983d50 0x55bc0e986140 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f71942ee640 1 -- 172.21.15.110:0/3263091135 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f71840083f0 con 0x55bc0e983d50 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e987490 0x55bc0e989880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e989ee0 con 0x55bc0e987490 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f7193aed640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e987490 0x55bc0e989880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.749+0000 7f7193aed640 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e987490 0x55bc0e989880 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7193aed640 1 -- 172.21.15.110:0/3263091135 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e987490 2024-04-27T03:18:08.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e98b790 con 0x55bc0e983d50 2024-04-27T03:18:08.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f71942ee640 1 -- 172.21.15.110:0/3263091135 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f71840083f0 con 0x55bc0e983d50 2024-04-27T03:18:08.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e98a9c0 con 0x55bc0e987490 2024-04-27T03:18:08.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7193aed640 1 -- 172.21.15.110:0/3263091135 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e987490 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e98adb0 con 0x55bc0e983d50 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f71942ee640 1 -- 172.21.15.110:0/3263091135 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f71840083f0 con 0x55bc0e983d50 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 Processor -- start 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 -- start start 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e99a0d0 0x55bc0e99a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bc0e791310 con 0x55bc0e99a0d0 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f71942ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e99a0d0 0x55bc0e99a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f71942ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e99a0d0 0x55bc0e99a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51146/0 (socket says 172.21.15.110:51146) 2024-04-27T03:18:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f71942ee640 1 -- 172.21.15.110:0/4036839187 learned_addr learned my addr 172.21.15.110:0/4036839187 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f71942ee640 1 -- 172.21.15.110:0/4036839187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0e99aa30 con 0x55bc0e99a0d0 2024-04-27T03:18:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f71942ee640 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e99a0d0 0x55bc0e99a4f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f718400aae0 tx=0x7f7184008ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f70797fa640 1 -- 172.21.15.110:0/4036839187 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7184007ce0 con 0x55bc0e99a0d0 2024-04-27T03:18:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f70797fa640 1 -- 172.21.15.110:0/4036839187 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184018440 con 0x55bc0e99a0d0 2024-04-27T03:18:08.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bc0e99b4d0 con 0x55bc0e99a0d0 2024-04-27T03:18:08.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f70797fa640 1 -- 172.21.15.110:0/4036839187 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7184018720 con 0x55bc0e99a0d0 2024-04-27T03:18:08.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.753+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0e99b9a0 con 0x55bc0e99a0d0 2024-04-27T03:18:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f70797fa640 1 -- 172.21.15.110:0/4036839187 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7184013070 con 0x55bc0e99a0d0 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f70797fa640 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f706c034760 0x7f706c036c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f70797fa640 1 -- 172.21.15.110:0/4036839187 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f718404b710 con 0x55bc0e99a0d0 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7193aed640 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f706c034760 0x7f706c036c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 Processor -- start 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- start start 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7193aed640 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f706c034760 0x7f706c036c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55bc0e98a4e0 tx=0x7f717c0e3540 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bc0e9783a0 con 0x55bc0e9a7280 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f71942ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f71942ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9a76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51156/0 (socket says 172.21.15.110:51156) 2024-04-27T03:18:08.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f71942ee640 1 -- 172.21.15.110:0/1117227724 learned_addr learned my addr 172.21.15.110:0/1117227724 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f71942ee640 1 -- 172.21.15.110:0/1117227724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0e9a7c70 con 0x55bc0e9a7280 2024-04-27T03:18:08.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f71942ee640 1 --2- 172.21.15.110:0/1117227724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9a76a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7184049b20 tx=0x7f71840163f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31dff97a07a33ce server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7072ffd640 1 -- 172.21.15.110:0/1117227724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7184040070 con 0x55bc0e9a7280 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7072ffd640 1 -- 172.21.15.110:0/1117227724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184013ba0 con 0x55bc0e9a7280 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7072ffd640 1 -- 172.21.15.110:0/1117227724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7184013ea0 con 0x55bc0e9a7280 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- 172.21.15.110:0/1117227724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 msgr2=0x55bc0e9a76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 --2- 172.21.15.110:0/1117227724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9a76a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7184049b20 tx=0x7f71840163f0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- 172.21.15.110:0/1117227724 shutdown_connections 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 --2- 172.21.15.110:0/1117227724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9a76a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- 172.21.15.110:0/1117227724 >> 172.21.15.110:0/1117227724 conn(0x55bc0e9a3610 msgr2=0x55bc0e9a1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- 172.21.15.110:0/1117227724 shutdown_connections 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- 172.21.15.110:0/1117227724 wait complete. 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 Processor -- start 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.757+0000 7f7195fd2980 1 -- start start 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bc0e9a8310 con 0x55bc0e9a7280 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71942ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71942ee640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51172/0 (socket says 172.21.15.110:51172) 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 learned_addr learned my addr 172.21.15.110:0/21970897 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc0e9aeca0 con 0x55bc0e9a7280 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71942ee640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9ae760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7184016fd0 tx=0x7f7184049ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f70717fa640 1 -- 172.21.15.110:0/21970897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7184040040 con 0x55bc0e9a7280 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bc0e9aef30 con 0x55bc0e9a7280 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc0e9b1c10 con 0x55bc0e9a7280 2024-04-27T03:18:08.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f70717fa640 1 -- 172.21.15.110:0/21970897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f718404b430 con 0x55bc0e9a7280 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f70717fa640 1 -- 172.21.15.110:0/21970897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71840189f0 con 0x55bc0e9a7280 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f70717fa640 1 -- 172.21.15.110:0/21970897 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7184033020 con 0x55bc0e9a7280 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f70717fa640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f705c034650 0x7f705c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f70717fa640 1 -- 172.21.15.110:0/21970897 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7184031020 con 0x55bc0e9a7280 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7193aed640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f705c034650 0x7f705c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e9b9da0 0x55bc0e9bc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9bc990 con 0x55bc0e9b9da0 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71932ec640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e9b9da0 0x55bc0e9bc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7193aed640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f705c034650 0x7f705c036b10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f717c0e5110 tx=0x7f717c0e0000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71932ec640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e9b9da0 0x55bc0e9bc240 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f718c00d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 20 realm 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.761+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9b92b0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f718c00d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9bd490 con 0x55bc0e9b9da0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f718c00d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e9be270 0x55bc0e9c0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f71942ee640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e9be270 0x55bc0e9c0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f71942ee640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e9be270 0x55bc0e9c0660 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c1590 con 0x55bc0e9b9da0 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f718c00d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 20 searching for the correct realm 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bc0e9c3300 0x55bc0e9c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c5cd0 con 0x55bc0e9c3300 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7193aed640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bc0e9c3300 0x55bc0e9c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7193aed640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bc0e9c3300 0x55bc0e9c56f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c5cd0 con 0x55bc0e9be270 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.765+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71942ee640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7184022c20 0x7f7184046ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x7f7184022c20 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7184022c20 0x7f7184046ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7184022c20 0x7f7184046ab0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c00d040 con 0x7f7184022c20 2024-04-27T03:18:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c00d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9c3300 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x7f7184022c20 2024-04-27T03:18:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x7f7184022c20 2024-04-27T03:18:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.769+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x7f7184022c20 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f718c02a040 con 0x7f7184022c20 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9c3300 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9c3300 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9c3300 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x7f7184022c20 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x7f7184022c20 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c02a040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x7f7184022c20 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.773+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9c3300 2024-04-27T03:18:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x7f7184022c20 2024-04-27T03:18:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9c3300 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c0cf0 con 0x55bc0e9be270 2024-04-27T03:18:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c1de0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.777+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c21d0 con 0x55bc0e9be270 2024-04-27T03:18:08.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c21d0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 zone default found 2024-04-27T03:18:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 4 Realm: () 2024-04-27T03:18:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 zonegroup default 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c25f0 con 0x7f7184022c20 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c25f0 con 0x55bc0e9c3300 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c7b30 con 0x55bc0e9c3300 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 started sync module instance, tier type = 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.781+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c6fd0 con 0x55bc0e9be270 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94266187347184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c6fd0 con 0x55bc0e9be270 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c8530 con 0x55bc0e9b9da0 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94266187350080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c8460 con 0x55bc0e9b9da0 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c9420 con 0x55bc0e9be270 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94266187350080] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94266187347184] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94266187344672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c8f70 con 0x55bc0e9be270 2024-04-27T03:18:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.785+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c9cc0 con 0x7f7184022c20 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94266187356176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c8f70 con 0x7f7184022c20 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9ca780 con 0x55bc0e9be270 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94266187344672] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94266187356176] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94266187357232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9ca740 con 0x55bc0e9be270 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cab70 con 0x55bc0e9c3300 2024-04-27T03:18:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94266187352400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cb410 con 0x55bc0e9c3300 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.789+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cbe10 con 0x55bc0e9c3300 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94266187357232] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94266187364640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc6b0 con 0x55bc0e9c3300 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9ccf80 con 0x7f7184022c20 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94266187352400] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94266187364640] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'69 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94266187362144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc6b0 con 0x7f7184022c20 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=0 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=1 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=2 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=3 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=4 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=5 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=6 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94266187362144] v27'70 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 20 add_watcher() i=7 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc6b0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.793+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94266187366912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc620 con 0x55bc0e9b9da0 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94266187366912] v27'86 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7049fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 5 note: GC not initialized 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7048fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 20 Filter name: none 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f6fc27fc640 20 reqs_thread_entry: start 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9f3290 con 0x55bc0e9b9da0 2024-04-27T03:18:08.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~673 out=673b,getxattrs out=75b] v0'0 uv8 ondisk = 0) ==== 231+0+796 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 20 rados_obj.operate() r=0 bl.length=673 2024-04-27T03:18:08.810 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo1","permissions":"full-control"},{"id":"foo:foo2","permissions":""}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo1","secret_key":"gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL","active":true,"create_date":"0.000000"},{"user":"foo:foo2","secret_key":"ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy","active":true,"create_date":"2024-04-27T03:18:08.622385Z"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:08.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7049fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.797+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94266187366912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9f37d0 con 0x55bc0e9b9da0 2024-04-27T03:18:08.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.801+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94266187366912] v27'87 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.801+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94266187347184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc200 con 0x55bc0e9be270 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.801+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94266187347184] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.801+0000 7f7195fd2980 20 remove_watcher() i=0 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.801+0000 7f7195fd2980 2 removed watcher, disabling cache 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.801+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94266187350080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc200 con 0x55bc0e9b9da0 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94266187350080] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x55bc0e9b9da0 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f7195fd2980 20 remove_watcher() i=1 2024-04-27T03:18:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94266187344672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c8040 con 0x55bc0e9be270 2024-04-27T03:18:08.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94266187344672] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f7195fd2980 20 remove_watcher() i=2 2024-04-27T03:18:08.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94266187356176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc200 con 0x7f7184022c20 2024-04-27T03:18:08.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94266187356176] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f7195fd2980 20 remove_watcher() i=3 2024-04-27T03:18:08.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.805+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94266187357232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c6b20 con 0x55bc0e9be270 2024-04-27T03:18:08.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f71942ee640 1 -- 172.21.15.110:0/21970897 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94266187357232] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7184043040 con 0x55bc0e9be270 2024-04-27T03:18:08.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7195fd2980 20 remove_watcher() i=4 2024-04-27T03:18:08.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94266187352400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c9cc0 con 0x55bc0e9c3300 2024-04-27T03:18:08.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94266187352400] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7195fd2980 20 remove_watcher() i=5 2024-04-27T03:18:08.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94266187364640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9cc200 con 0x55bc0e9c3300 2024-04-27T03:18:08.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7193aed640 1 -- 172.21.15.110:0/21970897 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94266187364640] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f717c0bf410 con 0x55bc0e9c3300 2024-04-27T03:18:08.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7195fd2980 20 remove_watcher() i=6 2024-04-27T03:18:08.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.809+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94266187362144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bc0e9c8950 con 0x7f7184022c20 2024-04-27T03:18:08.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94266187362144] v27'71 uv69 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f718c03d040 con 0x7f7184022c20 2024-04-27T03:18:08.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 20 remove_watcher() i=7 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e9be270 msgr2=0x55bc0e9c0660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e9be270 0x55bc0e9c0660 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e9b9da0 msgr2=0x55bc0e9bc240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e9b9da0 0x55bc0e9bc240 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7184022c20 msgr2=0x7f7184046ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7184022c20 0x7f7184046ab0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bc0e9c3300 msgr2=0x55bc0e9c56f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bc0e9c3300 0x55bc0e9c56f0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f705c034650 msgr2=0x7f705c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f705c034650 0x7f705c036b10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f717c0e5110 tx=0x7f717c0e0000 comp rx=0 tx=0).stop 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 msgr2=0x55bc0e9ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f71932ec640 1 -- 172.21.15.110:0/21970897 reap_dead start 2024-04-27T03:18:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9ae760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7184016fd0 tx=0x7f7184049ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 shutdown_connections 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/21970897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e9a7280 0x55bc0e9ae760 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 >> 172.21.15.110:0/21970897 conn(0x55bc0e9a3610 msgr2=0x55bc0e9a0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 shutdown_connections 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/21970897 wait complete. 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f706c034760 msgr2=0x7f706c036c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f706c034760 0x7f706c036c20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55bc0e98a4e0 tx=0x7f717c0e3540 comp rx=0 tx=0).stop 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e99a0d0 msgr2=0x55bc0e99a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e99a0d0 0x55bc0e99a4f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f718400aae0 tx=0x7f7184008ed0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 shutdown_connections 2024-04-27T03:18:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f706c034760 0x7f706c036c20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/4036839187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e99a0d0 0x55bc0e99a4f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 >> 172.21.15.110:0/4036839187 conn(0x55bc0e98eee0 msgr2=0x55bc0e98f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 shutdown_connections 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/4036839187 wait complete. 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e987490 msgr2=0x55bc0e989880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e987490 0x55bc0e989880 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e983d50 msgr2=0x55bc0e986140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e983d50 0x55bc0e986140 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f707c034650 msgr2=0x7f707c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f707c034650 0x7f707c036b10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f718c001e20 tx=0x7f718c0013a0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88cdd0 msgr2=0x55bc0e97ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88cdd0 0x55bc0e97ee80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f717c01b0d0 tx=0x7f717c017e30 comp rx=0 tx=0).stop 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 shutdown_connections 2024-04-27T03:18:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bc0e987490 0x55bc0e989880 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bc0e983d50 0x55bc0e986140 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f707c034650 0x7f707c036b10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 --2- 172.21.15.110:0/3263091135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bc0e88cdd0 0x55bc0e97ee80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 >> 172.21.15.110:0/3263091135 conn(0x55bc0e7ffb90 msgr2=0x55bc0e9723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 shutdown_connections 2024-04-27T03:18:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.813+0000 7f7195fd2980 1 -- 172.21.15.110:0/3263091135 wait complete. 2024-04-27T03:18:08.830 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'active': True, 'create_date': '0.000000'}, {'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'active': True, 'create_date': '2024-04-27T03:18:08.622385Z'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:08.831 INFO:tasks.util.rgw:rgwadmin: client.0 : ['key', 'rm', '--subuser', 'foo:foo1', '--key-type', 'swift'] 2024-04-27T03:18:08.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'key', 'rm', '--subuser', 'foo:foo1', '--key-type', 'swift'] 2024-04-27T03:18:08.831 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph key rm --subuser foo:foo1 --key-type swift 2024-04-27T03:18:08.908 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:08.909 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:08.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b6eab980 1 Processor -- start 2024-04-27T03:18:08.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b6eab980 1 -- start start 2024-04-27T03:18:08.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b6eab980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a1888ca7a0 0x55a1888ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b6eab980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a1887850d0 con 0x55a1888c62e0 2024-04-27T03:18:08.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b6eab980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a18877a430 con 0x55a1888ca7a0 2024-04-27T03:18:08.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b41c5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a1888ca7a0 0x55a1888ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b41c5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a1888ca7a0 0x55a1888ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51174/0 (socket says 172.21.15.110:51174) 2024-04-27T03:18:08.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1953063569 learned_addr learned my addr 172.21.15.110:0/1953063569 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1953063569 >> v1:172.21.15.77:6789/0 conn(0x55a1888c62e0 legacy=0x55a1888c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.901+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1953063569 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a188773510 con 0x55a1888ca7a0 2024-04-27T03:18:08.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b41c5640 1 --2- 172.21.15.110:0/1953063569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888ca7a0 0x55a1888ccb90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800a5c0 tx=0x7fe5a801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac219440f0976c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b39c4640 1 -- 172.21.15.110:0/1953063569 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5a80a6070 con 0x55a1888ca7a0 2024-04-27T03:18:08.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b39c4640 1 -- 172.21.15.110:0/1953063569 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a80af070 con 0x55a1888ca7a0 2024-04-27T03:18:08.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b39c4640 1 -- 172.21.15.110:0/1953063569 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5a80aa5c0 con 0x55a1888ca7a0 2024-04-27T03:18:08.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1953063569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888ca7a0 msgr2=0x55a1888ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1953063569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888ca7a0 0x55a1888ccb90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800a5c0 tx=0x7fe5a801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:08.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1953063569 shutdown_connections 2024-04-27T03:18:08.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1953063569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888ca7a0 0x55a1888ccb90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1953063569 >> 172.21.15.110:0/1953063569 conn(0x55a1888c1bb0 msgr2=0x55a1888c3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1953063569 shutdown_connections 2024-04-27T03:18:08.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.905+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1953063569 wait complete. 2024-04-27T03:18:08.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.917+0000 7fe5b6eab980 1 Processor -- start 2024-04-27T03:18:08.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.917+0000 7fe5b6eab980 1 -- start start 2024-04-27T03:18:08.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.917+0000 7fe5b6eab980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.917+0000 7fe5b6eab980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a1887c92d0 con 0x55a1888c62e0 2024-04-27T03:18:08.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51186/0 (socket says 172.21.15.110:51186) 2024-04-27T03:18:08.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 -- 172.21.15.110:0/3043222456 learned_addr learned my addr 172.21.15.110:0/3043222456 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 -- 172.21.15.110:0/3043222456 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a18874fae0 con 0x55a1888c62e0 2024-04-27T03:18:08.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/3043222456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b13d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40015f0 tx=0x7fe5a4001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9c7a7235e56545 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe4a2e0d640 1 -- 172.21.15.110:0/3043222456 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5a4016070 con 0x55a1888c62e0 2024-04-27T03:18:08.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe4a2e0d640 1 -- 172.21.15.110:0/3043222456 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a400f640 con 0x55a1888c62e0 2024-04-27T03:18:08.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe4a2e0d640 1 -- 172.21.15.110:0/3043222456 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5a400f960 con 0x55a1888c62e0 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3043222456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 msgr2=0x55a1889b13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/3043222456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b13d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe5a40015f0 tx=0x7fe5a4001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3043222456 shutdown_connections 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/3043222456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b13d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3043222456 >> 172.21.15.110:0/3043222456 conn(0x55a1888c1bb0 msgr2=0x55a1889acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3043222456 shutdown_connections 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3043222456 wait complete. 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 Processor -- start 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- start start 2024-04-27T03:18:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a1888cfbc0 con 0x55a1888c62e0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51188/0 (socket says 172.21.15.110:51188) 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 -- 172.21.15.110:0/2239740615 learned_addr learned my addr 172.21.15.110:0/2239740615 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1889b8920 con 0x55a1888c62e0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b83e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4001af0 tx=0x7fe5a4000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe4a160a640 1 -- 172.21.15.110:0/2239740615 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5a4016040 con 0x55a1888c62e0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a1889b8bb0 con 0x55a1888c62e0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe4a160a640 1 -- 172.21.15.110:0/2239740615 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a4022070 con 0x55a1888c62e0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1889bb830 con 0x55a1888c62e0 2024-04-27T03:18:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.921+0000 7fe4a160a640 1 -- 172.21.15.110:0/2239740615 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5a4006050 con 0x55a1888c62e0 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe4a160a640 1 -- 172.21.15.110:0/2239740615 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe5a40061f0 con 0x55a1888c62e0 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe4a160a640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe49c034650 0x7fe49c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe4a160a640 1 -- 172.21.15.110:0/2239740615 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe5a4049700 con 0x55a1888c62e0 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b41c5640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe49c034650 0x7fe49c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1888ca7a0 0x55a1889bf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889bf9a0 con 0x55a1888ca7a0 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1888ca7a0 0x55a1889bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b41c5640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe49c034650 0x7fe49c036b10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80ae000 tx=0x7fe5a80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1888ca7a0 0x55a1889bf330 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b51c7640 1 -- 172.21.15.110:0/2239740615 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe5ac0083f0 con 0x55a1888ca7a0 2024-04-27T03:18:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889c0790 0x55a1889c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889c31c0 con 0x55a1889c0790 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889c0790 0x55a1889c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889c0790 0x55a1889c2b80 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b49c6640 1 -- 172.21.15.110:0/2239740615 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe5a400f070 con 0x55a1889c0790 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.925+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889c4a70 con 0x55a1888ca7a0 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b51c7640 1 -- 172.21.15.110:0/2239740615 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fe5ac0083f0 con 0x55a1888ca7a0 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889c3ca0 con 0x55a1889c0790 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b49c6640 1 -- 172.21.15.110:0/2239740615 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe5a400f070 con 0x55a1889c0790 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889c4090 con 0x55a1888ca7a0 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b51c7640 1 -- 172.21.15.110:0/2239740615 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fe5ac0083f0 con 0x55a1888ca7a0 2024-04-27T03:18:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 Processor -- start 2024-04-27T03:18:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 -- start start 2024-04-27T03:18:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889d33b0 0x55a1889d37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a1887cd310 con 0x55a1889d33b0 2024-04-27T03:18:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b51c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889d33b0 0x55a1889d37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b51c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889d33b0 0x55a1889d37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51190/0 (socket says 172.21.15.110:51190) 2024-04-27T03:18:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b51c7640 1 -- 172.21.15.110:0/3214235287 learned_addr learned my addr 172.21.15.110:0/3214235287 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b51c7640 1 -- 172.21.15.110:0/3214235287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1889d3d10 con 0x55a1889d33b0 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889d33b0 0x55a1889d37d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00aab0 tx=0x7fe5ac00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe49a7fc640 1 -- 172.21.15.110:0/3214235287 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5ac007d30 con 0x55a1889d33b0 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a1889d47b0 con 0x55a1889d33b0 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe49a7fc640 1 -- 172.21.15.110:0/3214235287 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5ac01d070 con 0x55a1889d33b0 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1889d4c80 con 0x55a1889d33b0 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.929+0000 7fe49a7fc640 1 -- 172.21.15.110:0/3214235287 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5ac022460 con 0x55a1889d33b0 2024-04-27T03:18:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe49a7fc640 1 -- 172.21.15.110:0/3214235287 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe5ac013070 con 0x55a1889d33b0 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe49a7fc640 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe48c034650 0x7fe48c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe49a7fc640 1 -- 172.21.15.110:0/3214235287 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe5ac04b5c0 con 0x55a1889d33b0 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe48c034650 0x7fe48c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 Processor -- start 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe48c034650 0x7fe48c036b10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55a1889c37c0 tx=0x7fe5a4033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- start start 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a1889b1b20 con 0x55a1889e0530 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b51c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b51c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51192/0 (socket says 172.21.15.110:51192) 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b51c7640 1 -- 172.21.15.110:0/831637238 learned_addr learned my addr 172.21.15.110:0/831637238 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b51c7640 1 -- 172.21.15.110:0/831637238 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1889e0f20 con 0x55a1889e0530 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/831637238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e0950 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac0204a0 tx=0x7fe5ac049b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98716a041a0b1911 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe48bfff640 1 -- 172.21.15.110:0/831637238 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5ac047070 con 0x55a1889e0530 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe48bfff640 1 -- 172.21.15.110:0/831637238 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5ac03b2e0 con 0x55a1889e0530 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe48bfff640 1 -- 172.21.15.110:0/831637238 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5ac021c90 con 0x55a1889e0530 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- 172.21.15.110:0/831637238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 msgr2=0x55a1889e0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/831637238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e0950 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac0204a0 tx=0x7fe5ac049b10 comp rx=0 tx=0).stop 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- 172.21.15.110:0/831637238 shutdown_connections 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/831637238 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e0950 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- 172.21.15.110:0/831637238 >> 172.21.15.110:0/831637238 conn(0x55a1889dc870 msgr2=0x55a1889da920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- 172.21.15.110:0/831637238 shutdown_connections 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- 172.21.15.110:0/831637238 wait complete. 2024-04-27T03:18:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 Processor -- start 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.933+0000 7fe5b6eab980 1 -- start start 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a1889e15c0 con 0x55a1889e0530 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b51c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b51c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51208/0 (socket says 172.21.15.110:51208) 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 learned_addr learned my addr 172.21.15.110:0/1266807099 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1889e7e90 con 0x55a1889e0530 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e7950 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac016ae0 tx=0x7fe5ac0498a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe48a7fc640 1 -- 172.21.15.110:0/1266807099 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5ac030070 con 0x55a1889e0530 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a1889e8120 con 0x55a1889e0530 2024-04-27T03:18:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe48a7fc640 1 -- 172.21.15.110:0/1266807099 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5ac022ad0 con 0x55a1889e0530 2024-04-27T03:18:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1889eae00 con 0x55a1889e0530 2024-04-27T03:18:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe48a7fc640 1 -- 172.21.15.110:0/1266807099 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5ac03a060 con 0x55a1889e0530 2024-04-27T03:18:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe48a7fc640 1 -- 172.21.15.110:0/1266807099 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe5ac033070 con 0x55a1889e0530 2024-04-27T03:18:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe48a7fc640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe480034600 0x7fe480036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe48a7fc640 1 -- 172.21.15.110:0/1266807099 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe5ac01a870 con 0x55a1889e0530 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe480034600 0x7fe480036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1889f2fe0 0x55a1889f5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889f5bc0 con 0x55a1889f2fe0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b41c5640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1889f2fe0 0x55a1889f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe480034600 0x7fe480036ac0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4026000 tx=0x7fe5a403a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b41c5640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1889f2fe0 0x55a1889f5470 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 20 realm 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889f5ff0 con 0x55a1889f2fe0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.937+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889f68c0 con 0x55a1889f2fe0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889f72e0 0x55a1889f9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889f9e80 con 0x55a1889f72e0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889f72e0 0x55a1889f9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889f72e0 0x55a1889f9710 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889fa6d0 con 0x55a1889f2fe0 2024-04-27T03:18:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 20 searching for the correct realm 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a1889fc440 0x55a1889fe830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889fc440 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a1889fc440 0x55a1889fe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b49c6640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a1889fc440 0x55a1889fe830 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.941+0000 7fe5b51c7640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe5ac035070 0x7fe5ac04a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x7fe5ac035070 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe5ac035070 0x7fe5ac04a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe5ac035070 0x7fe5ac04a960 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889fc440 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x7fe5ac035070 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.945+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x7fe5ac035070 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889fc440 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889fc440 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889fc440 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x7fe5ac035070 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f2fe0 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.949+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x7fe5ac035070 2024-04-27T03:18:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889fc440 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x7fe5ac035070 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889fc440 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a1889feeb0 con 0x55a1889f72e0 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.953+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889fb050 con 0x55a1889f2fe0 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889fb8f0 con 0x55a1889f72e0 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889fac90 con 0x55a1889f2fe0 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 zone default found 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 4 Realm: () 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 zonegroup default 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a1889ff7c0 con 0x7fe5ac035070 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a188a01a40 con 0x55a1889fc440 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a188a01de0 con 0x55a1889fc440 2024-04-27T03:18:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:08.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 started sync module instance, tier type = 2024-04-27T03:18:08.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:08.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.957+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a00ed0 con 0x55a1889f72e0 2024-04-27T03:18:08.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94152270287552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a02630 con 0x55a1889f72e0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a02e80 con 0x55a1889f2fe0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94152270292528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a02fe0 con 0x55a1889f2fe0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a03b00 con 0x55a1889f72e0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94152270292528] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94152270287552] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94152270291328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a03ac0 con 0x55a1889f72e0 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.961+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a043a0 con 0x7fe5ac035070 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94152270298864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a03ac0 con 0x7fe5ac035070 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a04e30 con 0x55a1889f72e0 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94152270291328] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94152270298864] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94152270299920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a04e30 con 0x55a1889f72e0 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a05220 con 0x55a1889fc440 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94152270296528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a05a70 con 0x55a1889fc440 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.965+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a06470 con 0x55a1889fc440 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94152270309472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a07130 con 0x55a1889fc440 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a07b80 con 0x7fe5ac035070 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94152270299920] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94152270296528] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94152270309472] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'72 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94152270304704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a04960 con 0x7fe5ac035070 2024-04-27T03:18:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=0 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=1 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=2 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=3 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=4 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=5 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=6 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94152270304704] v27'73 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 20 add_watcher() i=7 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a188a06c80 con 0x55a1889f2fe0 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.969+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94152270310528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a070d0 con 0x55a1889f2fe0 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94152270310528] v27'88 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe46afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 5 note: GC not initialized 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe469fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 20 Filter name: none 2024-04-27T03:18:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe3db7fe640 20 reqs_thread_entry: start 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 20 rados->read ofs=0 len=0 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2dcf0 con 0x55a1889f2fe0 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~673 out=673b,getxattrs out=75b] v0'0 uv8 ondisk = 0) ==== 231+0+796 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 20 rados_obj.operate() r=0 bl.length=673 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a348c0 con 0x55a1889f2fe0 2024-04-27T03:18:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [delete,create,call,call,writefull 0~582] v27'9 uv9 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.973+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94152270496560 in=770b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a188a30030 con 0x55a1889f72e0 2024-04-27T03:18:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94152270299920 notify 115964117004 ret 0) ==== 800+0+0 (crc 0 0 0) 0x55a1889eae00 con 0x55a1889f72e0 2024-04-27T03:18:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe48bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117004 cookie 94152270299920 notifier 4394 bl.length()=758 2024-04-27T03:18:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe48bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe48bfff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe48bfff640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fe478003eb0 con 0x55a1889f72e0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94152270496560 out=8b] v0'0 uv72 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94152270496560 notify 115964117004 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a1889e8120 con 0x55a1889f72e0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 8.8 8:17d68d04:users.swift::foo%3afoo1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a34330 con 0x55a1889f2fe0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(66 foo:foo1 [delete] v27'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b6eab980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.swift:foo:foo1, ofs0, ns] 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94152270496560 in=171b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2fb80 con 0x55a1889f72e0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94152270287552 notify 115964117005 ret 0) ==== 201+0+0 (crc 0 0 0) 0x55a1889e8120 con 0x55a1889f72e0 2024-04-27T03:18:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe48b7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117005 cookie 94152270287552 notifier 4394 bl.length()=159 2024-04-27T03:18:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe48b7fe640 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fe3e80032d0 con 0x55a1889f72e0 2024-04-27T03:18:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(67 notify.0 [notify cookie 94152270496560 out=8b] v0'0 uv72 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.977+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(68 notify.0 [notify-ack] v0'0 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.981+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== watch-notify(notify_complete (2) cookie 94152270496560 notify 115964117005 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a1889e7e90 con 0x55a1889f72e0 2024-04-27T03:18:08.993 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo1","permissions":"full-control"},{"id":"foo:foo2","permissions":""}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo2","secret_key":"ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy","active":true,"create_date":"2024-04-27T03:18:08.622385Z"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.981+0000 7fe46afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.981+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94152270310528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x55a1889f2fe0 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.981+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(69 data_loggenerations_metadata [watch unwatch cookie 94152270310528] v27'89 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.981+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94152270287552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x55a1889f72e0 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.985+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(70 notify.0 [watch unwatch cookie 94152270287552] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.985+0000 7fe5b6eab980 20 remove_watcher() i=0 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.985+0000 7fe5b6eab980 2 removed watcher, disabling cache 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.985+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:71 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94152270292528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x55a1889f2fe0 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.985+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(71 notify.1 [watch unwatch cookie 94152270292528] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x55a1889f2fe0 2024-04-27T03:18:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.985+0000 7fe5b6eab980 20 remove_watcher() i=1 2024-04-27T03:18:08.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.985+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94152270291328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x55a1889f72e0 2024-04-27T03:18:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(72 notify.2 [watch unwatch cookie 94152270291328] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b6eab980 20 remove_watcher() i=2 2024-04-27T03:18:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94152270298864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x7fe5ac035070 2024-04-27T03:18:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(73 notify.3 [watch unwatch cookie 94152270298864] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b6eab980 20 remove_watcher() i=3 2024-04-27T03:18:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94152270299920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x55a1889f72e0 2024-04-27T03:18:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b51c7640 1 -- 172.21.15.110:0/1266807099 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(74 notify.4 [watch unwatch cookie 94152270299920] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5ac032030 con 0x55a1889f72e0 2024-04-27T03:18:08.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b6eab980 20 remove_watcher() i=4 2024-04-27T03:18:08.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.989+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94152270296528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x55a1889fc440 2024-04-27T03:18:08.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(75 notify.5 [watch unwatch cookie 94152270296528] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 20 remove_watcher() i=5 2024-04-27T03:18:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94152270309472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x55a1889fc440 2024-04-27T03:18:09.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b49c6640 1 -- 172.21.15.110:0/1266807099 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(76 notify.6 [watch unwatch cookie 94152270309472] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a4052070 con 0x55a1889fc440 2024-04-27T03:18:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 20 remove_watcher() i=6 2024-04-27T03:18:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:77 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94152270304704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a188a2ba20 con 0x7fe5ac035070 2024-04-27T03:18:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(77 notify.7 [watch unwatch cookie 94152270304704] v27'74 uv72 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe5a80aaa40 con 0x7fe5ac035070 2024-04-27T03:18:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 20 remove_watcher() i=7 2024-04-27T03:18:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889f72e0 msgr2=0x55a1889f9710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889f72e0 0x55a1889f9710 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1889f2fe0 msgr2=0x55a1889f5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1889f2fe0 0x55a1889f5470 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe5ac035070 msgr2=0x7fe5ac04a960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe5ac035070 0x7fe5ac04a960 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a1889fc440 msgr2=0x55a1889fe830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a1889fc440 0x55a1889fe830 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe480034600 msgr2=0x7fe480036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe480034600 0x7fe480036ac0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4026000 tx=0x7fe5a403a000 comp rx=0 tx=0).stop 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 msgr2=0x55a1889e7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b41c5640 1 -- 172.21.15.110:0/1266807099 reap_dead start 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e7950 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac016ae0 tx=0x7fe5ac0498a0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 shutdown_connections 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/1266807099 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889e0530 0x55a1889e7950 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 >> 172.21.15.110:0/1266807099 conn(0x55a1889dc870 msgr2=0x55a1889d9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 shutdown_connections 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/1266807099 wait complete. 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe48c034650 msgr2=0x7fe48c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe48c034650 0x7fe48c036b10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55a1889c37c0 tx=0x7fe5a4033040 comp rx=0 tx=0).stop 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889d33b0 msgr2=0x55a1889d37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889d33b0 0x55a1889d37d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac00aab0 tx=0x7fe5ac00af70 comp rx=0 tx=0).stop 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 shutdown_connections 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe48c034650 0x7fe48c036b10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/3214235287 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1889d33b0 0x55a1889d37d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 >> 172.21.15.110:0/3214235287 conn(0x55a1889c81c0 msgr2=0x55a1889c8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 shutdown_connections 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/3214235287 wait complete. 2024-04-27T03:18:09.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889c0790 msgr2=0x55a1889c2b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889c0790 0x55a1889c2b80 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1888ca7a0 msgr2=0x55a1889bf330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1888ca7a0 0x55a1889bf330 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe49c034650 msgr2=0x7fe49c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe49c034650 0x7fe49c036b10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe5a80ae000 tx=0x7fe5a80a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 msgr2=0x55a1889b83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b83e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4001af0 tx=0x7fe5a4000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 shutdown_connections 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a1889c0790 0x55a1889c2b80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a1888ca7a0 0x55a1889bf330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe49c034650 0x7fe49c036b10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 --2- 172.21.15.110:0/2239740615 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a1888c62e0 0x55a1889b83e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 >> 172.21.15.110:0/2239740615 conn(0x55a1888c1bb0 msgr2=0x55a1889aba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 shutdown_connections 2024-04-27T03:18:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:08.993+0000 7fe5b6eab980 1 -- 172.21.15.110:0/2239740615 wait complete. 2024-04-27T03:18:09.012 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': 'full-control'}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'active': True, 'create_date': '2024-04-27T03:18:08.622385Z'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:09.012 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'rm', '--subuser', 'foo:foo1'] 2024-04-27T03:18:09.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'rm', '--subuser', 'foo:foo1'] 2024-04-27T03:18:09.012 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser rm --subuser foo:foo1 2024-04-27T03:18:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.073+0000 7f7a94631980 1 Processor -- start 2024-04-27T03:18:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.073+0000 7f7a94631980 1 -- start start 2024-04-27T03:18:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.073+0000 7f7a94631980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x555b484864b0 0x555b48486890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.073+0000 7f7a94631980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x555b483430d0 con 0x555b48486dd0 2024-04-27T03:18:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.073+0000 7f7a94631980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x555b48338430 con 0x555b484864b0 2024-04-27T03:18:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9214c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x555b484864b0 0x555b48486890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9214c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x555b484864b0 0x555b48486890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51216/0 (socket says 172.21.15.110:51216) 2024-04-27T03:18:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9214c640 1 -- 172.21.15.110:0/2965337072 learned_addr learned my addr 172.21.15.110:0/2965337072 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9194b640 1 --1- 172.21.15.110:0/2965337072 >> v1:172.21.15.77:6789/0 conn(0x555b48486dd0 0x555b48487790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34720/0 (socket says 172.21.15.110:34720) 2024-04-27T03:18:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9214c640 1 -- 172.21.15.110:0/2965337072 >> v1:172.21.15.77:6789/0 conn(0x555b48486dd0 legacy=0x555b48487790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9214c640 1 -- 172.21.15.110:0/2965337072 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48331510 con 0x555b484864b0 2024-04-27T03:18:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9214c640 1 --2- 172.21.15.110:0/2965337072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48486890 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400a5c0 tx=0x7f7a8401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddca2a671026ac69 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9114a640 1 -- 172.21.15.110:0/2965337072 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a840a6070 con 0x555b484864b0 2024-04-27T03:18:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9114a640 1 -- 172.21.15.110:0/2965337072 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a840af070 con 0x555b484864b0 2024-04-27T03:18:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a9114a640 1 -- 172.21.15.110:0/2965337072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a840aa5c0 con 0x555b484864b0 2024-04-27T03:18:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a94631980 1 -- 172.21.15.110:0/2965337072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 msgr2=0x555b48486890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a94631980 1 --2- 172.21.15.110:0/2965337072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48486890 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400a5c0 tx=0x7f7a8401b340 comp rx=0 tx=0).stop 2024-04-27T03:18:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a94631980 1 -- 172.21.15.110:0/2965337072 shutdown_connections 2024-04-27T03:18:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a94631980 1 --2- 172.21.15.110:0/2965337072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48486890 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a94631980 1 -- 172.21.15.110:0/2965337072 >> 172.21.15.110:0/2965337072 conn(0x555b483f9b90 msgr2=0x555b483f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.077+0000 7f7a94631980 1 -- 172.21.15.110:0/2965337072 shutdown_connections 2024-04-27T03:18:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.081+0000 7f7a94631980 1 -- 172.21.15.110:0/2965337072 wait complete. 2024-04-27T03:18:09.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 Processor -- start 2024-04-27T03:18:09.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- start start 2024-04-27T03:18:09.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48571d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b483872d0 con 0x555b484864b0 2024-04-27T03:18:09.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a9214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48571d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a9214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48571d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51220/0 (socket says 172.21.15.110:51220) 2024-04-27T03:18:09.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a9214c640 1 -- 172.21.15.110:0/2468281673 learned_addr learned my addr 172.21.15.110:0/2468281673 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a9214c640 1 -- 172.21.15.110:0/2468281673 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b4830dae0 con 0x555b484864b0 2024-04-27T03:18:09.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a9214c640 1 --2- 172.21.15.110:0/2468281673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48571d10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400a5c0 tx=0x7f7a84003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b319a255e70d9a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7980608640 1 -- 172.21.15.110:0/2468281673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a840aa9b0 con 0x555b484864b0 2024-04-27T03:18:09.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7980608640 1 -- 172.21.15.110:0/2468281673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a840aab50 con 0x555b484864b0 2024-04-27T03:18:09.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7980608640 1 -- 172.21.15.110:0/2468281673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a840aae50 con 0x555b484864b0 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- 172.21.15.110:0/2468281673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 msgr2=0x555b48571d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 --2- 172.21.15.110:0/2468281673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48571d10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7a8400a5c0 tx=0x7f7a84003030 comp rx=0 tx=0).stop 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- 172.21.15.110:0/2468281673 shutdown_connections 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 --2- 172.21.15.110:0/2468281673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48571d10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- 172.21.15.110:0/2468281673 >> 172.21.15.110:0/2468281673 conn(0x555b483f9b90 msgr2=0x555b4856d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- 172.21.15.110:0/2468281673 shutdown_connections 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- 172.21.15.110:0/2468281673 wait complete. 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 Processor -- start 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.093+0000 7f7a94631980 1 -- start start 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48578cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b48490500 con 0x555b484864b0 2024-04-27T03:18:09.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48578cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48578cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51224/0 (socket says 172.21.15.110:51224) 2024-04-27T03:18:09.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9214c640 1 -- 172.21.15.110:0/609992739 learned_addr learned my addr 172.21.15.110:0/609992739 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9214c640 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b48579230 con 0x555b484864b0 2024-04-27T03:18:09.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9214c640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48578cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7a8401b380 tx=0x7f7a840180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f797ee05640 1 -- 172.21.15.110:0/609992739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a840c6750 con 0x555b484864b0 2024-04-27T03:18:09.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b485794c0 con 0x555b484864b0 2024-04-27T03:18:09.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f797ee05640 1 -- 172.21.15.110:0/609992739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a840cc070 con 0x555b484864b0 2024-04-27T03:18:09.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f797ee05640 1 -- 172.21.15.110:0/609992739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a840c6a20 con 0x555b484864b0 2024-04-27T03:18:09.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b4857c0f0 con 0x555b484864b0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f797ee05640 1 -- 172.21.15.110:0/609992739 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7a840c6c90 con 0x555b484864b0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f797ee05640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7978034650 0x7f7978036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f797ee05640 1 -- 172.21.15.110:0/609992739 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7a840e5670 con 0x555b484864b0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9194b640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7978034650 0x7f7978036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48486dd0 0x555b4857fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48580160 con 0x555b48486dd0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9294d640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48486dd0 0x555b4857fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9194b640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7978034650 0x7f7978036b10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000f70 tx=0x7f7a88001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9294d640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48486dd0 0x555b4857fbd0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a9294d640 1 -- 172.21.15.110:0/609992739 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7a8c00a4d0 con 0x555b48486dd0 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48580f50 0x555b48583340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.097+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48583970 con 0x555b48580f50 2024-04-27T03:18:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9214c640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48580f50 0x555b48583340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9214c640 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48580f50 0x555b48583340 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9214c640 1 -- 172.21.15.110:0/609992739 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b48580f50 2024-04-27T03:18:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48585220 con 0x555b48486dd0 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9294d640 1 -- 172.21.15.110:0/609992739 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f7a8c00a4d0 con 0x555b48486dd0 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48584450 con 0x555b48580f50 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9214c640 1 -- 172.21.15.110:0/609992739 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b48580f50 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b48584870 con 0x555b48486dd0 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9294d640 1 -- 172.21.15.110:0/609992739 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f7a8c00a4d0 con 0x555b48486dd0 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a94631980 1 Processor -- start 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a94631980 1 -- start start 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a94631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48593b50 0x555b48593f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a94631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b4838b310 con 0x555b48593b50 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48593b50 0x555b48593f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48593b50 0x555b48593f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51232/0 (socket says 172.21.15.110:51232) 2024-04-27T03:18:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9294d640 1 -- 172.21.15.110:0/3725188761 learned_addr learned my addr 172.21.15.110:0/3725188761 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.101+0000 7f7a9294d640 1 -- 172.21.15.110:0/3725188761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b485944b0 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9294d640 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48593b50 0x555b48593f70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00dfc0 tx=0x7f7a8c0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7977fff640 1 -- 172.21.15.110:0/3725188761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a8c0157d0 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b48594f50 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7977fff640 1 -- 172.21.15.110:0/3725188761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8c015970 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7977fff640 1 -- 172.21.15.110:0/3725188761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a8c015c50 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b48595420 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7977fff640 1 -- 172.21.15.110:0/3725188761 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7a8c020440 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7977fff640 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f796c034650 0x7f796c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7977fff640 1 -- 172.21.15.110:0/3725188761 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7a8c00e040 con 0x555b48593b50 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9214c640 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f796c034650 0x7f796c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a94631980 1 Processor -- start 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a94631980 1 -- start start 2024-04-27T03:18:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9214c640 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f796c034650 0x7f796c036b10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x555b48580740 tx=0x7f7a840bdf90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a94631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a94631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b485723a0 con 0x555b485a0d50 2024-04-27T03:18:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51236/0 (socket says 172.21.15.110:51236) 2024-04-27T03:18:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9294d640 1 -- 172.21.15.110:0/1414422490 learned_addr learned my addr 172.21.15.110:0/1414422490 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9294d640 1 -- 172.21.15.110:0/1414422490 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b485a1740 con 0x555b485a0d50 2024-04-27T03:18:09.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7a9294d640 1 --2- 172.21.15.110:0/1414422490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a1170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c04aaa0 tx=0x7f7a8c020940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b6e3c4bd6ebf98 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7975ffb640 1 -- 172.21.15.110:0/1414422490 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a8c048070 con 0x555b485a0d50 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7975ffb640 1 -- 172.21.15.110:0/1414422490 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8c0113a0 con 0x555b485a0d50 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.105+0000 7f7975ffb640 1 -- 172.21.15.110:0/1414422490 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a8c0133b0 con 0x555b485a0d50 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1414422490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 msgr2=0x555b485a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 --2- 172.21.15.110:0/1414422490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a1170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c04aaa0 tx=0x7f7a8c020940 comp rx=0 tx=0).stop 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1414422490 shutdown_connections 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 --2- 172.21.15.110:0/1414422490 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a1170 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1414422490 >> 172.21.15.110:0/1414422490 conn(0x555b4859d090 msgr2=0x555b4859b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1414422490 shutdown_connections 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1414422490 wait complete. 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 Processor -- start 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- start start 2024-04-27T03:18:09.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x555b485a1ea0 con 0x555b485a0d50 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a9294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a9294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51244/0 (socket says 172.21.15.110:51244) 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 learned_addr learned my addr 172.21.15.110:0/1594977414 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b485a8770 con 0x555b485a0d50 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a9294d640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a8230 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0118e0 tx=0x7f7a8c04aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7963fff640 1 -- 172.21.15.110:0/1594977414 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a8c037020 con 0x555b485a0d50 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b485a8a00 con 0x555b485a0d50 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7963fff640 1 -- 172.21.15.110:0/1594977414 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8c01fcb0 con 0x555b485a0d50 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b485ab6e0 con 0x555b485a0d50 2024-04-27T03:18:09.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7963fff640 1 -- 172.21.15.110:0/1594977414 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a8c051060 con 0x555b485a0d50 2024-04-27T03:18:09.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7963fff640 1 -- 172.21.15.110:0/1594977414 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7a8c03a020 con 0x555b485a0d50 2024-04-27T03:18:09.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7963fff640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f795c034650 0x7f795c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7963fff640 1 -- 172.21.15.110:0/1594977414 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7a8c02f070 con 0x555b485a0d50 2024-04-27T03:18:09.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a9214c640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f795c034650 0x7f795c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b485b38a0 0x555b485b5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485b6490 con 0x555b485b38a0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.109+0000 7f7a9194b640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b485b38a0 0x555b485b5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9194b640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b485b38a0 0x555b485b5d40 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9214c640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f795c034650 0x7f795c036b10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7a840d0b00 tx=0x7f7a840ee000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7a8800f620 con 0x555b485b38a0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 realm 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485b2d70 con 0x555b485b38a0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7a8800f620 con 0x555b485b38a0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485b6f90 con 0x555b485b38a0 2024-04-27T03:18:09.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7a8800f620 con 0x555b485b38a0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b485b7d70 0x555b485ba160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485ba7e0 con 0x555b485b7d70 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9294d640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b485b7d70 0x555b485ba160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9294d640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b485b7d70 0x555b485ba160 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485bb030 con 0x555b485b38a0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f7a8800f620 con 0x555b485b38a0 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 20 searching for the correct realm 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b485bcda0 0x555b485bf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485bcda0 2024-04-27T03:18:09.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9214c640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b485bcda0 0x555b485bf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9214c640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b485bcda0 0x555b485bf190 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.113+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9294d640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7a8c01f3e0 0x7f7a8c034070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7a8c01f3e0 0x7f7a8c034070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7a8c01f3e0 0x7f7a8c034070 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8800f620 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8800f620 con 0x555b485b38a0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485bcda0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.117+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f7a8802b040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485bcda0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485bcda0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485bcda0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.121+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b38a0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8802b040 con 0x555b485b38a0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485bcda0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485bcda0 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x555b485bf810 con 0x555b485b7d70 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485bb980 con 0x555b485b38a0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.125+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485bc220 con 0x555b485b7d70 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485bb5c0 con 0x555b485b38a0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 zone default found 2024-04-27T03:18:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 4 Realm: () 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 zonegroup default 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485c0120 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485c1d10 con 0x555b485bcda0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485c2220 con 0x555b485bcda0 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 started sync module instance, tier type = 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c0be0 con 0x555b485b7d70 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.129+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93850544377808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c0840 con 0x555b485b7d70 2024-04-27T03:18:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c3150 con 0x555b485b38a0 2024-04-27T03:18:09.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93850544385184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c3570 con 0x555b485b38a0 2024-04-27T03:18:09.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c3fb0 con 0x555b485b7d70 2024-04-27T03:18:09.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93850544377808] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93850544385184] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93850544388736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c30c0 con 0x555b485b7d70 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.133+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c4850 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93850544391072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c30c0 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c5310 con 0x555b485b7d70 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93850544391072] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93850544388736] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93850544383616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c52d0 con 0x555b485b7d70 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c5700 con 0x555b485bcda0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93850544392128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c5fa0 con 0x555b485bcda0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c6960 con 0x555b485bcda0 2024-04-27T03:18:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93850544383616] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93850544392128] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.137+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93850544393696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c6960 con 0x555b485bcda0 2024-04-27T03:18:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c7260 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93850544393696] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'75 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93850544401744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c6960 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=0 2024-04-27T03:18:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=1 2024-04-27T03:18:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=2 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=3 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=4 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=5 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=6 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93850544401744] v27'76 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 20 add_watcher() i=7 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485ca6b0 con 0x555b485b38a0 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.141+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93850544400000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485c7bd0 con 0x555b485b38a0 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93850544400000] v27'90 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7948fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 5 note: GC not initialized 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f78c37fe640 20 reqs_thread_entry: start 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 20 Filter name: none 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f78c17fa640 20 reqs_thread_entry: start 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485edf10 con 0x555b485b38a0 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~582 out=582b,getxattrs out=75b] v0'0 uv9 ondisk = 0) ==== 231+0+705 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 20 rados_obj.operate() r=0 bl.length=582 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~556 in=556b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485f4550 con 0x555b485b38a0 2024-04-27T03:18:09.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [delete,create,call,call,writefull 0~556] v27'10 uv10 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 93850544586688 in=744b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x555b485efed0 con 0x555b485b7d70 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 93850544383616 notify 115964117006 ret 0) ==== 774+0+0 (crc 0 0 0) 0x555b485ab6e0 con 0x555b485b7d70 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7975ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117006 cookie 93850544383616 notifier 4401 bl.length()=732 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7975ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7975ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7975ffb640 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f7958003eb0 con 0x555b485b7d70 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.145+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 93850544586688 out=8b] v0'0 uv75 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.149+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.149+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 93850544586688 notify 115964117006 ret 0) ==== 42+0+48 (crc 0 0 0) 0x555b485a8a00 con 0x555b485b7d70 2024-04-27T03:18:09.157 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[{"id":"foo:foo2","permissions":""}],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[{"user":"foo:foo2","secret_key":"ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy","active":true,"create_date":"2024-04-27T03:18:08.622385Z"}],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.149+0000 7f7948fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.149+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93850544400000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x555b485b38a0 2024-04-27T03:18:09.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.149+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 93850544400000] v27'91 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.149+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93850544377808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x555b485b7d70 2024-04-27T03:18:09.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.153+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 93850544377808] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.153+0000 7f7a94631980 20 remove_watcher() i=0 2024-04-27T03:18:09.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.153+0000 7f7a94631980 2 removed watcher, disabling cache 2024-04-27T03:18:09.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.153+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93850544385184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x555b485b38a0 2024-04-27T03:18:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.153+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 93850544385184] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x555b485b38a0 2024-04-27T03:18:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.153+0000 7f7a94631980 20 remove_watcher() i=1 2024-04-27T03:18:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.153+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93850544388736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x555b485b7d70 2024-04-27T03:18:09.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.157+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 93850544388736] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.157+0000 7f7a94631980 20 remove_watcher() i=2 2024-04-27T03:18:09.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.157+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93850544391072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.157+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 93850544391072] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.157+0000 7f7a94631980 20 remove_watcher() i=3 2024-04-27T03:18:09.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.157+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93850544383616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x555b485b7d70 2024-04-27T03:18:09.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a9294d640 1 -- 172.21.15.110:0/1594977414 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 93850544383616] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8c026020 con 0x555b485b7d70 2024-04-27T03:18:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 20 remove_watcher() i=4 2024-04-27T03:18:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93850544392128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x555b485bcda0 2024-04-27T03:18:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 93850544392128] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 20 remove_watcher() i=5 2024-04-27T03:18:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93850544393696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x555b485bcda0 2024-04-27T03:18:09.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a9214c640 1 -- 172.21.15.110:0/1594977414 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 93850544393696] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a840c5d00 con 0x555b485bcda0 2024-04-27T03:18:09.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 20 remove_watcher() i=6 2024-04-27T03:18:09.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93850544401744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x555b485ebc40 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 93850544401744] v27'77 uv75 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7a8803e040 con 0x7f7a8c01f3e0 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 20 remove_watcher() i=7 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b485b7d70 msgr2=0x555b485ba160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b485b7d70 0x555b485ba160 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b485b38a0 msgr2=0x555b485b5d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b485b38a0 0x555b485b5d40 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7a8c01f3e0 msgr2=0x7f7a8c034070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7a8c01f3e0 0x7f7a8c034070 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b485bcda0 msgr2=0x555b485bf190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x555b485bcda0 0x555b485bf190 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f795c034650 msgr2=0x7f795c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f795c034650 0x7f795c036b10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7a840d0b00 tx=0x7f7a840ee000 comp rx=0 tx=0).stop 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a9194b640 1 -- 172.21.15.110:0/1594977414 reap_dead start 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 msgr2=0x555b485a8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a8230 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0118e0 tx=0x7f7a8c04aa30 comp rx=0 tx=0).stop 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 shutdown_connections 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 --2- 172.21.15.110:0/1594977414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b485a0d50 0x555b485a8230 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 >> 172.21.15.110:0/1594977414 conn(0x555b4859d090 msgr2=0x555b4859a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 shutdown_connections 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.161+0000 7f7a94631980 1 -- 172.21.15.110:0/1594977414 wait complete. 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f796c034650 msgr2=0x7f796c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f796c034650 0x7f796c036b10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x555b48580740 tx=0x7f7a840bdf90 comp rx=0 tx=0).stop 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48593b50 msgr2=0x555b48593f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48593b50 0x555b48593f70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00dfc0 tx=0x7f7a8c0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 shutdown_connections 2024-04-27T03:18:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f796c034650 0x7f796c036b10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/3725188761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b48593b50 0x555b48593f70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 >> 172.21.15.110:0/3725188761 conn(0x555b48588960 msgr2=0x555b48589470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 shutdown_connections 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/3725188761 wait complete. 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48580f50 msgr2=0x555b48583340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48580f50 0x555b48583340 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48486dd0 msgr2=0x555b4857fbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48486dd0 0x555b4857fbd0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7978034650 msgr2=0x7f7978036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7978034650 0x7f7978036b10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000f70 tx=0x7f7a88001380 comp rx=0 tx=0).stop 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 msgr2=0x555b48578cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48578cf0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7a8401b380 tx=0x7f7a840180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 shutdown_connections 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x555b48580f50 0x555b48583340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x555b48486dd0 0x555b4857fbd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7978034650 0x7f7978036b10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 --2- 172.21.15.110:0/609992739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x555b484864b0 0x555b48578cf0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 >> 172.21.15.110:0/609992739 conn(0x555b483f9b90 msgr2=0x555b4856c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 shutdown_connections 2024-04-27T03:18:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.165+0000 7f7a94631980 1 -- 172.21.15.110:0/609992739 wait complete. 2024-04-27T03:18:09.181 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'active': True, 'create_date': '2024-04-27T03:18:08.622385Z'}], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:09.181 INFO:tasks.util.rgw:rgwadmin: client.0 : ['subuser', 'rm', '--subuser', 'foo:foo2', '--key-type', 'swift', '--purge-keys'] 2024-04-27T03:18:09.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'subuser', 'rm', '--subuser', 'foo:foo2', '--key-type', 'swift', '--purge-keys'] 2024-04-27T03:18:09.181 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph subuser rm --subuser foo:foo2 --key-type swift --purge-keys 2024-04-27T03:18:09.252 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:09.252 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:09.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0ad50980 1 Processor -- start 2024-04-27T03:18:09.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0ad50980 1 -- start start 2024-04-27T03:18:09.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0ad50980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564967acd7f0 0x564967b5bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0ad50980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x564967a180d0 con 0x564967b5c420 2024-04-27T03:18:09.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0ad50980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x564967a0d430 con 0x564967acd7f0 2024-04-27T03:18:09.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0886b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564967acd7f0 0x564967b5bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0886b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564967acd7f0 0x564967b5bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51256/0 (socket says 172.21.15.110:51256) 2024-04-27T03:18:09.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0806a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x564967b5c420 0x564967b5e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34726/0 (socket says 172.21.15.110:34726) 2024-04-27T03:18:09.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0886b640 1 -- 172.21.15.110:0/4106380866 learned_addr learned my addr 172.21.15.110:0/4106380866 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0886b640 1 -- 172.21.15.110:0/4106380866 >> v1:172.21.15.77:6789/0 conn(0x564967b5c420 legacy=0x564967b5e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.245+0000 7f5b0886b640 1 -- 172.21.15.110:0/4106380866 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x564967a06510 con 0x564967acd7f0 2024-04-27T03:18:09.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0886b640 1 --2- 172.21.15.110:0/4106380866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b5bee0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5af800a4e0 tx=0x7f5af801b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969b8c26a1b09ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b07869640 1 -- 172.21.15.110:0/4106380866 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af80aa540 con 0x564967acd7f0 2024-04-27T03:18:09.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b07869640 1 -- 172.21.15.110:0/4106380866 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af80aa6e0 con 0x564967acd7f0 2024-04-27T03:18:09.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b07869640 1 -- 172.21.15.110:0/4106380866 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af80aaa40 con 0x564967acd7f0 2024-04-27T03:18:09.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0ad50980 1 -- 172.21.15.110:0/4106380866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 msgr2=0x564967b5bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/4106380866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b5bee0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5af800a4e0 tx=0x7f5af801b090 comp rx=0 tx=0).stop 2024-04-27T03:18:09.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0ad50980 1 -- 172.21.15.110:0/4106380866 shutdown_connections 2024-04-27T03:18:09.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/4106380866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b5bee0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0ad50980 1 -- 172.21.15.110:0/4106380866 >> 172.21.15.110:0/4106380866 conn(0x564967b539c0 msgr2=0x564967b55de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0ad50980 1 -- 172.21.15.110:0/4106380866 shutdown_connections 2024-04-27T03:18:09.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.249+0000 7f5b0ad50980 1 -- 172.21.15.110:0/4106380866 wait complete. 2024-04-27T03:18:09.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.261+0000 7f5b0ad50980 1 Processor -- start 2024-04-27T03:18:09.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.261+0000 7f5b0ad50980 1 -- start start 2024-04-27T03:18:09.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.261+0000 7f5b0ad50980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b59640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.261+0000 7f5b0ad50980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564967a5c2d0 con 0x564967acd7f0 2024-04-27T03:18:09.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.261+0000 7f5b0886b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b59640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.261+0000 7f5b0886b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b59640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51266/0 (socket says 172.21.15.110:51266) 2024-04-27T03:18:09.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.261+0000 7f5b0886b640 1 -- 172.21.15.110:0/2142739901 learned_addr learned my addr 172.21.15.110:0/2142739901 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0886b640 1 -- 172.21.15.110:0/2142739901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649679e2ae0 con 0x564967acd7f0 2024-04-27T03:18:09.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0886b640 1 --2- 172.21.15.110:0/2142739901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b59640 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5af8018cc0 tx=0x7f5af80033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a128bdee6f1e4a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f59f6e0d640 1 -- 172.21.15.110:0/2142739901 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af80b9070 con 0x564967acd7f0 2024-04-27T03:18:09.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f59f6e0d640 1 -- 172.21.15.110:0/2142739901 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af80aaa80 con 0x564967acd7f0 2024-04-27T03:18:09.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f59f6e0d640 1 -- 172.21.15.110:0/2142739901 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af80aad60 con 0x564967acd7f0 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2142739901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 msgr2=0x564967b59640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/2142739901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b59640 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5af8018cc0 tx=0x7f5af80033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2142739901 shutdown_connections 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/2142739901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967b59640 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2142739901 >> 172.21.15.110:0/2142739901 conn(0x564967b539c0 msgr2=0x564967c3ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2142739901 shutdown_connections 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2142739901 wait complete. 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 Processor -- start 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- start start 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967c4d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564967b60a40 con 0x564967acd7f0 2024-04-27T03:18:09.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0886b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967c4d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0886b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967c4d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51278/0 (socket says 172.21.15.110:51278) 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0886b640 1 -- 172.21.15.110:0/3788264149 learned_addr learned my addr 172.21.15.110:0/3788264149 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0886b640 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564967c4db30 con 0x564967acd7f0 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0886b640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967c4d5f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5af801aec0 tx=0x7f5af8017e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f59f560a640 1 -- 172.21.15.110:0/3788264149 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af80b9070 con 0x564967acd7f0 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564967c4ddc0 con 0x564967acd7f0 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f59f560a640 1 -- 172.21.15.110:0/3788264149 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af80c7410 con 0x564967acd7f0 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f59f560a640 1 -- 172.21.15.110:0/3788264149 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5af80c76f0 con 0x564967acd7f0 2024-04-27T03:18:09.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.265+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564967c509f0 con 0x564967acd7f0 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f59f560a640 1 -- 172.21.15.110:0/3788264149 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5af80b3040 con 0x564967acd7f0 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f59f560a640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59f0034650 0x7f59f0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f59f560a640 1 -- 172.21.15.110:0/3788264149 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5af80c4070 con 0x564967acd7f0 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0806a640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59f0034650 0x7f59f0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967b5c420 0x564967c544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c54a50 con 0x564967b5c420 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0906c640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967b5c420 0x564967c544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0806a640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59f0034650 0x7f59f0036b10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5afc001e50 tx=0x7f5afc0013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0906c640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967b5c420 0x564967c544d0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0906c640 1 -- 172.21.15.110:0/3788264149 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5b000083f0 con 0x564967b5c420 2024-04-27T03:18:09.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c557e0 0x564967c57bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c58230 con 0x564967c557e0 2024-04-27T03:18:09.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0886b640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c557e0 0x564967c57bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0886b640 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c557e0 0x564967c57bd0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0886b640 1 -- 172.21.15.110:0/3788264149 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c557e0 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.269+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c59ae0 con 0x564967b5c420 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0906c640 1 -- 172.21.15.110:0/3788264149 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f5b000083f0 con 0x564967b5c420 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c58d10 con 0x564967c557e0 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0886b640 1 -- 172.21.15.110:0/3788264149 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c557e0 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c59100 con 0x564967b5c420 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0906c640 1 -- 172.21.15.110:0/3788264149 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f5b000083f0 con 0x564967b5c420 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 Processor -- start 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 -- start start 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c68410 0x564967c68830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564967a60310 con 0x564967c68410 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0906c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c68410 0x564967c68830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0906c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c68410 0x564967c68830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51286/0 (socket says 172.21.15.110:51286) 2024-04-27T03:18:09.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0906c640 1 -- 172.21.15.110:0/3860691335 learned_addr learned my addr 172.21.15.110:0/3860691335 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0906c640 1 -- 172.21.15.110:0/3860691335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564967c68d70 con 0x564967c68410 2024-04-27T03:18:09.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0906c640 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c68410 0x564967c68830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000aae0 tx=0x7f5b00008ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f59ee7fc640 1 -- 172.21.15.110:0/3860691335 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b00007ce0 con 0x564967c68410 2024-04-27T03:18:09.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f59ee7fc640 1 -- 172.21.15.110:0/3860691335 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00018440 con 0x564967c68410 2024-04-27T03:18:09.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564967c69810 con 0x564967c68410 2024-04-27T03:18:09.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f59ee7fc640 1 -- 172.21.15.110:0/3860691335 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b00018720 con 0x564967c68410 2024-04-27T03:18:09.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.273+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564967c69ce0 con 0x564967c68410 2024-04-27T03:18:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f59ee7fc640 1 -- 172.21.15.110:0/3860691335 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5b00013070 con 0x564967c68410 2024-04-27T03:18:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f59ee7fc640 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59e00346a0 0x7f59e0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f59ee7fc640 1 -- 172.21.15.110:0/3860691335 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5b0004b710 con 0x564967c68410 2024-04-27T03:18:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0886b640 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59e00346a0 0x7f59e0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 Processor -- start 2024-04-27T03:18:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- start start 2024-04-27T03:18:09.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0886b640 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59e00346a0 0x7f59e0036b60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x564967c58830 tx=0x7f5af80aa670 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c75990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564967b59cd0 con 0x564967c75570 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c75990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c75990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51298/0 (socket says 172.21.15.110:51298) 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 -- 172.21.15.110:0/2219221174 learned_addr learned my addr 172.21.15.110:0/2219221174 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 -- 172.21.15.110:0/2219221174 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564967c75f60 con 0x564967c75570 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 --2- 172.21.15.110:0/2219221174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c75990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5b00049b20 tx=0x7f5b000163f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0399773c9393ea server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f59dffff640 1 -- 172.21.15.110:0/2219221174 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b00040070 con 0x564967c75570 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f59dffff640 1 -- 172.21.15.110:0/2219221174 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00013ba0 con 0x564967c75570 2024-04-27T03:18:09.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f59dffff640 1 -- 172.21.15.110:0/2219221174 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b00013ea0 con 0x564967c75570 2024-04-27T03:18:09.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2219221174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 msgr2=0x564967c75990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/2219221174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c75990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5b00049b20 tx=0x7f5b000163f0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2219221174 shutdown_connections 2024-04-27T03:18:09.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/2219221174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c75990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2219221174 >> 172.21.15.110:0/2219221174 conn(0x564967c71900 msgr2=0x564967c6f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2219221174 shutdown_connections 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- 172.21.15.110:0/2219221174 wait complete. 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 Processor -- start 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- start start 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0ad50980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564967c766d0 con 0x564967c75570 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c7caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c7caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51302/0 (socket says 172.21.15.110:51302) 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 learned_addr learned my addr 172.21.15.110:0/1339676288 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.277+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564967c7cfe0 con 0x564967c75570 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0906c640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c7caa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5b00016dc0 tx=0x7f5b00049ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f59de7fc640 1 -- 172.21.15.110:0/1339676288 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b00040040 con 0x564967c75570 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564967c7d270 con 0x564967c75570 2024-04-27T03:18:09.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f59de7fc640 1 -- 172.21.15.110:0/1339676288 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0004b430 con 0x564967c75570 2024-04-27T03:18:09.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f59de7fc640 1 -- 172.21.15.110:0/1339676288 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5b000189f0 con 0x564967c75570 2024-04-27T03:18:09.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564967c7ff50 con 0x564967c75570 2024-04-27T03:18:09.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f59de7fc640 1 -- 172.21.15.110:0/1339676288 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5b00033020 con 0x564967c75570 2024-04-27T03:18:09.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f59de7fc640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59d4034650 0x7f59d4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f59de7fc640 1 -- 172.21.15.110:0/1339676288 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5b00031020 con 0x564967c75570 2024-04-27T03:18:09.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0886b640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59d4034650 0x7f59d4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967c88130 0x564967c8a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c8ad10 con 0x564967c88130 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0806a640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967c88130 0x564967c8a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0886b640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59d4034650 0x7f59d4036b10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5af80b5d60 tx=0x7f5af80e7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0806a640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967c88130 0x564967c8a5c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5afc00f6a0 con 0x564967c88130 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 realm 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c87640 con 0x564967c88130 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5afc00f6a0 con 0x564967c88130 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c8b7e0 con 0x564967c88130 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5afc00f6a0 con 0x564967c88130 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c8c5c0 0x564967c8e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.281+0000 7f5b0906c640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c8c5c0 0x564967c8e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0906c640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c8c5c0 0x564967c8e9b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0ad50980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f880 con 0x564967c88130 2024-04-27T03:18:09.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f5afc00f6a0 con 0x564967c88130 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0ad50980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0ad50980 20 searching for the correct realm 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564967c915f0 0x564967c939e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c94060 con 0x564967c915f0 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0886b640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564967c915f0 0x564967c939e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0886b640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564967c915f0 0x564967c939e0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c94060 con 0x564967c8c5c0 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0906c640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5b00022c20 0x7f5b00039ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x7f5b00022c20 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0806a640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5b00022c20 0x7f5b00039ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0806a640 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5b00022c20 0x7f5b00039ad0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc00f6a0 con 0x7f5b00022c20 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc00f6a0 con 0x564967c88130 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.285+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c915f0 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x7f5b00022c20 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x7f5b00022c20 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x7f5b00022c20 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f5afc02b040 con 0x7f5b00022c20 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c915f0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.289+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c915f0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c915f0 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x7f5b00022c20 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x7f5b00022c20 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c88130 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc02b040 con 0x564967c88130 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x7f5b00022c20 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c915f0 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x7f5b00022c20 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.293+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c915f0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x564967c8f030 con 0x564967c8c5c0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c900d0 con 0x564967c88130 2024-04-27T03:18:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c904c0 con 0x564967c8c5c0 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c904c0 con 0x564967c88130 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 zone default found 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 4 Realm: () 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 zonegroup default 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c944c0 con 0x7f5b00022c20 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c95ea0 con 0x564967c915f0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.297+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c963a0 con 0x564967c915f0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 20 started sync module instance, tier type = 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c95340 con 0x564967c8c5c0 2024-04-27T03:18:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94873273849952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c95340 con 0x564967c8c5c0 2024-04-27T03:18:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c96e20 con 0x564967c88130 2024-04-27T03:18:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94873273854320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c97240 con 0x564967c88130 2024-04-27T03:18:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c97d30 con 0x564967c8c5c0 2024-04-27T03:18:09.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.301+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94873273849952] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94873273854320] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94873273858048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c96d90 con 0x564967c8c5c0 2024-04-27T03:18:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c985d0 con 0x7f5b00022c20 2024-04-27T03:18:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94873273860384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c96d90 con 0x7f5b00022c20 2024-04-27T03:18:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c990a0 con 0x564967c8c5c0 2024-04-27T03:18:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94873273860384] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.305+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94873273858048] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94873273847440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c96d90 con 0x564967c8c5c0 2024-04-27T03:18:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c994c0 con 0x564967c915f0 2024-04-27T03:18:09.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94873273861440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c99d10 con 0x564967c915f0 2024-04-27T03:18:09.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c9a6d0 con 0x564967c915f0 2024-04-27T03:18:09.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94873273861440] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94873273869344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c99d10 con 0x564967c915f0 2024-04-27T03:18:09.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c9b160 con 0x7f5b00022c20 2024-04-27T03:18:09.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94873273847440] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'78 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94873273871408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c9b120 con 0x7f5b00022c20 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 20 add_watcher() i=0 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 20 add_watcher() i=1 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 20 add_watcher() i=2 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 20 add_watcher() i=3 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 20 add_watcher() i=4 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 20 add_watcher() i=5 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94873273869344] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.309+0000 7f5b0ad50980 20 add_watcher() i=6 2024-04-27T03:18:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94873273871408] v27'79 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 20 add_watcher() i=7 2024-04-27T03:18:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967c99860 con 0x564967c88130 2024-04-27T03:18:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94873273866336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967c9b600 con 0x564967c88130 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94873273866336] v27'92 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f59bf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 5 note: GC not initialized 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f59be7e4640 20 reqs_thread_entry: start 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 20 Filter name: none 2024-04-27T03:18:09.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f59b77fe640 20 reqs_thread_entry: start 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967cc19e0 con 0x564967c88130 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~556 out=556b,getxattrs out=75b] v0'0 uv10 ondisk = 0) ==== 231+0+679 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 20 rados_obj.operate() r=0 bl.length=556 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:09.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.313+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cc8080 con 0x564967c88130 2024-04-27T03:18:09.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [delete,create,call,call,writefull 0~391] v27'11 uv11 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0ad50980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:09.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0ad50980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:09.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0ad50980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:09.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94873274055408 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967cc37f0 con 0x564967c8c5c0 2024-04-27T03:18:09.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94873273847440 notify 115964117007 ret 0) ==== 609+0+0 (crc 0 0 0) 0x564967c7ff50 con 0x564967c8c5c0 2024-04-27T03:18:09.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f59dffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117007 cookie 94873273847440 notifier 4408 bl.length()=567 2024-04-27T03:18:09.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94873274055408 out=8b] v0'0 uv78 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f59dffff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:09.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f59dffff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:09.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f59dffff640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f59cc003eb0 con 0x564967c8c5c0 2024-04-27T03:18:09.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== watch-notify(notify_complete (2) cookie 94873274055408 notify 115964117007 ret 0) ==== 42+0+48 (crc 0 0 0) 0x564967c7d270 con 0x564967c8c5c0 2024-04-27T03:18:09.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.2 8:45671577:users.swift::foo%3afoo2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cc7af0 con 0x7f5b00022c20 2024-04-27T03:18:09.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.317+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(66 foo:foo2 [delete] v27'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0ad50980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.swift:foo:foo2, ofs0, ns] 2024-04-27T03:18:09.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94873274055408 in=171b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x564967cc3340 con 0x564967c8c5c0 2024-04-27T03:18:09.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94873273849952 notify 115964117008 ret 0) ==== 201+0+0 (crc 0 0 0) 0x564967c7d270 con 0x564967c8c5c0 2024-04-27T03:18:09.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f59dffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117008 cookie 94873273849952 notifier 4408 bl.length()=159 2024-04-27T03:18:09.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(67 notify.0 [notify cookie 94873274055408 out=8b] v0'0 uv78 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f59dffff640 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f59cc003eb0 con 0x564967c8c5c0 2024-04-27T03:18:09.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== watch-notify(notify_complete (2) cookie 94873274055408 notify 115964117008 ret 0) ==== 42+0+48 (crc 0 0 0) 0x564967c7cfe0 con 0x564967c8c5c0 2024-04-27T03:18:09.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(68 notify.0 [notify-ack] v0'0 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.330 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:09.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f59bf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.321+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94873273866336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x564967c88130 2024-04-27T03:18:09.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(69 data_loggenerations_metadata [watch unwatch cookie 94873273866336] v27'93 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94873273849952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x564967c8c5c0 2024-04-27T03:18:09.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(70 notify.0 [watch unwatch cookie 94873273849952] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0ad50980 20 remove_watcher() i=0 2024-04-27T03:18:09.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0ad50980 2 removed watcher, disabling cache 2024-04-27T03:18:09.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:71 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94873273854320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x564967c88130 2024-04-27T03:18:09.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(71 notify.1 [watch unwatch cookie 94873273854320] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x564967c88130 2024-04-27T03:18:09.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0ad50980 20 remove_watcher() i=1 2024-04-27T03:18:09.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.325+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94873273858048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x564967c8c5c0 2024-04-27T03:18:09.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.329+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(72 notify.2 [watch unwatch cookie 94873273858048] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.329+0000 7f5b0ad50980 20 remove_watcher() i=2 2024-04-27T03:18:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.329+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94873273860384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x7f5b00022c20 2024-04-27T03:18:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.329+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(73 notify.3 [watch unwatch cookie 94873273860384] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.329+0000 7f5b0ad50980 20 remove_watcher() i=3 2024-04-27T03:18:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.329+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94873273847440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x564967c8c5c0 2024-04-27T03:18:09.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0906c640 1 -- 172.21.15.110:0/1339676288 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(74 notify.4 [watch unwatch cookie 94873273847440] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5b00043040 con 0x564967c8c5c0 2024-04-27T03:18:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0ad50980 20 remove_watcher() i=4 2024-04-27T03:18:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94873273861440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x564967c915f0 2024-04-27T03:18:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(75 notify.5 [watch unwatch cookie 94873273861440] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0ad50980 20 remove_watcher() i=5 2024-04-27T03:18:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94873273869344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x564967c915f0 2024-04-27T03:18:09.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0886b640 1 -- 172.21.15.110:0/1339676288 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(76 notify.6 [watch unwatch cookie 94873273869344] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5af80cd060 con 0x564967c915f0 2024-04-27T03:18:09.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0ad50980 20 remove_watcher() i=6 2024-04-27T03:18:09.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.333+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:77 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94873273871408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x564967cbf710 con 0x7f5b00022c20 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(77 notify.7 [watch unwatch cookie 94873273871408] v27'80 uv78 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5afc03e040 con 0x7f5b00022c20 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 20 remove_watcher() i=7 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c8c5c0 msgr2=0x564967c8e9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c8c5c0 0x564967c8e9b0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967c88130 msgr2=0x564967c8a5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967c88130 0x564967c8a5c0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5b00022c20 msgr2=0x7f5b00039ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5b00022c20 0x7f5b00039ad0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564967c915f0 msgr2=0x564967c939e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564967c915f0 0x564967c939e0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59d4034650 msgr2=0x7f59d4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59d4034650 0x7f59d4036b10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5af80b5d60 tx=0x7f5af80e7000 comp rx=0 tx=0).stop 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 msgr2=0x564967c7caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0806a640 1 -- 172.21.15.110:0/1339676288 reap_dead start 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c7caa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5b00016dc0 tx=0x7f5b00049ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 shutdown_connections 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/1339676288 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c75570 0x564967c7caa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 >> 172.21.15.110:0/1339676288 conn(0x564967c71900 msgr2=0x564967c76df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 shutdown_connections 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/1339676288 wait complete. 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59e00346a0 msgr2=0x7f59e0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59e00346a0 0x7f59e0036b60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x564967c58830 tx=0x7f5af80aa670 comp rx=0 tx=0).stop 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c68410 msgr2=0x564967c68830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c68410 0x564967c68830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000aae0 tx=0x7f5b00008ed0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 shutdown_connections 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59e00346a0 0x7f59e0036b60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3860691335 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967c68410 0x564967c68830 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 >> 172.21.15.110:0/3860691335 conn(0x564967c5d220 msgr2=0x564967c5dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 shutdown_connections 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3860691335 wait complete. 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c557e0 msgr2=0x564967c57bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c557e0 0x564967c57bd0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967b5c420 msgr2=0x564967c544d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967b5c420 0x564967c544d0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59f0034650 msgr2=0x7f59f0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59f0034650 0x7f59f0036b10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5afc001e50 tx=0x7f5afc0013b0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 msgr2=0x564967c4d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967c4d5f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5af801aec0 tx=0x7f5af8017e30 comp rx=0 tx=0).stop 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 shutdown_connections 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564967c557e0 0x564967c57bd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564967b5c420 0x564967c544d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f59f0034650 0x7f59f0036b10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 --2- 172.21.15.110:0/3788264149 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564967acd7f0 0x564967c4d5f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 >> 172.21.15.110:0/3788264149 conn(0x564967b539c0 msgr2=0x564967b58130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 shutdown_connections 2024-04-27T03:18:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.337+0000 7f5b0ad50980 1 -- 172.21.15.110:0/3788264149 wait complete. 2024-04-27T03:18:09.354 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:09.354 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--uid', 'foo'] 2024-04-27T03:18:09.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--uid', 'foo'] 2024-04-27T03:18:09.355 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --uid foo 2024-04-27T03:18:09.430 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:09.431 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:09.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f302b980 1 Processor -- start 2024-04-27T03:18:09.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f302b980 1 -- start start 2024-04-27T03:18:09.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f302b980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x563b28d0d150 0x563b28d0d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f302b980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x563b28bcd0d0 con 0x563b28d0da70 2024-04-27T03:18:09.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f302b980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x563b28bc2430 con 0x563b28d0d150 2024-04-27T03:18:09.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f0b46640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x563b28d0d150 0x563b28d0d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f0b46640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x563b28d0d150 0x563b28d0d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51308/0 (socket says 172.21.15.110:51308) 2024-04-27T03:18:09.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2271183349 learned_addr learned my addr 172.21.15.110:0/2271183349 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2271183349 >> v1:172.21.15.77:6789/0 conn(0x563b28d0da70 legacy=0x563b28d0e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2271183349 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b28bbb510 con 0x563b28d0d150 2024-04-27T03:18:09.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.425+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/2271183349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28d0d530 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000a5c0 tx=0x7fd2e001b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36e906171da65ed server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2efb44640 1 -- 172.21.15.110:0/2271183349 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00a6070 con 0x563b28d0d150 2024-04-27T03:18:09.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2efb44640 1 -- 172.21.15.110:0/2271183349 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e00af070 con 0x563b28d0d150 2024-04-27T03:18:09.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2efb44640 1 -- 172.21.15.110:0/2271183349 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00aa5c0 con 0x563b28d0d150 2024-04-27T03:18:09.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2f302b980 1 -- 172.21.15.110:0/2271183349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 msgr2=0x563b28d0d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2271183349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28d0d530 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000a5c0 tx=0x7fd2e001b340 comp rx=0 tx=0).stop 2024-04-27T03:18:09.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2f302b980 1 -- 172.21.15.110:0/2271183349 shutdown_connections 2024-04-27T03:18:09.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2271183349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28d0d530 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2f302b980 1 -- 172.21.15.110:0/2271183349 >> 172.21.15.110:0/2271183349 conn(0x563b28d08a20 msgr2=0x563b28d0ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2f302b980 1 -- 172.21.15.110:0/2271183349 shutdown_connections 2024-04-27T03:18:09.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.429+0000 7fd2f302b980 1 -- 172.21.15.110:0/2271183349 wait complete. 2024-04-27T03:18:09.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 Processor -- start 2024-04-27T03:18:09.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- start start 2024-04-27T03:18:09.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28df78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563b28c112d0 con 0x563b28d0d150 2024-04-27T03:18:09.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f0b46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28df78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f0b46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28df78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51314/0 (socket says 172.21.15.110:51314) 2024-04-27T03:18:09.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f0b46640 1 -- 172.21.15.110:0/1447251417 learned_addr learned my addr 172.21.15.110:0/1447251417 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f0b46640 1 -- 172.21.15.110:0/1447251417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b28b97ae0 con 0x563b28d0d150 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/1447251417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28df78c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00199f0 tx=0x7fd2e0003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90209bc859510f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd1dee0d640 1 -- 172.21.15.110:0/1447251417 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00aaa00 con 0x563b28d0d150 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd1dee0d640 1 -- 172.21.15.110:0/1447251417 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e00aaba0 con 0x563b28d0d150 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd1dee0d640 1 -- 172.21.15.110:0/1447251417 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00bd3f0 con 0x563b28d0d150 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- 172.21.15.110:0/1447251417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 msgr2=0x563b28df78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 --2- 172.21.15.110:0/1447251417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28df78c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00199f0 tx=0x7fd2e0003030 comp rx=0 tx=0).stop 2024-04-27T03:18:09.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- 172.21.15.110:0/1447251417 shutdown_connections 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 --2- 172.21.15.110:0/1447251417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28df78c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- 172.21.15.110:0/1447251417 >> 172.21.15.110:0/1447251417 conn(0x563b28d08a20 msgr2=0x563b28df3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- 172.21.15.110:0/1447251417 shutdown_connections 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- 172.21.15.110:0/1447251417 wait complete. 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 Processor -- start 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.441+0000 7fd2f302b980 1 -- start start 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28dfe840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563b28c81eb0 con 0x563b28d0d150 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f0b46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28dfe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f0b46640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28dfe840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51318/0 (socket says 172.21.15.110:51318) 2024-04-27T03:18:09.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f0b46640 1 -- 172.21.15.110:0/3567032358 learned_addr learned my addr 172.21.15.110:0/3567032358 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f0b46640 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b28dfed80 con 0x563b28d0d150 2024-04-27T03:18:09.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28dfe840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd2e001b170 tx=0x7fd2e00180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd1dd60a640 1 -- 172.21.15.110:0/3567032358 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00bd7c0 con 0x563b28d0d150 2024-04-27T03:18:09.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563b28dff010 con 0x563b28d0d150 2024-04-27T03:18:09.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd1dd60a640 1 -- 172.21.15.110:0/3567032358 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e00bd960 con 0x563b28d0d150 2024-04-27T03:18:09.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd1dd60a640 1 -- 172.21.15.110:0/3567032358 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00bdc40 con 0x563b28d0d150 2024-04-27T03:18:09.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b28e01bf0 con 0x563b28d0d150 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd1dd60a640 1 -- 172.21.15.110:0/3567032358 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd2e00ce020 con 0x563b28d0d150 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd1dd60a640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d8034650 0x7fd1d8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd1dd60a640 1 -- 172.21.15.110:0/3567032358 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fd2e00cc020 con 0x563b28d0d150 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f0345640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d8034650 0x7fd1d8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28d0da70 0x563b28e056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e05d10 con 0x563b28d0da70 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f1347640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28d0da70 0x563b28e056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f0345640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d8034650 0x7fd1d8036b10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4000f70 tx=0x7fd2e4001420 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f1347640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28d0da70 0x563b28e056f0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f1347640 1 -- 172.21.15.110:0/3567032358 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e800a4d0 con 0x563b28d0da70 2024-04-27T03:18:09.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e06ae0 0x563b28e08ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.445+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e09530 con 0x563b28e06ae0 2024-04-27T03:18:09.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e06ae0 0x563b28e08ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e06ae0 0x563b28e08ed0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f0b46640 1 -- 172.21.15.110:0/3567032358 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e06ae0 2024-04-27T03:18:09.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e0ade0 con 0x563b28d0da70 2024-04-27T03:18:09.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f1347640 1 -- 172.21.15.110:0/3567032358 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fd2e800a4d0 con 0x563b28d0da70 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e0a040 con 0x563b28e06ae0 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f0b46640 1 -- 172.21.15.110:0/3567032358 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e06ae0 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e0a430 con 0x563b28d0da70 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f1347640 1 -- 172.21.15.110:0/3567032358 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fd2e800a4d0 con 0x563b28d0da70 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f302b980 1 Processor -- start 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f302b980 1 -- start start 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f302b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e19750 0x563b28e19b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f302b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563b28c15310 con 0x563b28e19750 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f1347640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e19750 0x563b28e19b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f1347640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e19750 0x563b28e19b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51330/0 (socket says 172.21.15.110:51330) 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.449+0000 7fd2f1347640 1 -- 172.21.15.110:0/3464035392 learned_addr learned my addr 172.21.15.110:0/3464035392 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f1347640 1 -- 172.21.15.110:0/3464035392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b28e1a0b0 con 0x563b28e19750 2024-04-27T03:18:09.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f1347640 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e19750 0x563b28e19b70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800dfc0 tx=0x7fd2e80178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd1d67fc640 1 -- 172.21.15.110:0/3464035392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e80157d0 con 0x563b28e19750 2024-04-27T03:18:09.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd1d67fc640 1 -- 172.21.15.110:0/3464035392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e8015970 con 0x563b28e19750 2024-04-27T03:18:09.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563b28e1ab50 con 0x563b28e19750 2024-04-27T03:18:09.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd1d67fc640 1 -- 172.21.15.110:0/3464035392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e8015c70 con 0x563b28e19750 2024-04-27T03:18:09.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b28e1b020 con 0x563b28e19750 2024-04-27T03:18:09.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd1d67fc640 1 -- 172.21.15.110:0/3464035392 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd2e8029440 con 0x563b28e19750 2024-04-27T03:18:09.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd1d67fc640 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c8034650 0x7fd1c8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd1d67fc640 1 -- 172.21.15.110:0/3464035392 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fd2e800e040 con 0x563b28e19750 2024-04-27T03:18:09.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c8034650 0x7fd1c8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f302b980 1 Processor -- start 2024-04-27T03:18:09.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f302b980 1 -- start start 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c8034650 0x7fd1c8036b10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x563b28e09b60 tx=0x7fd2e0000bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f302b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e2b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f302b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563b28df7f50 con 0x563b28e2ad60 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f1347640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e2b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f1347640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e2b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51346/0 (socket says 172.21.15.110:51346) 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f1347640 1 -- 172.21.15.110:0/3006250246 learned_addr learned my addr 172.21.15.110:0/3006250246 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f1347640 1 -- 172.21.15.110:0/3006250246 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b28e2b750 con 0x563b28e2ad60 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.453+0000 7fd2f1347640 1 --2- 172.21.15.110:0/3006250246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e2b180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8015fa0 tx=0x7fd2e8015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ad5e3151530345 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd1c7fff640 1 -- 172.21.15.110:0/3006250246 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e80113a0 con 0x563b28e2ad60 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd1c7fff640 1 -- 172.21.15.110:0/3006250246 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e8011540 con 0x563b28e2ad60 2024-04-27T03:18:09.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd1c7fff640 1 -- 172.21.15.110:0/3006250246 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e804c240 con 0x563b28e2ad60 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- 172.21.15.110:0/3006250246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 msgr2=0x563b28e2b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3006250246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e2b180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8015fa0 tx=0x7fd2e8015fd0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- 172.21.15.110:0/3006250246 shutdown_connections 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3006250246 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e2b180 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- 172.21.15.110:0/3006250246 >> 172.21.15.110:0/3006250246 conn(0x563b28e270a0 msgr2=0x563b28e25100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- 172.21.15.110:0/3006250246 shutdown_connections 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- 172.21.15.110:0/3006250246 wait complete. 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 Processor -- start 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- start start 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e32200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563b28e2beb0 con 0x563b28e2ad60 2024-04-27T03:18:09.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f1347640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e32200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f1347640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e32200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51362/0 (socket says 172.21.15.110:51362) 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 learned_addr learned my addr 172.21.15.110:0/2612450914 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b28e32740 con 0x563b28e2ad60 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f1347640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e32200 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd2e801eb40 tx=0x7fd2e804a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd1c67fc640 1 -- 172.21.15.110:0/2612450914 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e80113a0 con 0x563b28e2ad60 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd1c67fc640 1 -- 172.21.15.110:0/2612450914 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e8013dd0 con 0x563b28e2ad60 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd1c67fc640 1 -- 172.21.15.110:0/2612450914 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e804adb0 con 0x563b28e2ad60 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563b28e329d0 con 0x563b28e2ad60 2024-04-27T03:18:09.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b28e356b0 con 0x563b28e2ad60 2024-04-27T03:18:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.457+0000 7fd1c67fc640 1 -- 172.21.15.110:0/2612450914 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd2e80491b0 con 0x563b28e2ad60 2024-04-27T03:18:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd1c67fc640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1bc034650 0x7fd1bc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd1c67fc640 1 -- 172.21.15.110:0/2612450914 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fd2e8002390 con 0x563b28e2ad60 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1bc034650 0x7fd1bc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28e3d8d0 0x563b28e3fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e404c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0345640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28e3d8d0 0x563b28e3fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1bc034650 0x7fd1bc036b10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00c5a70 tx=0x7fd2e00e8040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0345640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28e3d8d0 0x563b28e3fd70 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e400d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 realm 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e3cda0 con 0x563b28e3d8d0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e400d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e40fa0 con 0x563b28e3d8d0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e400d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e41d80 0x563b28e44170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e44840 con 0x563b28e41d80 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f1347640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e41d80 0x563b28e44170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f1347640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e41d80 0x563b28e44170 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e450e0 con 0x563b28e3d8d0 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fd2e400d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 20 searching for the correct realm 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563b28e46e50 0x563b28e49240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e46e50 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563b28e46e50 0x563b28e49240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.461+0000 7fd2f0b46640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563b28e46e50 0x563b28e49240 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f1347640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8014030 0x7fd2e801a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x7fd2e8014030 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8014030 0x7fd2e801a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8014030 0x7fd2e801a9b0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e400d040 con 0x7fd2e8014030 2024-04-27T03:18:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e400d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e46e50 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.465+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x7fd2e8014030 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x7fd2e8014030 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x7fd2e8014030 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fd2e402a040 con 0x7fd2e8014030 2024-04-27T03:18:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e46e50 2024-04-27T03:18:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e46e50 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.469+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e46e50 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x7fd2e8014030 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x7fd2e8014030 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e3d8d0 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e402a040 con 0x563b28e3d8d0 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x7fd2e8014030 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e46e50 2024-04-27T03:18:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x7fd2e8014030 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e46e50 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.473+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563b28e498c0 con 0x563b28e41d80 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e45a60 con 0x563b28e3d8d0 2024-04-27T03:18:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e45e50 con 0x563b28e41d80 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e462a0 con 0x563b28e3d8d0 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 zone default found 2024-04-27T03:18:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 4 Realm: () 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 zonegroup default 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e49d70 con 0x7fd2e8014030 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e45670 con 0x563b28e46e50 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4bd20 con 0x563b28e46e50 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 started sync module instance, tier type = 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.477+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4b090 con 0x563b28e41d80 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94812089136256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4c630 con 0x563b28e41d80 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4ce80 con 0x563b28e3d8d0 2024-04-27T03:18:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94812089140784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4cfc0 con 0x563b28e3d8d0 2024-04-27T03:18:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4dae0 con 0x563b28e41d80 2024-04-27T03:18:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94812089140784] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94812089136256] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94812089139584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4daa0 con 0x563b28e41d80 2024-04-27T03:18:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.481+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4e380 con 0x7fd2e8014030 2024-04-27T03:18:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94812089147088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4daa0 con 0x7fd2e8014030 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4ee10 con 0x563b28e41d80 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94812089147088] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94812089139584] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94812089144752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4ee10 con 0x563b28e41d80 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4f200 con 0x563b28e46e50 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94812089148144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4fa50 con 0x563b28e46e50 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e50450 con 0x563b28e46e50 2024-04-27T03:18:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94812089144752] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94812089149760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e50410 con 0x563b28e46e50 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e50ea0 con 0x7fd2e8014030 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.485+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94812089148144] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94812089149760] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'81 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94812089152928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4ff60 con 0x7fd2e8014030 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=0 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=1 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=2 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=3 2024-04-27T03:18:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=4 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=5 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=6 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94812089152928] v27'82 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 20 add_watcher() i=7 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4ff60 con 0x563b28e3d8d0 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.489+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94812089158064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4ff60 con 0x563b28e3d8d0 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94812089158064] v27'94 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd1a6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 5 note: GC not initialized 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd1a5fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 Filter name: none 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd1177fe640 20 reqs_thread_entry: start 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e77100 con 0x563b28e3d8d0 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e77af0 con 0x563b28e3d8d0 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563b28e78b00 con 0x7fd2e8014030 2024-04-27T03:18:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(64 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 155+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd1a6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.493+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94812089158064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e562f0 con 0x563b28e3d8d0 2024-04-27T03:18:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 data_loggenerations_metadata [watch unwatch cookie 94812089158064] v27'95 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94812089136256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e509b0 con 0x563b28e41d80 2024-04-27T03:18:09.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(66 notify.0 [watch unwatch cookie 94812089136256] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f302b980 20 remove_watcher() i=0 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f302b980 2 removed watcher, disabling cache 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94812089140784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4b480 con 0x563b28e3d8d0 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(67 notify.1 [watch unwatch cookie 94812089140784] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x563b28e3d8d0 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f302b980 20 remove_watcher() i=1 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.497+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94812089139584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4b480 con 0x563b28e41d80 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.501+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [watch unwatch cookie 94812089139584] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.501+0000 7fd2f302b980 20 remove_watcher() i=2 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.501+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94812089147088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4c180 con 0x7fd2e8014030 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.501+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [watch unwatch cookie 94812089147088] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.501+0000 7fd2f302b980 20 remove_watcher() i=3 2024-04-27T03:18:09.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.501+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94812089144752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4b480 con 0x563b28e41d80 2024-04-27T03:18:09.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f1347640 1 -- 172.21.15.110:0/2612450914 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(70 notify.4 [watch unwatch cookie 94812089144752] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8026020 con 0x563b28e41d80 2024-04-27T03:18:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 20 remove_watcher() i=4 2024-04-27T03:18:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94812089148144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4ded0 con 0x563b28e46e50 2024-04-27T03:18:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(71 notify.5 [watch unwatch cookie 94812089148144] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 20 remove_watcher() i=5 2024-04-27T03:18:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94812089149760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4c180 con 0x563b28e46e50 2024-04-27T03:18:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f0b46640 1 -- 172.21.15.110:0/2612450914 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(72 notify.6 [watch unwatch cookie 94812089149760] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00e50b0 con 0x563b28e46e50 2024-04-27T03:18:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 20 remove_watcher() i=6 2024-04-27T03:18:09.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94812089152928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563b28e4d5b0 con 0x7fd2e8014030 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(73 notify.7 [watch unwatch cookie 94812089152928] v27'83 uv81 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e403d040 con 0x7fd2e8014030 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 20 remove_watcher() i=7 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e41d80 msgr2=0x563b28e44170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e41d80 0x563b28e44170 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28e3d8d0 msgr2=0x563b28e3fd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28e3d8d0 0x563b28e3fd70 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8014030 msgr2=0x7fd2e801a9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8014030 0x7fd2e801a9b0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563b28e46e50 msgr2=0x563b28e49240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563b28e46e50 0x563b28e49240 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1bc034650 msgr2=0x7fd1bc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1bc034650 0x7fd1bc036b10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00c5a70 tx=0x7fd2e00e8040 comp rx=0 tx=0).stop 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f0345640 1 -- 172.21.15.110:0/2612450914 reap_dead start 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 msgr2=0x563b28e32200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e32200 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd2e801eb40 tx=0x7fd2e804a880 comp rx=0 tx=0).stop 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 shutdown_connections 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 --2- 172.21.15.110:0/2612450914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e2ad60 0x563b28e32200 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 >> 172.21.15.110:0/2612450914 conn(0x563b28e270a0 msgr2=0x563b28e202f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 shutdown_connections 2024-04-27T03:18:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.505+0000 7fd2f302b980 1 -- 172.21.15.110:0/2612450914 wait complete. 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c8034650 msgr2=0x7fd1c8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c8034650 0x7fd1c8036b10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x563b28e09b60 tx=0x7fd2e0000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e19750 msgr2=0x563b28e19b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e19750 0x563b28e19b70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800dfc0 tx=0x7fd2e80178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 shutdown_connections 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c8034650 0x7fd1c8036b10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3464035392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28e19750 0x563b28e19b70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 >> 172.21.15.110:0/3464035392 conn(0x563b28e0e560 msgr2=0x563b28e0f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 shutdown_connections 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3464035392 wait complete. 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e06ae0 msgr2=0x563b28e08ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e06ae0 0x563b28e08ed0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28d0da70 msgr2=0x563b28e056f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28d0da70 0x563b28e056f0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d8034650 msgr2=0x7fd1d8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d8034650 0x7fd1d8036b10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4000f70 tx=0x7fd2e4001420 comp rx=0 tx=0).stop 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 msgr2=0x563b28dfe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28dfe840 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd2e001b170 tx=0x7fd2e00180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 shutdown_connections 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563b28e06ae0 0x563b28e08ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563b28d0da70 0x563b28e056f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d8034650 0x7fd1d8036b10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 --2- 172.21.15.110:0/3567032358 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563b28d0d150 0x563b28dfe840 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 >> 172.21.15.110:0/3567032358 conn(0x563b28d08a20 msgr2=0x563b28df1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 shutdown_connections 2024-04-27T03:18:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.509+0000 7fd2f302b980 1 -- 172.21.15.110:0/3567032358 wait complete. 2024-04-27T03:18:09.525 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2024-04-27T03:18:09.525 DEBUG:tasks.util.rgw: json result: [] 2024-04-27T03:18:09.525 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list', '--uid', 'foo'] 2024-04-27T03:18:09.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list', '--uid', 'foo'] 2024-04-27T03:18:09.525 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list --uid foo 2024-04-27T03:18:09.596 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:09.596 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9066ee9980 1 Processor -- start 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9066ee9980 1 -- start start 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9066ee9980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5639172d9850 0x56391736a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9066ee9980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5639172240d0 con 0x56391736a880 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9066ee9980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x563917219430 con 0x5639172d9850 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9064a04640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5639172d9850 0x56391736a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9064203640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x56391736a880 0x56391736cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34750/0 (socket says 172.21.15.110:34750) 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9064a04640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5639172d9850 0x56391736a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51368/0 (socket says 172.21.15.110:51368) 2024-04-27T03:18:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9064203640 1 -- 172.21.15.110:0/883681182 learned_addr learned my addr 172.21.15.110:0/883681182 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9064a04640 1 -- 172.21.15.110:0/883681182 >> v1:172.21.15.77:6789/0 conn(0x56391736a880 legacy=0x56391736cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.589+0000 7f9064a04640 1 -- 172.21.15.110:0/883681182 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x563917212510 con 0x5639172d9850 2024-04-27T03:18:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9064a04640 1 --2- 172.21.15.110:0/883681182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x56391736a340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f904c00a480 tx=0x7f904c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9408d8415163f5e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9063a02640 1 -- 172.21.15.110:0/883681182 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f904c0b0070 con 0x5639172d9850 2024-04-27T03:18:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9063a02640 1 -- 172.21.15.110:0/883681182 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c0ab540 con 0x5639172d9850 2024-04-27T03:18:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9063a02640 1 -- 172.21.15.110:0/883681182 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f904c0ab840 con 0x5639172d9850 2024-04-27T03:18:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9066ee9980 1 -- 172.21.15.110:0/883681182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 msgr2=0x56391736a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9066ee9980 1 --2- 172.21.15.110:0/883681182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x56391736a340 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f904c00a480 tx=0x7f904c01b280 comp rx=0 tx=0).stop 2024-04-27T03:18:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9066ee9980 1 -- 172.21.15.110:0/883681182 shutdown_connections 2024-04-27T03:18:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9066ee9980 1 --2- 172.21.15.110:0/883681182 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x56391736a340 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9066ee9980 1 -- 172.21.15.110:0/883681182 >> 172.21.15.110:0/883681182 conn(0x56391735fa20 msgr2=0x563917361e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9066ee9980 1 -- 172.21.15.110:0/883681182 shutdown_connections 2024-04-27T03:18:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.593+0000 7f9066ee9980 1 -- 172.21.15.110:0/883681182 wait complete. 2024-04-27T03:18:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 Processor -- start 2024-04-27T03:18:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- start start 2024-04-27T03:18:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917452620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639172682d0 con 0x5639172d9850 2024-04-27T03:18:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917452620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917452620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51374/0 (socket says 172.21.15.110:51374) 2024-04-27T03:18:09.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 -- 172.21.15.110:0/3718555105 learned_addr learned my addr 172.21.15.110:0/3718555105 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 -- 172.21.15.110:0/3718555105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639171eeae0 con 0x5639172d9850 2024-04-27T03:18:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 --2- 172.21.15.110:0/3718555105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917452620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f904c0ae040 tx=0x7f904c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb46cdd7fde25ce server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f8f52e0d640 1 -- 172.21.15.110:0/3718555105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f904c0b0040 con 0x5639172d9850 2024-04-27T03:18:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f8f52e0d640 1 -- 172.21.15.110:0/3718555105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c0ab9d0 con 0x5639172d9850 2024-04-27T03:18:09.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f8f52e0d640 1 -- 172.21.15.110:0/3718555105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f904c0abcf0 con 0x5639172d9850 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- 172.21.15.110:0/3718555105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 msgr2=0x563917452620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3718555105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917452620 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f904c0ae040 tx=0x7f904c003030 comp rx=0 tx=0).stop 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- 172.21.15.110:0/3718555105 shutdown_connections 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3718555105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917452620 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- 172.21.15.110:0/3718555105 >> 172.21.15.110:0/3718555105 conn(0x56391735fa20 msgr2=0x56391744e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- 172.21.15.110:0/3718555105 shutdown_connections 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- 172.21.15.110:0/3718555105 wait complete. 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 Processor -- start 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- start start 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917459630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9066ee9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563917370df0 con 0x5639172d9850 2024-04-27T03:18:09.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917459630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917459630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51386/0 (socket says 172.21.15.110:51386) 2024-04-27T03:18:09.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.609+0000 7f9064a04640 1 -- 172.21.15.110:0/2461546150 learned_addr learned my addr 172.21.15.110:0/2461546150 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9064a04640 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563917459b70 con 0x5639172d9850 2024-04-27T03:18:09.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9064a04640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917459630 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f904c01b2c0 tx=0x7f904c018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f8f5160a640 1 -- 172.21.15.110:0/2461546150 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f904c0b0050 con 0x5639172d9850 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563917459e00 con 0x5639172d9850 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f8f5160a640 1 -- 172.21.15.110:0/2461546150 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c0b6870 con 0x5639172d9850 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f8f5160a640 1 -- 172.21.15.110:0/2461546150 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f904c0b6b50 con 0x5639172d9850 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391745ca30 con 0x5639172d9850 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f8f5160a640 1 -- 172.21.15.110:0/2461546150 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f904c0b7420 con 0x5639172d9850 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f8f5160a640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f4c034650 0x7f8f4c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f8f5160a640 1 -- 172.21.15.110:0/2461546150 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f904c0bc070 con 0x5639172d9850 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9064203640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f4c034650 0x7f8f4c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56391736a880 0x563917460510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563917460ae0 con 0x56391736a880 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9065205640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56391736a880 0x563917460510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9064203640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f4c034650 0x7f8f4c036b10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90540012f0 tx=0x7f9054006700 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9065205640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56391736a880 0x563917460510 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9065205640 1 -- 172.21.15.110:0/2461546150 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f905c0083f0 con 0x56391736a880 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917461890 0x563917463c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174642c0 con 0x563917461890 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.613+0000 7f9064a04640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917461890 0x563917463c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9064a04640 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917461890 0x563917463c80 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9064a04640 1 -- 172.21.15.110:0/2461546150 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f904c0c7020 con 0x563917461890 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563917465b70 con 0x56391736a880 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9065205640 1 -- 172.21.15.110:0/2461546150 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f905c0083f0 con 0x56391736a880 2024-04-27T03:18:09.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563917464da0 con 0x563917461890 2024-04-27T03:18:09.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9064a04640 1 -- 172.21.15.110:0/2461546150 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f904c0c7020 con 0x563917461890 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563917465190 con 0x56391736a880 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9065205640 1 -- 172.21.15.110:0/2461546150 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f905c0083f0 con 0x56391736a880 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 Processor -- start 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 -- start start 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917474480 0x5639174748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56391726c310 con 0x563917474480 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9065205640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917474480 0x5639174748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9065205640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917474480 0x5639174748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51390/0 (socket says 172.21.15.110:51390) 2024-04-27T03:18:09.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9065205640 1 -- 172.21.15.110:0/3177108299 learned_addr learned my addr 172.21.15.110:0/3177108299 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9065205640 1 -- 172.21.15.110:0/3177108299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563917474de0 con 0x563917474480 2024-04-27T03:18:09.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9065205640 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917474480 0x5639174748a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f905c00aab0 tx=0x7f905c00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f8f4a7fc640 1 -- 172.21.15.110:0/3177108299 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c007d30 con 0x563917474480 2024-04-27T03:18:09.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563917475880 con 0x563917474480 2024-04-27T03:18:09.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f8f4a7fc640 1 -- 172.21.15.110:0/3177108299 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f905c01d070 con 0x563917474480 2024-04-27T03:18:09.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f8f4a7fc640 1 -- 172.21.15.110:0/3177108299 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c018440 con 0x563917474480 2024-04-27T03:18:09.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.617+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563917475d50 con 0x563917474480 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f8f4a7fc640 1 -- 172.21.15.110:0/3177108299 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f905c013070 con 0x563917474480 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f8f4a7fc640 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f3c034650 0x7f8f3c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f8f4a7fc640 1 -- 172.21.15.110:0/3177108299 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f905c04b5e0 con 0x563917474480 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9064a04640 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f3c034650 0x7f8f3c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 Processor -- start 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 -- start start 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9064a04640 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f3c034650 0x7f8f3c036b10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x5639174648c0 tx=0x7f904c0e2de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917481a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563917452cb0 con 0x563917481660 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9065205640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917481a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9065205640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917481a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51402/0 (socket says 172.21.15.110:51402) 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9065205640 1 -- 172.21.15.110:0/1246247331 learned_addr learned my addr 172.21.15.110:0/1246247331 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9065205640 1 -- 172.21.15.110:0/1246247331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563917482050 con 0x563917481660 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9065205640 1 --2- 172.21.15.110:0/1246247331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917481a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f905c049930 tx=0x7f905c049b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cd3bb000e2099a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f8f3bfff640 1 -- 172.21.15.110:0/1246247331 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c047070 con 0x563917481660 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f8f3bfff640 1 -- 172.21.15.110:0/1246247331 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f905c022460 con 0x563917481660 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f8f3bfff640 1 -- 172.21.15.110:0/1246247331 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c018bc0 con 0x563917481660 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 -- 172.21.15.110:0/1246247331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 msgr2=0x563917481a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 --2- 172.21.15.110:0/1246247331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917481a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f905c049930 tx=0x7f905c049b30 comp rx=0 tx=0).stop 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 -- 172.21.15.110:0/1246247331 shutdown_connections 2024-04-27T03:18:09.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 --2- 172.21.15.110:0/1246247331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917481a80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 -- 172.21.15.110:0/1246247331 >> 172.21.15.110:0/1246247331 conn(0x56391747d9a0 msgr2=0x56391747ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 -- 172.21.15.110:0/1246247331 shutdown_connections 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.621+0000 7f9066ee9980 1 -- 172.21.15.110:0/1246247331 wait complete. 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 Processor -- start 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 -- start start 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917488b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639174827d0 con 0x563917481660 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9065205640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917488b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9065205640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917488b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51404/0 (socket says 172.21.15.110:51404) 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 learned_addr learned my addr 172.21.15.110:0/3738501433 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563917489080 con 0x563917481660 2024-04-27T03:18:09.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9065205640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917488b40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f905c049930 tx=0x7f905c0498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f8f3a7fc640 1 -- 172.21.15.110:0/3738501433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c031070 con 0x563917481660 2024-04-27T03:18:09.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563917489310 con 0x563917481660 2024-04-27T03:18:09.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f8f3a7fc640 1 -- 172.21.15.110:0/3738501433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f905c016d60 con 0x563917481660 2024-04-27T03:18:09.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f8f3a7fc640 1 -- 172.21.15.110:0/3738501433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f905c020820 con 0x563917481660 2024-04-27T03:18:09.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56391748bff0 con 0x563917481660 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f8f3a7fc640 1 -- 172.21.15.110:0/3738501433 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f905c034070 con 0x563917481660 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f8f3a7fc640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f30034650 0x7f8f30036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f8f3a7fc640 1 -- 172.21.15.110:0/3738501433 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f905c037c90 con 0x563917481660 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9064a04640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f30034650 0x7f8f30036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639174941c0 0x563917496660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563917496db0 con 0x5639174941c0 2024-04-27T03:18:09.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9064203640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639174941c0 0x563917496660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9064a04640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f30034650 0x7f8f30036b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f904c0befd0 tx=0x7f904c0e1000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9064203640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639174941c0 0x563917496660 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 20 realm 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.625+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563917493690 con 0x5639174941c0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174978b0 con 0x5639174941c0 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917498490 0x56391749a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56391749b0c0 con 0x563917498490 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9065205640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917498490 0x56391749a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9065205640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917498490 0x56391749a950 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56391749b960 con 0x5639174941c0 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 20 searching for the correct realm 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56391749d6d0 0x56391749fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x56391749d6d0 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9064a04640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56391749d6d0 0x56391749fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9064a04640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56391749d6d0 0x56391749fac0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.629+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9065205640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f905c036070 0x7f905c0218a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x7f905c036070 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f905c036070 0x7f905c0218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f905c036070 0x7f905c0218a0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x7f905c036070 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x56391749d6d0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x7f905c036070 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x7f905c036070 2024-04-27T03:18:09.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.633+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x7f905c036070 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f905400f070 con 0x7f905c036070 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x56391749d6d0 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x56391749d6d0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x56391749d6d0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x7f905c036070 2024-04-27T03:18:09.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x7f905c036070 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x5639174941c0 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x7f905c036070 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x7f905c036070 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x56391749d6d0 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.637+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x7f905c036070 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905400f070 con 0x7f905c036070 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x56391749d6d0 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5639174a00a0 con 0x563917498490 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:09.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56391749c310 con 0x5639174941c0 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f905400f070 con 0x5639174941c0 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.641+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56391749c790 con 0x563917498490 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56391749cbb0 con 0x5639174941c0 2024-04-27T03:18:09.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 zone default found 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 4 Realm: () 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 zonegroup default 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174a09d0 con 0x7f905c036070 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174a2750 con 0x56391749d6d0 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174a2c60 con 0x56391749d6d0 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 started sync module instance, tier type = 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a1bf0 con 0x563917498490 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94803203858400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a1850 con 0x563917498490 2024-04-27T03:18:09.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.645+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a3b90 con 0x5639174941c0 2024-04-27T03:18:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94803203864288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a3fb0 con 0x5639174941c0 2024-04-27T03:18:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a49d0 con 0x563917498490 2024-04-27T03:18:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94803203858400] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94803203864288] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94803203867840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a3b00 con 0x563917498490 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a5270 con 0x7f905c036070 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94803203870144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a3b00 con 0x7f905c036070 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.649+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a5d10 con 0x563917498490 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94803203867840] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94803203870144] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94803203871200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a5cd0 con 0x563917498490 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a6100 con 0x56391749d6d0 2024-04-27T03:18:09.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94803203862720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a69a0 con 0x56391749d6d0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a7360 con 0x56391749d6d0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94803203871200] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94803203862720] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94803203878576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a69a0 con 0x56391749d6d0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.653+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a7c00 con 0x7f905c036070 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'84 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94803203880784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a8020 con 0x7f905c036070 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94803203878576] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=0 2024-04-27T03:18:09.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=1 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=2 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=3 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=4 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=5 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=6 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94803203880784] v27'85 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 20 add_watcher() i=7 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174aac90 con 0x5639174941c0 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94803203876080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174a8020 con 0x5639174941c0 2024-04-27T03:18:09.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.657+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94803203876080] v27'96 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f8f1afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 5 note: GC not initialized 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f8f19fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 Filter name: none 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f8e8b7fe640 20 reqs_thread_entry: start 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174ce0b0 con 0x5639174941c0 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:09.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174ceaa0 con 0x5639174941c0 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5639174cfab0 con 0x7f905c036070 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(64 foo.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 155+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f8f1afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.661+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94803203876080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x5639174941c0 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 data_loggenerations_metadata [watch unwatch cookie 94803203876080] v27'97 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94803203858400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x563917498490 2024-04-27T03:18:09.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(66 notify.0 [watch unwatch cookie 94803203858400] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9066ee9980 20 remove_watcher() i=0 2024-04-27T03:18:09.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9066ee9980 2 removed watcher, disabling cache 2024-04-27T03:18:09.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94803203864288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x5639174941c0 2024-04-27T03:18:09.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(67 notify.1 [watch unwatch cookie 94803203864288] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x5639174941c0 2024-04-27T03:18:09.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9066ee9980 20 remove_watcher() i=1 2024-04-27T03:18:09.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.665+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94803203867840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x563917498490 2024-04-27T03:18:09.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.669+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [watch unwatch cookie 94803203867840] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.669+0000 7f9066ee9980 20 remove_watcher() i=2 2024-04-27T03:18:09.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.669+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94803203870144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x7f905c036070 2024-04-27T03:18:09.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.669+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [watch unwatch cookie 94803203870144] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.669+0000 7f9066ee9980 20 remove_watcher() i=3 2024-04-27T03:18:09.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.669+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94803203871200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x563917498490 2024-04-27T03:18:09.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9065205640 1 -- 172.21.15.110:0/3738501433 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(70 notify.4 [watch unwatch cookie 94803203871200] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f905c033030 con 0x563917498490 2024-04-27T03:18:09.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 20 remove_watcher() i=4 2024-04-27T03:18:09.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94803203862720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x56391749d6d0 2024-04-27T03:18:09.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(71 notify.5 [watch unwatch cookie 94803203862720] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 20 remove_watcher() i=5 2024-04-27T03:18:09.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94803203878576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x56391749d6d0 2024-04-27T03:18:09.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9064a04640 1 -- 172.21.15.110:0/3738501433 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(72 notify.6 [watch unwatch cookie 94803203878576] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f904c0c7020 con 0x56391749d6d0 2024-04-27T03:18:09.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 20 remove_watcher() i=6 2024-04-27T03:18:09.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94803203880784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5639174cbe40 con 0x7f905c036070 2024-04-27T03:18:09.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(73 notify.7 [watch unwatch cookie 94803203880784] v27'86 uv84 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f90540401c0 con 0x7f905c036070 2024-04-27T03:18:09.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 20 remove_watcher() i=7 2024-04-27T03:18:09.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917498490 msgr2=0x56391749a950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917498490 0x56391749a950 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639174941c0 msgr2=0x563917496660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639174941c0 0x563917496660 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f905c036070 msgr2=0x7f905c0218a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f905c036070 0x7f905c0218a0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56391749d6d0 msgr2=0x56391749fac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56391749d6d0 0x56391749fac0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f30034650 msgr2=0x7f8f30036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f30034650 0x7f8f30036b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f904c0befd0 tx=0x7f904c0e1000 comp rx=0 tx=0).stop 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 msgr2=0x563917488b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9064203640 1 -- 172.21.15.110:0/3738501433 reap_dead start 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917488b40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f905c049930 tx=0x7f905c0498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 shutdown_connections 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3738501433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917481660 0x563917488b40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 >> 172.21.15.110:0/3738501433 conn(0x56391747d9a0 msgr2=0x56391747b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 shutdown_connections 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.673+0000 7f9066ee9980 1 -- 172.21.15.110:0/3738501433 wait complete. 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f3c034650 msgr2=0x7f8f3c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f3c034650 0x7f8f3c036b10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x5639174648c0 tx=0x7f904c0e2de0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917474480 msgr2=0x5639174748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917474480 0x5639174748a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f905c00aab0 tx=0x7f905c00af70 comp rx=0 tx=0).stop 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 shutdown_connections 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f3c034650 0x7f8f3c036b10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/3177108299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563917474480 0x5639174748a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 >> 172.21.15.110:0/3177108299 conn(0x563917469290 msgr2=0x563917469da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 shutdown_connections 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/3177108299 wait complete. 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917461890 msgr2=0x563917463c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917461890 0x563917463c80 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56391736a880 msgr2=0x563917460510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56391736a880 0x563917460510 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f4c034650 msgr2=0x7f8f4c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f4c034650 0x7f8f4c036b10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f90540012f0 tx=0x7f9054006700 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 msgr2=0x563917459630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917459630 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f904c01b2c0 tx=0x7f904c018020 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 shutdown_connections 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563917461890 0x563917463c80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56391736a880 0x563917460510 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f4c034650 0x7f8f4c036b10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 --2- 172.21.15.110:0/2461546150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639172d9850 0x563917459630 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 >> 172.21.15.110:0/2461546150 conn(0x56391735fa20 msgr2=0x56391744ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 shutdown_connections 2024-04-27T03:18:09.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.677+0000 7f9066ee9980 1 -- 172.21.15.110:0/2461546150 wait complete. 2024-04-27T03:18:09.693 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2024-04-27T03:18:09.693 DEBUG:tasks.util.rgw: json result: [] 2024-04-27T03:18:09.710 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:09.710 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list', '--uid', 'foo'] 2024-04-27T03:18:09.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list', '--uid', 'foo'] 2024-04-27T03:18:09.710 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list --uid foo 2024-04-27T03:18:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe85931f980 1 Processor -- start 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe85931f980 1 -- start start 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe85931f980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5578f25b34b0 0x5578f25b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe85931f980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5578f24700d0 con 0x5578f25b3dd0 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe85931f980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5578f2465430 con 0x5578f25b34b0 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe856e3a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5578f25b34b0 0x5578f25b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe856e3a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5578f25b34b0 0x5578f25b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51414/0 (socket says 172.21.15.110:51414) 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe856e3a640 1 -- 172.21.15.110:0/277050222 learned_addr learned my addr 172.21.15.110:0/277050222 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe856e3a640 1 -- 172.21.15.110:0/277050222 >> v1:172.21.15.77:6789/0 conn(0x5578f25b3dd0 legacy=0x5578f25b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.757+0000 7fe856e3a640 1 -- 172.21.15.110:0/277050222 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f245e510 con 0x5578f25b34b0 2024-04-27T03:18:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.761+0000 7fe856e3a640 1 --2- 172.21.15.110:0/277050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f25b3890 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe84800a5c0 tx=0x7fe84801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e54dd7e91e11903 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.761+0000 7fe855e38640 1 -- 172.21.15.110:0/277050222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8480a6070 con 0x5578f25b34b0 2024-04-27T03:18:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.761+0000 7fe855e38640 1 -- 172.21.15.110:0/277050222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8480af070 con 0x5578f25b34b0 2024-04-27T03:18:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.761+0000 7fe855e38640 1 -- 172.21.15.110:0/277050222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8480aa5c0 con 0x5578f25b34b0 2024-04-27T03:18:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.765+0000 7fe85931f980 1 -- 172.21.15.110:0/277050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 msgr2=0x5578f25b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.765+0000 7fe85931f980 1 --2- 172.21.15.110:0/277050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f25b3890 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe84800a5c0 tx=0x7fe84801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.765+0000 7fe85931f980 1 -- 172.21.15.110:0/277050222 shutdown_connections 2024-04-27T03:18:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.765+0000 7fe85931f980 1 --2- 172.21.15.110:0/277050222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f25b3890 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.765+0000 7fe85931f980 1 -- 172.21.15.110:0/277050222 >> 172.21.15.110:0/277050222 conn(0x5578f2526b90 msgr2=0x5578f2526fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.765+0000 7fe85931f980 1 -- 172.21.15.110:0/277050222 shutdown_connections 2024-04-27T03:18:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.765+0000 7fe85931f980 1 -- 172.21.15.110:0/277050222 wait complete. 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 Processor -- start 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 -- start start 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f269ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5578f24b42d0 con 0x5578f25b34b0 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe856e3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f269ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe856e3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f269ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51430/0 (socket says 172.21.15.110:51430) 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe856e3a640 1 -- 172.21.15.110:0/935543295 learned_addr learned my addr 172.21.15.110:0/935543295 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe856e3a640 1 -- 172.21.15.110:0/935543295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f243aae0 con 0x5578f25b34b0 2024-04-27T03:18:09.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe856e3a640 1 --2- 172.21.15.110:0/935543295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f269ed10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe8480199f0 tx=0x7fe848003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381cf8c20babeaef server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe744e09640 1 -- 172.21.15.110:0/935543295 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8480aaa00 con 0x5578f25b34b0 2024-04-27T03:18:09.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe744e09640 1 -- 172.21.15.110:0/935543295 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8480aaba0 con 0x5578f25b34b0 2024-04-27T03:18:09.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe744e09640 1 -- 172.21.15.110:0/935543295 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8480bd3f0 con 0x5578f25b34b0 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 -- 172.21.15.110:0/935543295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 msgr2=0x5578f269ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 --2- 172.21.15.110:0/935543295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f269ed10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe8480199f0 tx=0x7fe848003030 comp rx=0 tx=0).stop 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 -- 172.21.15.110:0/935543295 shutdown_connections 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 --2- 172.21.15.110:0/935543295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f269ed10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 -- 172.21.15.110:0/935543295 >> 172.21.15.110:0/935543295 conn(0x5578f2526b90 msgr2=0x5578f269a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 -- 172.21.15.110:0/935543295 shutdown_connections 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 -- 172.21.15.110:0/935543295 wait complete. 2024-04-27T03:18:09.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.777+0000 7fe85931f980 1 Processor -- start 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 -- start start 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f26a5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5578f25bd500 con 0x5578f25b34b0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe856e3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f26a5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe856e3a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f26a5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51444/0 (socket says 172.21.15.110:51444) 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe856e3a640 1 -- 172.21.15.110:0/2866176370 learned_addr learned my addr 172.21.15.110:0/2866176370 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe856e3a640 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f26a61d0 con 0x5578f25b34b0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe856e3a640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f26a5c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe84801b170 tx=0x7fe8480180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe743606640 1 -- 172.21.15.110:0/2866176370 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8480bd7c0 con 0x5578f25b34b0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5578f26a6460 con 0x5578f25b34b0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe743606640 1 -- 172.21.15.110:0/2866176370 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8480bd960 con 0x5578f25b34b0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578f26a9040 con 0x5578f25b34b0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe743606640 1 -- 172.21.15.110:0/2866176370 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8480bdc40 con 0x5578f25b34b0 2024-04-27T03:18:09.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe743606640 1 -- 172.21.15.110:0/2866176370 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe8480ce020 con 0x5578f25b34b0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe743606640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe73c034650 0x7fe73c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe743606640 1 -- 172.21.15.110:0/2866176370 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe8480cc020 con 0x5578f25b34b0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe856639640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe73c034650 0x7fe73c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f25b3dd0 0x5578f26acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ad0a0 con 0x5578f25b3dd0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85763b640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f25b3dd0 0x5578f26acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe856639640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe73c034650 0x7fe73c036b10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe84c001e20 tx=0x7fe84c0013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85763b640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f25b3dd0 0x5578f26acb20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85763b640 1 -- 172.21.15.110:0/2866176370 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe85000a4d0 con 0x5578f25b3dd0 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26ade90 0x5578f26b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.781+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26b08e0 con 0x5578f26ade90 2024-04-27T03:18:09.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe856e3a640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26ade90 0x5578f26b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe856e3a640 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26ade90 0x5578f26b0280 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe856e3a640 1 -- 172.21.15.110:0/2866176370 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26ade90 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26b2190 con 0x5578f25b3dd0 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85763b640 1 -- 172.21.15.110:0/2866176370 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fe85000a4d0 con 0x5578f25b3dd0 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26b13c0 con 0x5578f26ade90 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe856e3a640 1 -- 172.21.15.110:0/2866176370 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26ade90 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26b17e0 con 0x5578f25b3dd0 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85763b640 1 -- 172.21.15.110:0/2866176370 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fe85000a4d0 con 0x5578f25b3dd0 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85931f980 1 Processor -- start 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85931f980 1 -- start start 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85931f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26c0ae0 0x5578f26c0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85931f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5578f24b8310 con 0x5578f26c0ae0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85763b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26c0ae0 0x5578f26c0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85763b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26c0ae0 0x5578f26c0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51448/0 (socket says 172.21.15.110:51448) 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.785+0000 7fe85763b640 1 -- 172.21.15.110:0/1547261460 learned_addr learned my addr 172.21.15.110:0/1547261460 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85763b640 1 -- 172.21.15.110:0/1547261460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f26c1440 con 0x5578f26c0ae0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85763b640 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26c0ae0 0x5578f26c0f00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe85000dfc0 tx=0x7fe8500178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe740e01640 1 -- 172.21.15.110:0/1547261460 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8500157d0 con 0x5578f26c0ae0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5578f26c1ee0 con 0x5578f26c0ae0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe740e01640 1 -- 172.21.15.110:0/1547261460 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe850015970 con 0x5578f26c0ae0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578f26c23b0 con 0x5578f26c0ae0 2024-04-27T03:18:09.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe740e01640 1 -- 172.21.15.110:0/1547261460 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe850015c50 con 0x5578f26c0ae0 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe740e01640 1 -- 172.21.15.110:0/1547261460 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe850020440 con 0x5578f26c0ae0 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe740e01640 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe7300346a0 0x7fe730036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe740e01640 1 -- 172.21.15.110:0/1547261460 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe85000e040 con 0x5578f26c0ae0 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe856e3a640 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe7300346a0 0x7fe730036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85931f980 1 Processor -- start 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85931f980 1 -- start start 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe856e3a640 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe7300346a0 0x7fe730036b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5578f26ad650 tx=0x7fe8480de040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85931f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85931f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5578f269f3a0 con 0x5578f26cdc90 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85763b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85763b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51450/0 (socket says 172.21.15.110:51450) 2024-04-27T03:18:09.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85763b640 1 -- 172.21.15.110:0/3107847569 learned_addr learned my addr 172.21.15.110:0/3107847569 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.789+0000 7fe85763b640 1 -- 172.21.15.110:0/3107847569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f26ce680 con 0x5578f26cdc90 2024-04-27T03:18:09.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85763b640 1 --2- 172.21.15.110:0/3107847569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26ce0b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe85001f9e0 tx=0x7fe85003c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fd8b5ad19cef22 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe73a7fc640 1 -- 172.21.15.110:0/3107847569 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe850048030 con 0x5578f26cdc90 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe73a7fc640 1 -- 172.21.15.110:0/3107847569 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe850012570 con 0x5578f26cdc90 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe73a7fc640 1 -- 172.21.15.110:0/3107847569 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe850020c30 con 0x5578f26cdc90 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- 172.21.15.110:0/3107847569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 msgr2=0x5578f26ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 --2- 172.21.15.110:0/3107847569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26ce0b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe85001f9e0 tx=0x7fe85003c4a0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- 172.21.15.110:0/3107847569 shutdown_connections 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 --2- 172.21.15.110:0/3107847569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26ce0b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- 172.21.15.110:0/3107847569 >> 172.21.15.110:0/3107847569 conn(0x5578f26ca020 msgr2=0x5578f26c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- 172.21.15.110:0/3107847569 shutdown_connections 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- 172.21.15.110:0/3107847569 wait complete. 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 Processor -- start 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- start start 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26d5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5578f26ceda0 con 0x5578f26cdc90 2024-04-27T03:18:09.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85763b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26d5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85763b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26d5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51462/0 (socket says 172.21.15.110:51462) 2024-04-27T03:18:09.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 learned_addr learned my addr 172.21.15.110:0/2424125186 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f26d56b0 con 0x5578f26cdc90 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85763b640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26d5170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe8500498f0 tx=0x7fe85004a9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe738ff9640 1 -- 172.21.15.110:0/2424125186 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe850038020 con 0x5578f26cdc90 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe738ff9640 1 -- 172.21.15.110:0/2424125186 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85001faa0 con 0x5578f26cdc90 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5578f26d5940 con 0x5578f26cdc90 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578f26d8620 con 0x5578f26cdc90 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.793+0000 7fe738ff9640 1 -- 172.21.15.110:0/2424125186 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe850051060 con 0x5578f26cdc90 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe738ff9640 1 -- 172.21.15.110:0/2424125186 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe85003b020 con 0x5578f26cdc90 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe738ff9640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe720034650 0x7fe720036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe738ff9640 1 -- 172.21.15.110:0/2424125186 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe85002f040 con 0x5578f26cdc90 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856e3a640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe720034650 0x7fe720036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f26e0780 0x5578f26e2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26e3360 con 0x5578f26e0780 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856639640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f26e0780 0x5578f26e2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856e3a640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe720034650 0x7fe720036b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe8480bf020 tx=0x7fe8480e7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856639640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f26e0780 0x5578f26e2c10 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe84c00d040 con 0x5578f26e0780 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 realm 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26dfc90 con 0x5578f26e0780 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe84c00d040 con 0x5578f26e0780 2024-04-27T03:18:09.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26e3e60 con 0x5578f26e0780 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe84c00d040 con 0x5578f26e0780 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26e4c40 0x5578f26e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26e76e0 con 0x5578f26e4c40 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85763b640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26e4c40 0x5578f26e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85763b640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26e4c40 0x5578f26e7030 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26e7f80 con 0x5578f26e0780 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fe84c00d040 con 0x5578f26e0780 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 20 searching for the correct realm 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5578f26e9cf0 0x5578f26ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e9cf0 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.797+0000 7fe856e3a640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5578f26e9cf0 0x5578f26ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856e3a640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5578f26e9cf0 0x5578f26ec0e0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe85763b640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe850033070 0x7fe850029730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x7fe850033070 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe850033070 0x7fe850029730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe850033070 0x7fe850029730 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c00d040 con 0x7fe850033070 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c00d040 con 0x5578f26e0780 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e9cf0 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.801+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x7fe850033070 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x7fe850033070 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x7fe850033070 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fe84c02a040 con 0x7fe850033070 2024-04-27T03:18:09.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e9cf0 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.805+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e9cf0 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e9cf0 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x7fe850033070 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x7fe850033070 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e0780 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c02a040 con 0x5578f26e0780 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x7fe850033070 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e9cf0 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x7fe850033070 2024-04-27T03:18:09.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e9cf0 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.809+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ec670 con 0x5578f26e4c40 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:09.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26e8900 con 0x5578f26e0780 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26e8d80 con 0x5578f26e4c40 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26e91a0 con 0x5578f26e0780 2024-04-27T03:18:09.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 zone default found 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 4 Realm: () 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 zonegroup default 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ecaf0 con 0x7fe850033070 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ee980 con 0x5578f26e9cf0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26eed20 con 0x5578f26e9cf0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 started sync module instance, tier type = 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.813+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26eddc0 con 0x5578f26e4c40 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93977951789536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26eddc0 con 0x5578f26e4c40 2024-04-27T03:18:09.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ef6e0 con 0x5578f26e0780 2024-04-27T03:18:09.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93977951793712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26efb00 con 0x5578f26e0780 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f0500 con 0x5578f26e4c40 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93977951793712] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.817+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93977951789536] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93977951787280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ef650 con 0x5578f26e4c40 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f0dd0 con 0x7fe850033070 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93977951799584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26ef650 con 0x7fe850033070 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f1850 con 0x5578f26e4c40 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93977951799584] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93977951787280] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93977951797264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f1360 con 0x5578f26e4c40 2024-04-27T03:18:09.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.821+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f20f0 con 0x5578f26e9cf0 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93977951804480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f1360 con 0x5578f26e9cf0 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f2f60 con 0x5578f26e9cf0 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93977951797264] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93977951800640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f2f60 con 0x5578f26e9cf0 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f3960 con 0x7fe850033070 2024-04-27T03:18:09.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 93977951804480] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93977951800640] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'87 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93977951806608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f2ab0 con 0x7fe850033070 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=0 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=1 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=2 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=3 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=4 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=5 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=6 2024-04-27T03:18:09.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93977951806608] v27'88 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 20 add_watcher() i=7 2024-04-27T03:18:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f2ab0 con 0x5578f26e0780 2024-04-27T03:18:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.825+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93977951810672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f26f2ab0 con 0x5578f26e0780 2024-04-27T03:18:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93977951810672] v27'98 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe70d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 5 note: GC not initialized 2024-04-27T03:18:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe687fff640 20 reqs_thread_entry: start 2024-04-27T03:18:09.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 Filter name: none 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe685ffb640 20 reqs_thread_entry: start 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f2719ce0 con 0x5578f26e0780 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f271a6d0 con 0x5578f26e0780 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5578f271b6e0 con 0x7fe850033070 2024-04-27T03:18:09.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(64 foo.buckets [call out=183b] v0'0 uv2 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe70d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:09.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.829+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93977951810672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x5578f26e0780 2024-04-27T03:18:09.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.833+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 data_loggenerations_metadata [watch unwatch cookie 93977951810672] v27'99 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.833+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93977951789536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x5578f26e4c40 2024-04-27T03:18:09.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(66 notify.0 [watch unwatch cookie 93977951789536] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85931f980 20 remove_watcher() i=0 2024-04-27T03:18:09.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85931f980 2 removed watcher, disabling cache 2024-04-27T03:18:09.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93977951793712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x5578f26e0780 2024-04-27T03:18:09.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(67 notify.1 [watch unwatch cookie 93977951793712] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x5578f26e0780 2024-04-27T03:18:09.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85931f980 20 remove_watcher() i=1 2024-04-27T03:18:09.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93977951787280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x5578f26e4c40 2024-04-27T03:18:09.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [watch unwatch cookie 93977951787280] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85931f980 20 remove_watcher() i=2 2024-04-27T03:18:09.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.837+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93977951799584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x7fe850033070 2024-04-27T03:18:09.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [watch unwatch cookie 93977951799584] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 20 remove_watcher() i=3 2024-04-27T03:18:09.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93977951797264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x5578f26e4c40 2024-04-27T03:18:09.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85763b640 1 -- 172.21.15.110:0/2424125186 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(70 notify.4 [watch unwatch cookie 93977951797264] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe850042020 con 0x5578f26e4c40 2024-04-27T03:18:09.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 20 remove_watcher() i=4 2024-04-27T03:18:09.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93977951804480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x5578f26e9cf0 2024-04-27T03:18:09.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(71 notify.5 [watch unwatch cookie 93977951804480] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 20 remove_watcher() i=5 2024-04-27T03:18:09.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93977951800640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x5578f26e9cf0 2024-04-27T03:18:09.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe856e3a640 1 -- 172.21.15.110:0/2424125186 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(72 notify.6 [watch unwatch cookie 93977951800640] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe8480e50b0 con 0x5578f26e9cf0 2024-04-27T03:18:09.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 20 remove_watcher() i=6 2024-04-27T03:18:09.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.841+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93977951806608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5578f2717a70 con 0x7fe850033070 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(73 notify.7 [watch unwatch cookie 93977951806608] v27'89 uv87 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe84c03d040 con 0x7fe850033070 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 20 remove_watcher() i=7 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26e4c40 msgr2=0x5578f26e7030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26e4c40 0x5578f26e7030 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f26e0780 msgr2=0x5578f26e2c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f26e0780 0x5578f26e2c10 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe850033070 msgr2=0x7fe850029730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe850033070 0x7fe850029730 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5578f26e9cf0 msgr2=0x5578f26ec0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5578f26e9cf0 0x5578f26ec0e0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe720034650 msgr2=0x7fe720036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe720034650 0x7fe720036b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe8480bf020 tx=0x7fe8480e7070 comp rx=0 tx=0).stop 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 msgr2=0x5578f26d5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe856639640 1 -- 172.21.15.110:0/2424125186 reap_dead start 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26d5170 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe8500498f0 tx=0x7fe85004a9b0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 shutdown_connections 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2424125186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26cdc90 0x5578f26d5170 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 >> 172.21.15.110:0/2424125186 conn(0x5578f26ca020 msgr2=0x5578f26cf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 shutdown_connections 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2424125186 wait complete. 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe7300346a0 msgr2=0x7fe730036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe7300346a0 0x7fe730036b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x5578f26ad650 tx=0x7fe8480de040 comp rx=0 tx=0).stop 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26c0ae0 msgr2=0x5578f26c0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26c0ae0 0x5578f26c0f00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe85000dfc0 tx=0x7fe8500178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 shutdown_connections 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe7300346a0 0x7fe730036b60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/1547261460 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f26c0ae0 0x5578f26c0f00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 >> 172.21.15.110:0/1547261460 conn(0x5578f26b58f0 msgr2=0x5578f26b6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 shutdown_connections 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/1547261460 wait complete. 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26ade90 msgr2=0x5578f26b0280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26ade90 0x5578f26b0280 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f25b3dd0 msgr2=0x5578f26acb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f25b3dd0 0x5578f26acb20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe73c034650 msgr2=0x7fe73c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe73c034650 0x7fe73c036b10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe84c001e20 tx=0x7fe84c0013a0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 msgr2=0x5578f26a5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f26a5c90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe84801b170 tx=0x7fe8480180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 shutdown_connections 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5578f26ade90 0x5578f26b0280 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5578f25b3dd0 0x5578f26acb20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe73c034650 0x7fe73c036b10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 --2- 172.21.15.110:0/2866176370 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5578f25b34b0 0x5578f26a5c90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 >> 172.21.15.110:0/2866176370 conn(0x5578f2526b90 msgr2=0x5578f26993b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 shutdown_connections 2024-04-27T03:18:09.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.845+0000 7fe85931f980 1 -- 172.21.15.110:0/2866176370 wait complete. 2024-04-27T03:18:09.862 INFO:teuthology.orchestra.run.smithi110.stdout:["myfoo"] 2024-04-27T03:18:09.862 DEBUG:tasks.util.rgw: json result: ['myfoo'] 2024-04-27T03:18:09.866 INFO:tasks.radosgw_admin:RL: list_buckets foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:09.867 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'list'] 2024-04-27T03:18:09.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'list'] 2024-04-27T03:18:09.867 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket list 2024-04-27T03:18:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:09.929 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.921+0000 7f5fdef3b980 1 Processor -- start 2024-04-27T03:18:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.921+0000 7f5fdef3b980 1 -- start start 2024-04-27T03:18:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.921+0000 7f5fdef3b980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c00131d790 0x55c00131fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.921+0000 7f5fdef3b980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55c0011d80d0 con 0x55c001319280 2024-04-27T03:18:09.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.921+0000 7f5fdef3b980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55c0011cd430 con 0x55c00131d790 2024-04-27T03:18:09.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.921+0000 7f5fd4255640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c00131d790 0x55c00131fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.921+0000 7f5fdca56640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55c001319280 0x55c001319660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34766/0 (socket says 172.21.15.110:34766) 2024-04-27T03:18:09.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdca56640 1 -- 172.21.15.110:0/3200015009 learned_addr learned my addr 172.21.15.110:0/3200015009 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fd4255640 1 -- 172.21.15.110:0/3200015009 >> v1:172.21.15.77:6789/0 conn(0x55c001319280 legacy=0x55c001319660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fd4255640 1 -- 172.21.15.110:0/3200015009 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0011c6510 con 0x55c00131d790 2024-04-27T03:18:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fd4255640 1 --2- 172.21.15.110:0/3200015009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c00131d790 0x55c00131fb80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400a480 tx=0x7f5fc401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7044496d65885b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fd7fff640 1 -- 172.21.15.110:0/3200015009 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fc40b0070 con 0x55c00131d790 2024-04-27T03:18:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fd7fff640 1 -- 172.21.15.110:0/3200015009 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc40ab540 con 0x55c00131d790 2024-04-27T03:18:09.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fd7fff640 1 -- 172.21.15.110:0/3200015009 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fc40ab840 con 0x55c00131d790 2024-04-27T03:18:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdef3b980 1 -- 172.21.15.110:0/3200015009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c00131d790 msgr2=0x55c00131fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/3200015009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c00131d790 0x55c00131fb80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400a480 tx=0x7f5fc401b280 comp rx=0 tx=0).stop 2024-04-27T03:18:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdef3b980 1 -- 172.21.15.110:0/3200015009 shutdown_connections 2024-04-27T03:18:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/3200015009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c00131d790 0x55c00131fb80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdef3b980 1 -- 172.21.15.110:0/3200015009 >> 172.21.15.110:0/3200015009 conn(0x55c001314b50 msgr2=0x55c001316f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdef3b980 1 -- 172.21.15.110:0/3200015009 shutdown_connections 2024-04-27T03:18:09.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.925+0000 7f5fdef3b980 1 -- 172.21.15.110:0/3200015009 wait complete. 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 Processor -- start 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 -- start start 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c001404550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c00121c2d0 con 0x55c001319280 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdca56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c001404550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdca56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c001404550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51474/0 (socket says 172.21.15.110:51474) 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdca56640 1 -- 172.21.15.110:0/1797708573 learned_addr learned my addr 172.21.15.110:0/1797708573 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdca56640 1 -- 172.21.15.110:0/1797708573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0011a2ae0 con 0x55c001319280 2024-04-27T03:18:09.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdca56640 1 --2- 172.21.15.110:0/1797708573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c001404550 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc001e20 tx=0x7f5fcc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116f72b191e84f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5ecae0d640 1 -- 172.21.15.110:0/1797708573 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc012020 con 0x55c001319280 2024-04-27T03:18:09.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5ecae0d640 1 -- 172.21.15.110:0/1797708573 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fcc00e7c0 con 0x55c001319280 2024-04-27T03:18:09.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5ecae0d640 1 -- 172.21.15.110:0/1797708573 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc00eb00 con 0x55c001319280 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1797708573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 msgr2=0x55c001404550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1797708573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c001404550 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc001e20 tx=0x7f5fcc00b790 comp rx=0 tx=0).stop 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1797708573 shutdown_connections 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1797708573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c001404550 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1797708573 >> 172.21.15.110:0/1797708573 conn(0x55c001314b50 msgr2=0x55c001400040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1797708573 shutdown_connections 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1797708573 wait complete. 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.941+0000 7f5fdef3b980 1 Processor -- start 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 -- start start 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c00128f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c001322bb0 con 0x55c001319280 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdca56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c00128f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdca56640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c00128f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51488/0 (socket says 172.21.15.110:51488) 2024-04-27T03:18:09.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdca56640 1 -- 172.21.15.110:0/1960179091 learned_addr learned my addr 172.21.15.110:0/1960179091 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdca56640 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c00128fc00 con 0x55c001319280 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdca56640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c00128f6c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc001e20 tx=0x7f5fcc000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5ec960a640 1 -- 172.21.15.110:0/1960179091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc012030 con 0x55c001319280 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c00128dec0 con 0x55c001319280 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5ec960a640 1 -- 172.21.15.110:0/1960179091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fcc001480 con 0x55c001319280 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5ec960a640 1 -- 172.21.15.110:0/1960179091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fcc01a940 con 0x55c001319280 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c00128e3b0 con 0x55c001319280 2024-04-27T03:18:09.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5ec960a640 1 -- 172.21.15.110:0/1960179091 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5fcc015020 con 0x55c001319280 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5ec960a640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ec4034650 0x7f5ec4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5ec960a640 1 -- 172.21.15.110:0/1960179091 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5fcc020030 con 0x55c001319280 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fd4255640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ec4034650 0x7f5ec4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c001412aa0 0x55c001414e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c0014153d0 con 0x55c001412aa0 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdd257640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c001412aa0 0x55c001414e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fd4255640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ec4034650 0x7f5ec4036b10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40ad000 tx=0x7f5fc40a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdd257640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c001412aa0 0x55c001414e90 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdd257640 1 -- 172.21.15.110:0/1960179091 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5fd800a4d0 con 0x55c001412aa0 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c001415d10 0x55c0014181a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.945+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c001418910 con 0x55c001415d10 2024-04-27T03:18:09.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdca56640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c001415d10 0x55c0014181a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdca56640 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c001415d10 0x55c0014181a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdca56640 1 -- 172.21.15.110:0/1960179091 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001415d10 2024-04-27T03:18:09.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c00141a1c0 con 0x55c001412aa0 2024-04-27T03:18:09.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdd257640 1 -- 172.21.15.110:0/1960179091 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f5fd800a4d0 con 0x55c001412aa0 2024-04-27T03:18:09.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c0014193f0 con 0x55c001415d10 2024-04-27T03:18:09.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdca56640 1 -- 172.21.15.110:0/1960179091 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001415d10 2024-04-27T03:18:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c0014197e0 con 0x55c001412aa0 2024-04-27T03:18:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdd257640 1 -- 172.21.15.110:0/1960179091 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f5fd800a4d0 con 0x55c001412aa0 2024-04-27T03:18:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdef3b980 1 Processor -- start 2024-04-27T03:18:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdef3b980 1 -- start start 2024-04-27T03:18:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdef3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001428b60 0x55c001428f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.949+0000 7f5fdef3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c001220310 con 0x55c001428b60 2024-04-27T03:18:09.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdd257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001428b60 0x55c001428f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdd257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001428b60 0x55c001428f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51504/0 (socket says 172.21.15.110:51504) 2024-04-27T03:18:09.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdd257640 1 -- 172.21.15.110:0/2895251180 learned_addr learned my addr 172.21.15.110:0/2895251180 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdd257640 1 -- 172.21.15.110:0/2895251180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0014294c0 con 0x55c001428b60 2024-04-27T03:18:09.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdd257640 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001428b60 0x55c001428f80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x55c00128ecd0 tx=0x7f5fd800df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5ec27fc640 1 -- 172.21.15.110:0/2895251180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fd80157d0 con 0x55c001428b60 2024-04-27T03:18:09.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5ec27fc640 1 -- 172.21.15.110:0/2895251180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd8015970 con 0x55c001428b60 2024-04-27T03:18:09.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c001429f60 con 0x55c001428b60 2024-04-27T03:18:09.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5ec27fc640 1 -- 172.21.15.110:0/2895251180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fd8015c50 con 0x55c001428b60 2024-04-27T03:18:09.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c00142a430 con 0x55c001428b60 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5ec27fc640 1 -- 172.21.15.110:0/2895251180 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5fd8020440 con 0x55c001428b60 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5ec27fc640 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5eb40346a0 0x7f5eb4036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5ec27fc640 1 -- 172.21.15.110:0/2895251180 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5fd800e040 con 0x55c001428b60 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdca56640 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5eb40346a0 0x7f5eb4036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdef3b980 1 Processor -- start 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.953+0000 7f5fdef3b980 1 -- start start 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdca56640 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5eb40346a0 0x7f5eb4036b60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55c001418f10 tx=0x7f5fcc048500 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c0014360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c001404ca0 con 0x55c001435ca0 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c0014360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c0014360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51520/0 (socket says 172.21.15.110:51520) 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 -- 172.21.15.110:0/356091900 learned_addr learned my addr 172.21.15.110:0/356091900 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 -- 172.21.15.110:0/356091900 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c001436690 con 0x55c001435ca0 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 --2- 172.21.15.110:0/356091900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c0014360c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5fd804a5d0 tx=0x7f5fd804a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d486e99a7c4b51 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5eb3fff640 1 -- 172.21.15.110:0/356091900 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fd803c070 con 0x55c001435ca0 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5eb3fff640 1 -- 172.21.15.110:0/356091900 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd8012420 con 0x55c001435ca0 2024-04-27T03:18:09.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5eb3fff640 1 -- 172.21.15.110:0/356091900 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fd80133b0 con 0x55c001435ca0 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- 172.21.15.110:0/356091900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 msgr2=0x55c0014360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/356091900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c0014360c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5fd804a5d0 tx=0x7f5fd804a7a0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- 172.21.15.110:0/356091900 shutdown_connections 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/356091900 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c0014360c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- 172.21.15.110:0/356091900 >> 172.21.15.110:0/356091900 conn(0x55c001432030 msgr2=0x55c001430090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- 172.21.15.110:0/356091900 shutdown_connections 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- 172.21.15.110:0/356091900 wait complete. 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 Processor -- start 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- start start 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c00143d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdef3b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c001436de0 con 0x55c001435ca0 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c00143d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c00143d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51526/0 (socket says 172.21.15.110:51526) 2024-04-27T03:18:09.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 learned_addr learned my addr 172.21.15.110:0/589459211 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:09.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c00143d710 con 0x55c001435ca0 2024-04-27T03:18:09.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.957+0000 7f5fdd257640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c00143d1d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5fd801efd0 tx=0x7f5fd804a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5eb27fc640 1 -- 172.21.15.110:0/589459211 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fd8037070 con 0x55c001435ca0 2024-04-27T03:18:09.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c00143d9a0 con 0x55c001435ca0 2024-04-27T03:18:09.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5eb27fc640 1 -- 172.21.15.110:0/589459211 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd803c070 con 0x55c001435ca0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5eb27fc640 1 -- 172.21.15.110:0/589459211 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fd8015df0 con 0x55c001435ca0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c001440680 con 0x55c001435ca0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5eb27fc640 1 -- 172.21.15.110:0/589459211 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5fd8011a60 con 0x55c001435ca0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5eb27fc640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea8034650 0x7f5ea8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5eb27fc640 1 -- 172.21.15.110:0/589459211 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5fd8033cb0 con 0x55c001435ca0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdca56640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea8034650 0x7f5ea8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c0014487f0 0x55c00144ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c00144b3e0 con 0x55c0014487f0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fd4255640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c0014487f0 0x55c00144ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdca56640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea8034650 0x7f5ea8036b10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0422f0 tx=0x7f5fcc035000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fd4255640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c0014487f0 0x55c00144ac90 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 realm 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c00144b810 con 0x55c0014487f0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c00144c0e0 con 0x55c0014487f0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c00144cb00 0x55c00144eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c00144f660 con 0x55c00144cb00 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdd257640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c00144cb00 0x55c00144eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.961+0000 7f5fdd257640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c00144cb00 0x55c00144eef0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdef3b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c00144ff00 con 0x55c0014487f0 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdef3b980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdef3b980 20 searching for the correct realm 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c001451c70 0x55c001454060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c001451c70 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdca56640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c001451c70 0x55c001454060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdca56640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c001451c70 0x55c001454060 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdd257640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5fd8014070 0x7f5fd8020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:09.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x7f5fd8014070 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5fd8014070 0x7f5fd8020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5fd8014070 0x7f5fd8020730 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.965+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c001451c70 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x7f5fd8014070 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x7f5fd8014070 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c001451c70 2024-04-27T03:18:09.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.969+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c001451c70 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c001451c70 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x7f5fd8014070 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c0014487f0 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x7f5fd8014070 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c001451c70 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x7f5fd8014070 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.973+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c001451c70 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014545f0 con 0x55c00144cb00 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:09.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c0014508b0 con 0x55c0014487f0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c001450d00 con 0x55c00144cb00 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.977+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:09.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 zone default found 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 4 Realm: () 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 zonegroup default 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c001454f20 con 0x7f5fd8014070 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c001456930 con 0x55c001451c70 2024-04-27T03:18:09.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c001456e10 con 0x55c001451c70 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 started sync module instance, tier type = 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001455d90 con 0x55c00144cb00 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94283143406272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001455d90 con 0x55c00144cb00 2024-04-27T03:18:09.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.981+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c0014578d0 con 0x55c0014487f0 2024-04-27T03:18:09.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94283143410656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001457d10 con 0x55c0014487f0 2024-04-27T03:18:09.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001458820 con 0x55c00144cb00 2024-04-27T03:18:09.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94283143410656] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94283143406272] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94283143403744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c0014587e0 con 0x55c00144cb00 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c0014590c0 con 0x7f5fd8014070 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94283143416848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c0014587e0 con 0x7f5fd8014070 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.985+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001459b40 con 0x55c00144cb00 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94283143416848] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94283143403744] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94283143414480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001459650 con 0x55c00144cb00 2024-04-27T03:18:09.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x55c001451c70 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94283143421744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001459650 con 0x55c001451c70 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145b250 con 0x55c001451c70 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94283143414480] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94283143421744] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94283143423872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145ada0 con 0x55c001451c70 2024-04-27T03:18:09.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.989+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145b670 con 0x7f5fd8014070 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94283143423872] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'90 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94283143417904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145b250 con 0x7f5fd8014070 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=0 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=1 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=2 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=3 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=4 2024-04-27T03:18:10.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=5 2024-04-27T03:18:10.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=6 2024-04-27T03:18:10.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94283143417904] v27'91 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 add_watcher() i=7 2024-04-27T03:18:10.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:10.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55c00145eb20 con 0x55c0014487f0 2024-04-27T03:18:10.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94283143425904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145c040 con 0x55c0014487f0 2024-04-27T03:18:10.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94283143425904] v27'100 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5e92fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:10.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:10.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.993+0000 7f5fdef3b980 5 note: GC not initialized 2024-04-27T03:18:10.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5e91fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:10.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdef3b980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:10.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdef3b980 20 Filter name: none 2024-04-27T03:18:10.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5e037fe640 20 reqs_thread_entry: start 2024-04-27T03:18:10.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:62 8.0 8:00000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c0014818a0 con 0x55c001451c70 2024-04-27T03:18:10.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(62 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:63 8.10 8:08000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x55c00144cb00 2024-04-27T03:18:10.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(63 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.8 8:10000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001450d00 con 0x55c0014487f0 2024-04-27T03:18:10.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 [pgnls start_epoch 27 out=49b] v27'2 uv2 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 8.18 8:18000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(65 [pgnls start_epoch 27 out=49b] v27'11 uv11 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.4 8:20000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x7f5fd8014070 2024-04-27T03:18:10.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(66 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 8.14 8:28000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c00144cb00 2024-04-27T03:18:10.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 8.c 8:30000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 8.1c 8:38000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(69 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:09.997+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 8.2 8:40000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x7f5fd8014070 2024-04-27T03:18:10.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 [pgnls start_epoch 27 out=127b] v27'3 uv3 ondisk = 1) ==== 144+0+127 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 8.12 8:48000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c00144cb00 2024-04-27T03:18:10.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(71 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 8.a 8:50000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(72 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 8.1a 8:58000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:74 8.6 8:60000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(74 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 8.16 8:68000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(75 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 8.e 8:70000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x7f5fd8014070 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(76 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 8.1e 8:78000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(77 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:78 8.1 8:80000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(78 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:79 8.11 8:88000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c00144cb00 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(79 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:80 8.9 8:90000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x7f5fd8014070 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(80 [pgnls start_epoch 27 out=70b] v27'2 uv2 ondisk = 1) ==== 144+0+70 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 8.19 8:98000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c00144cb00 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(81 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.001+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:82 8.5 8:a0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(82 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:83 8.15 8:a8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(83 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:84 8.d 8:b0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x7f5fd8014070 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(84 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 8.1d 8:b8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(85 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:86 8.3 8:c0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(86 [pgnls start_epoch 27 out=49b] v27'1 uv1 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 8.13 8:c8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 [pgnls start_epoch 27 out=49b] v27'2 uv2 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 8.b 8:d0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:89 8.1b 8:d8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x7f5fd8014070 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(89 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 8.7 8:e0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c0014487f0 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(90 [pgnls start_epoch 27 out=49b] v27'1 uv1 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 8.17 8:e8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c00144cb00 2024-04-27T03:18:10.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(91 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:92 8.f 8:f0000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x55c001451c70 2024-04-27T03:18:10.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(92 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 8.1f 8:f8000000:root:::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55c001451120 con 0x7f5fd8014070 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(93 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.005+0000 7f5fdef3b980 20 RGWRados::pool_iterate: got myfoo 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5e92fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94283143425904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x55c0014487f0 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(94 data_loggenerations_metadata [watch unwatch cookie 94283143425904] v27'101 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94283143406272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c001450d00 con 0x55c00144cb00 2024-04-27T03:18:10.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(95 notify.0 [watch unwatch cookie 94283143406272] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5fdef3b980 20 remove_watcher() i=0 2024-04-27T03:18:10.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5fdef3b980 2 removed watcher, disabling cache 2024-04-27T03:18:10.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.009+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94283143410656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x55c0014487f0 2024-04-27T03:18:10.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(96 notify.1 [watch unwatch cookie 94283143410656] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x55c0014487f0 2024-04-27T03:18:10.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fdef3b980 20 remove_watcher() i=1 2024-04-27T03:18:10.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94283143403744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x55c00144cb00 2024-04-27T03:18:10.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(97 notify.2 [watch unwatch cookie 94283143403744] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fdef3b980 20 remove_watcher() i=2 2024-04-27T03:18:10.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94283143416848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x7f5fd8014070 2024-04-27T03:18:10.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(98 notify.3 [watch unwatch cookie 94283143416848] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fdef3b980 20 remove_watcher() i=3 2024-04-27T03:18:10.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.013+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94283143414480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x55c00144cb00 2024-04-27T03:18:10.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.017+0000 7f5fdd257640 1 -- 172.21.15.110:0/589459211 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(99 notify.4 [watch unwatch cookie 94283143414480] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fd8039030 con 0x55c00144cb00 2024-04-27T03:18:10.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.017+0000 7f5fdef3b980 20 remove_watcher() i=4 2024-04-27T03:18:10.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.017+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94283143421744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00145a3e0 con 0x55c001451c70 2024-04-27T03:18:10.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.017+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(100 notify.5 [watch unwatch cookie 94283143421744] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.017+0000 7f5fdef3b980 20 remove_watcher() i=5 2024-04-27T03:18:10.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.017+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94283143423872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00147fbf0 con 0x55c001451c70 2024-04-27T03:18:10.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdca56640 1 -- 172.21.15.110:0/589459211 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(101 notify.6 [watch unwatch cookie 94283143423872] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fcc008030 con 0x55c001451c70 2024-04-27T03:18:10.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 20 remove_watcher() i=6 2024-04-27T03:18:10.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94283143417904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55c00147fbf0 con 0x7f5fd8014070 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(102 notify.7 [watch unwatch cookie 94283143417904] v27'92 uv90 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5fc40b3040 con 0x7f5fd8014070 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 20 remove_watcher() i=7 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c00144cb00 msgr2=0x55c00144eef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c00144cb00 0x55c00144eef0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c0014487f0 msgr2=0x55c00144ac90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c0014487f0 0x55c00144ac90 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5fd8014070 msgr2=0x7f5fd8020730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5fd8014070 0x7f5fd8020730 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c001451c70 msgr2=0x55c001454060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55c001451c70 0x55c001454060 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea8034650 msgr2=0x7f5ea8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea8034650 0x7f5ea8036b10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0422f0 tx=0x7f5fcc035000 comp rx=0 tx=0).stop 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fd4255640 1 -- 172.21.15.110:0/589459211 reap_dead start 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 msgr2=0x55c00143d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c00143d1d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5fd801efd0 tx=0x7f5fd804a530 comp rx=0 tx=0).stop 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 shutdown_connections 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/589459211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001435ca0 0x55c00143d1d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 >> 172.21.15.110:0/589459211 conn(0x55c001432030 msgr2=0x55c001437500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 shutdown_connections 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/589459211 wait complete. 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5eb40346a0 msgr2=0x7f5eb4036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5eb40346a0 0x7f5eb4036b60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55c001418f10 tx=0x7f5fcc048500 comp rx=0 tx=0).stop 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001428b60 msgr2=0x55c001428f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001428b60 0x55c001428f80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x55c00128ecd0 tx=0x7f5fd800df90 comp rx=0 tx=0).stop 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 shutdown_connections 2024-04-27T03:18:10.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5eb40346a0 0x7f5eb4036b60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/2895251180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001428b60 0x55c001428f80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 >> 172.21.15.110:0/2895251180 conn(0x55c00141d970 msgr2=0x55c00141e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 shutdown_connections 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/2895251180 wait complete. 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c001415d10 msgr2=0x55c0014181a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c001415d10 0x55c0014181a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c001412aa0 msgr2=0x55c001414e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c001412aa0 0x55c001414e90 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ec4034650 msgr2=0x7f5ec4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ec4034650 0x7f5ec4036b10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40ad000 tx=0x7f5fc40a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 msgr2=0x55c00128f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c00128f6c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc001e20 tx=0x7f5fcc000ee0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 shutdown_connections 2024-04-27T03:18:10.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c001415d10 0x55c0014181a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c001412aa0 0x55c001414e90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ec4034650 0x7f5ec4036b10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 --2- 172.21.15.110:0/1960179091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c001319280 0x55c00128f6c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 >> 172.21.15.110:0/1960179091 conn(0x55c001314b50 msgr2=0x55c0013feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 shutdown_connections 2024-04-27T03:18:10.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.021+0000 7f5fdef3b980 1 -- 172.21.15.110:0/1960179091 wait complete. 2024-04-27T03:18:10.038 INFO:teuthology.orchestra.run.smithi110.stdout:["myfoo"] 2024-04-27T03:18:10.039 DEBUG:tasks.util.rgw: json result: ['myfoo'] 2024-04-27T03:18:10.050 INFO:tasks.radosgw_admin:RL: create_bucket myfoo2 foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:10.062 INFO:tasks.radosgw_admin:RL: create_bucket myfoo3 foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:10.072 INFO:tasks.radosgw_admin:RL: create_bucket myfoo4 foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:10.115 INFO:tasks.radosgw_admin:RL: create_bucket myfoo5 foo bytes_out=0 bytes_in=237 failed=True 2024-04-27T03:18:10.180 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo2 foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:10.248 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo3 foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:10.317 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo4 foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:10.317 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--bucket', 'myfoo'] 2024-04-27T03:18:10.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--bucket', 'myfoo'] 2024-04-27T03:18:10.317 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --bucket myfoo 2024-04-27T03:18:10.354 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:10.354 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:10.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.345+0000 7f0682f51980 1 Processor -- start 2024-04-27T03:18:10.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.345+0000 7f0682f51980 1 -- start start 2024-04-27T03:18:10.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.345+0000 7f0682f51980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f568c14620 0x55f568c16a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.345+0000 7f0682f51980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55f568acf0d0 con 0x55f568b84890 2024-04-27T03:18:10.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.345+0000 7f0682f51980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55f568ac4430 con 0x55f568c14620 2024-04-27T03:18:10.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f068026b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f568c14620 0x55f568c16a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f0680a6c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55f568b84890 0x55f568c12f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34770/0 (socket says 172.21.15.110:34770) 2024-04-27T03:18:10.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f0680a6c640 1 -- 172.21.15.110:0/3564577987 learned_addr learned my addr 172.21.15.110:0/3564577987 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:10.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f068026b640 1 -- 172.21.15.110:0/3564577987 >> v1:172.21.15.77:6789/0 conn(0x55f568b84890 legacy=0x55f568c12f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f067fa6a640 1 -- 172.21.15.110:0/3564577987 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 31692576 0 0) 0x55f568acf0d0 con 0x55f568b84890 2024-04-27T03:18:10.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f068026b640 1 -- 172.21.15.110:0/3564577987 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f568abd510 con 0x55f568c14620 2024-04-27T03:18:10.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f068026b640 1 --2- 172.21.15.110:0/3564577987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568c14620 0x55f568c16a10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f066800a480 tx=0x7f066801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becea09f63eedd84 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f067fa6a640 1 -- 172.21.15.110:0/3564577987 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06680b0070 con 0x55f568c14620 2024-04-27T03:18:10.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f067fa6a640 1 -- 172.21.15.110:0/3564577987 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06680ab540 con 0x55f568c14620 2024-04-27T03:18:10.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.349+0000 7f067fa6a640 1 -- 172.21.15.110:0/3564577987 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06680ab840 con 0x55f568c14620 2024-04-27T03:18:10.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.353+0000 7f0682f51980 1 -- 172.21.15.110:0/3564577987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568c14620 msgr2=0x55f568c16a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.353+0000 7f0682f51980 1 --2- 172.21.15.110:0/3564577987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568c14620 0x55f568c16a10 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f066800a480 tx=0x7f066801b280 comp rx=0 tx=0).stop 2024-04-27T03:18:10.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.353+0000 7f0682f51980 1 -- 172.21.15.110:0/3564577987 shutdown_connections 2024-04-27T03:18:10.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.353+0000 7f0682f51980 1 --2- 172.21.15.110:0/3564577987 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568c14620 0x55f568c16a10 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.353+0000 7f0682f51980 1 -- 172.21.15.110:0/3564577987 >> 172.21.15.110:0/3564577987 conn(0x55f568c0aa20 msgr2=0x55f568c0ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.353+0000 7f0682f51980 1 -- 172.21.15.110:0/3564577987 shutdown_connections 2024-04-27T03:18:10.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.353+0000 7f0682f51980 1 -- 172.21.15.110:0/3564577987 wait complete. 2024-04-27T03:18:10.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.365+0000 7f0682f51980 1 Processor -- start 2024-04-27T03:18:10.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.365+0000 7f0682f51980 1 -- start start 2024-04-27T03:18:10.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568c10660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f568b132d0 con 0x55f568b84890 2024-04-27T03:18:10.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568c10660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568c10660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51538/0 (socket says 172.21.15.110:51538) 2024-04-27T03:18:10.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 -- 172.21.15.110:0/3895066410 learned_addr learned my addr 172.21.15.110:0/3895066410 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 -- 172.21.15.110:0/3895066410 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f568a99ae0 con 0x55f568b84890 2024-04-27T03:18:10.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 --2- 172.21.15.110:0/3895066410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568c10660 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f06700016c0 tx=0x7f0670001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2f74fbd897c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f056ee0d640 1 -- 172.21.15.110:0/3895066410 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670003460 con 0x55f568b84890 2024-04-27T03:18:10.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f056ee0d640 1 -- 172.21.15.110:0/3895066410 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0670003600 con 0x55f568b84890 2024-04-27T03:18:10.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f056ee0d640 1 -- 172.21.15.110:0/3895066410 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670003920 con 0x55f568b84890 2024-04-27T03:18:10.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- 172.21.15.110:0/3895066410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 msgr2=0x55f568c10660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 --2- 172.21.15.110:0/3895066410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568c10660 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f06700016c0 tx=0x7f0670001b80 comp rx=0 tx=0).stop 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- 172.21.15.110:0/3895066410 shutdown_connections 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 --2- 172.21.15.110:0/3895066410 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568c10660 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- 172.21.15.110:0/3895066410 >> 172.21.15.110:0/3895066410 conn(0x55f568c0aa20 msgr2=0x55f568cf6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- 172.21.15.110:0/3895066410 shutdown_connections 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- 172.21.15.110:0/3895066410 wait complete. 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 Processor -- start 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- start start 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568d04510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0682f51980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f568c17af0 con 0x55f568b84890 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568d04510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568d04510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51540/0 (socket says 172.21.15.110:51540) 2024-04-27T03:18:10.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.369+0000 7f0680a6c640 1 -- 172.21.15.110:0/3075809533 learned_addr learned my addr 172.21.15.110:0/3075809533 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0680a6c640 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f568d04a50 con 0x55f568b84890 2024-04-27T03:18:10.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0680a6c640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568d04510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f06700012f0 tx=0x7f0670001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f056d60a640 1 -- 172.21.15.110:0/3075809533 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670013630 con 0x55f568b84890 2024-04-27T03:18:10.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f056d60a640 1 -- 172.21.15.110:0/3075809533 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0670023070 con 0x55f568b84890 2024-04-27T03:18:10.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f568d04ce0 con 0x55f568b84890 2024-04-27T03:18:10.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f056d60a640 1 -- 172.21.15.110:0/3075809533 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0670013900 con 0x55f568b84890 2024-04-27T03:18:10.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f568d07910 con 0x55f568b84890 2024-04-27T03:18:10.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f056d60a640 1 -- 172.21.15.110:0/3075809533 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0670010070 con 0x55f568b84890 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f056d60a640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0568034650 0x7f0568036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f056d60a640 1 -- 172.21.15.110:0/3075809533 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f0670049b80 con 0x55f568b84890 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f068026b640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0568034650 0x7f0568036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568c14620 0x55f568d0b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d0b970 con 0x55f568c14620 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f068126d640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568c14620 0x55f568d0b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f068026b640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0568034650 0x7f0568036b10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f06680ad000 tx=0x7f06680a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f068126d640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568c14620 0x55f568d0b3f0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f068126d640 1 -- 172.21.15.110:0/3075809533 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f06780083f0 con 0x55f568c14620 2024-04-27T03:18:10.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d0c720 0x55f568d0eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d0f150 con 0x55f568d0c720 2024-04-27T03:18:10.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.373+0000 7f0680a6c640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d0c720 0x55f568d0eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0680a6c640 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d0c720 0x55f568d0eb10 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0680a6c640 1 -- 172.21.15.110:0/3075809533 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f067000f030 con 0x55f568d0c720 2024-04-27T03:18:10.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d10a00 con 0x55f568c14620 2024-04-27T03:18:10.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f068126d640 1 -- 172.21.15.110:0/3075809533 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f06780083f0 con 0x55f568c14620 2024-04-27T03:18:10.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d0fc30 con 0x55f568d0c720 2024-04-27T03:18:10.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0680a6c640 1 -- 172.21.15.110:0/3075809533 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f067000f030 con 0x55f568d0c720 2024-04-27T03:18:10.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d10020 con 0x55f568c14620 2024-04-27T03:18:10.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f068126d640 1 -- 172.21.15.110:0/3075809533 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f06780083f0 con 0x55f568c14620 2024-04-27T03:18:10.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0682f51980 1 Processor -- start 2024-04-27T03:18:10.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0682f51980 1 -- start start 2024-04-27T03:18:10.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0682f51980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d1f320 0x55f568d1f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f0682f51980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f568b17310 con 0x55f568d1f320 2024-04-27T03:18:10.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f068126d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d1f320 0x55f568d1f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f068126d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d1f320 0x55f568d1f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51544/0 (socket says 172.21.15.110:51544) 2024-04-27T03:18:10.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.377+0000 7f068126d640 1 -- 172.21.15.110:0/2513840917 learned_addr learned my addr 172.21.15.110:0/2513840917 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f068126d640 1 -- 172.21.15.110:0/2513840917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f568d1fc80 con 0x55f568d1f320 2024-04-27T03:18:10.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f068126d640 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d1f320 0x55f568d1f740 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f067800aab0 tx=0x7f067800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f05667fc640 1 -- 172.21.15.110:0/2513840917 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0678007d30 con 0x55f568d1f320 2024-04-27T03:18:10.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f568d20720 con 0x55f568d1f320 2024-04-27T03:18:10.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f05667fc640 1 -- 172.21.15.110:0/2513840917 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067801d070 con 0x55f568d1f320 2024-04-27T03:18:10.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f05667fc640 1 -- 172.21.15.110:0/2513840917 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0678018440 con 0x55f568d1f320 2024-04-27T03:18:10.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f568d20bf0 con 0x55f568d1f320 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f05667fc640 1 -- 172.21.15.110:0/2513840917 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0678013070 con 0x55f568d1f320 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f05667fc640 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0558034650 0x7f0558036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f05667fc640 1 -- 172.21.15.110:0/2513840917 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f067804b520 con 0x55f568d1f320 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0680a6c640 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0558034650 0x7f0558036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0682f51980 1 Processor -- start 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0682f51980 1 -- start start 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0680a6c640 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0558034650 0x7f0558036b10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55f568d0f750 tx=0x7f0670047f20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0682f51980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d2c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f0682f51980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f568c0ed90 con 0x55f568d2c4f0 2024-04-27T03:18:10.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f068126d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d2c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f068126d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d2c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51556/0 (socket says 172.21.15.110:51556) 2024-04-27T03:18:10.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f068126d640 1 -- 172.21.15.110:0/3801103650 learned_addr learned my addr 172.21.15.110:0/3801103650 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.381+0000 7f068126d640 1 -- 172.21.15.110:0/3801103650 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f568d2cee0 con 0x55f568d2c4f0 2024-04-27T03:18:10.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f068126d640 1 --2- 172.21.15.110:0/3801103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d2c910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0678049960 tx=0x7f06780186a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab0e92ade7723b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0557fff640 1 -- 172.21.15.110:0/3801103650 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06780225e0 con 0x55f568d2c4f0 2024-04-27T03:18:10.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0557fff640 1 -- 172.21.15.110:0/3801103650 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0678018bc0 con 0x55f568d2c4f0 2024-04-27T03:18:10.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0557fff640 1 -- 172.21.15.110:0/3801103650 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0678020360 con 0x55f568d2c4f0 2024-04-27T03:18:10.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- 172.21.15.110:0/3801103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 msgr2=0x55f568d2c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 --2- 172.21.15.110:0/3801103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d2c910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0678049960 tx=0x7f06780186a0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- 172.21.15.110:0/3801103650 shutdown_connections 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 --2- 172.21.15.110:0/3801103650 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d2c910 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- 172.21.15.110:0/3801103650 >> 172.21.15.110:0/3801103650 conn(0x55f568d28830 msgr2=0x55f568d26890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- 172.21.15.110:0/3801103650 shutdown_connections 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- 172.21.15.110:0/3801103650 wait complete. 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 Processor -- start 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- start start 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f568d2d600 con 0x55f568d2c4f0 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f068126d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f068126d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51558/0 (socket says 172.21.15.110:51558) 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 learned_addr learned my addr 172.21.15.110:0/188340642 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f568d33ef0 con 0x55f568d2c4f0 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f068126d640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d339b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f06780168b0 tx=0x7f06780498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f05567fc640 1 -- 172.21.15.110:0/188340642 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0678031070 con 0x55f568d2c4f0 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f05567fc640 1 -- 172.21.15.110:0/188340642 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0678016e00 con 0x55f568d2c4f0 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f568d34180 con 0x55f568d2c4f0 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f05567fc640 1 -- 172.21.15.110:0/188340642 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0678020a50 con 0x55f568d2c4f0 2024-04-27T03:18:10.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.385+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f568d36e60 con 0x55f568d2c4f0 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f05567fc640 1 -- 172.21.15.110:0/188340642 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0678034070 con 0x55f568d2c4f0 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f05567fc640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f054c034600 0x7f054c036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f05567fc640 1 -- 172.21.15.110:0/188340642 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f0678037c90 con 0x55f568d2c4f0 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0680a6c640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f054c034600 0x7f054c036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568d3f010 0x55f568d41480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d41bd0 con 0x55f568d3f010 2024-04-27T03:18:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068026b640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568d3f010 0x55f568d41480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0680a6c640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f054c034600 0x7f054c036ac0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f06700066b0 tx=0x7f0670034000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068026b640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568d3f010 0x55f568d41480 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 realm 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d42000 con 0x55f568d3f010 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d428d0 con 0x55f568d3f010 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d432f0 0x55f568d456e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d45e50 con 0x55f568d432f0 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068126d640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d432f0 0x55f568d456e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068126d640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d432f0 0x55f568d456e0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.389+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d466f0 con 0x55f568d3f010 2024-04-27T03:18:10.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:10.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0682f51980 20 searching for the correct realm 2024-04-27T03:18:10.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f568d48460 0x55f568d4a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d48460 2024-04-27T03:18:10.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0680a6c640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f568d48460 0x55f568d4a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0680a6c640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f568d48460 0x55f568d4a850 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068126d640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0678036070 0x7f0678021700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x7f0678036070 2024-04-27T03:18:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0678036070 0x7f0678021700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0678036070 0x7f0678021700 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.393+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d48460 2024-04-27T03:18:10.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x7f0678036070 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x7f0678036070 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.397+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d48460 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d48460 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d48460 2024-04-27T03:18:10.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x7f0678036070 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d3f010 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x7f0678036070 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d48460 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.401+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x7f0678036070 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d48460 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4ade0 con 0x55f568d432f0 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d470a0 con 0x55f568d3f010 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.405+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d474c0 con 0x55f568d432f0 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d47920 con 0x55f568d3f010 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 zone default found 2024-04-27T03:18:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 4 Realm: () 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 zonegroup default 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4b260 con 0x7f0678036070 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d46ce0 con 0x55f568d48460 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4d270 con 0x55f568d48460 2024-04-27T03:18:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 started sync module instance, tier type = 2024-04-27T03:18:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.409+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4c580 con 0x55f568d432f0 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94512514124144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4db80 con 0x55f568d432f0 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4e3d0 con 0x55f568d3f010 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94512514128768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4e550 con 0x55f568d3f010 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4f070 con 0x55f568d432f0 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94512514124144] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.413+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94512514128768] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94512514132800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4f030 con 0x55f568d432f0 2024-04-27T03:18:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4f910 con 0x7f0678036070 2024-04-27T03:18:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94512514135136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4f030 con 0x7f0678036070 2024-04-27T03:18:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d503a0 con 0x55f568d432f0 2024-04-27T03:18:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94512514132800] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94512514135136] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94512514136192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d503a0 con 0x55f568d432f0 2024-04-27T03:18:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.417+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d50790 con 0x55f568d48460 2024-04-27T03:18:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94512514127568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d50fe0 con 0x55f568d48460 2024-04-27T03:18:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d519a0 con 0x55f568d48460 2024-04-27T03:18:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94512514136192] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94512514127568] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94512514137808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d519a0 con 0x55f568d48460 2024-04-27T03:18:10.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d52240 con 0x7f0678036070 2024-04-27T03:18:10.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'93 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94512514137808] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94512514143952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d52240 con 0x7f0678036070 2024-04-27T03:18:10.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 20 add_watcher() i=0 2024-04-27T03:18:10.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 20 add_watcher() i=1 2024-04-27T03:18:10.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 20 add_watcher() i=2 2024-04-27T03:18:10.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 20 add_watcher() i=3 2024-04-27T03:18:10.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.421+0000 7f0682f51980 20 add_watcher() i=4 2024-04-27T03:18:10.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f0682f51980 20 add_watcher() i=5 2024-04-27T03:18:10.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f0682f51980 20 add_watcher() i=6 2024-04-27T03:18:10.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94512514143952] v27'94 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f0682f51980 20 add_watcher() i=7 2024-04-27T03:18:10.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f0682f51980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:10.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d556f0 con 0x55f568d3f010 2024-04-27T03:18:10.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94512514146144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d52c10 con 0x55f568d3f010 2024-04-27T03:18:10.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.425+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94512514146144] v27'102 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0536fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:10.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:10.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 5 note: GC not initialized 2024-04-27T03:18:10.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0535fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:10.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:10.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 Filter name: none 2024-04-27T03:18:10.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f04a77fe640 20 reqs_thread_entry: start 2024-04-27T03:18:10.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d78e10 con 0x7f0678036070 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d79740 con 0x7f0678036070 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d79780 con 0x7f0678036070 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(64 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d7ae60 con 0x7f0678036070 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(65 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:66 4.11 4:8a36b39c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d7a4c0 con 0x55f568d48460 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 4.39 4:9c046854:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d7f9f0 con 0x55f568d48460 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 4.15 4:aaf269a6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d7a500 con 0x55f568d48460 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 4.24 4:241b82a3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d84830 con 0x55f568d3f010 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 4.e 4:709f8441:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d874a0 con 0x7f0678036070 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 4.3d 4:bc3f4761:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d8a120 con 0x55f568d432f0 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 4.8 4:10a25087:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d8cfe0 con 0x55f568d48460 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 4.1 4:836fd4f5:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d8f9c0 con 0x7f0678036070 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:74 4.2e 4:773ce9c4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d924a0 con 0x55f568d48460 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(66 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(68 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(67 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:75 4.1 4:80e658b2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d92a60 con 0x7f0678036070 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 4.b 4:d2da04c3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f568d7d9d0 con 0x7f0678036070 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(72 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(74 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.429+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(76 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(71 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(73 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(70 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(75 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f0536fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:77 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94512514146144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d7ad70 con 0x55f568d3f010 2024-04-27T03:18:10.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(77 data_loggenerations_metadata [watch unwatch cookie 94512514146144] v27'103 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.433+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94512514124144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d555b0 con 0x55f568d432f0 2024-04-27T03:18:10.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.437+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(78 notify.0 [watch unwatch cookie 94512514124144] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.437+0000 7f0682f51980 20 remove_watcher() i=0 2024-04-27T03:18:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.437+0000 7f0682f51980 2 removed watcher, disabling cache 2024-04-27T03:18:10.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.437+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94512514128768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d555b0 con 0x55f568d3f010 2024-04-27T03:18:10.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.437+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(79 notify.1 [watch unwatch cookie 94512514128768] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x55f568d3f010 2024-04-27T03:18:10.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.437+0000 7f0682f51980 20 remove_watcher() i=1 2024-04-27T03:18:10.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.437+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94512514132800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4db80 con 0x55f568d432f0 2024-04-27T03:18:10.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.441+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(80 notify.2 [watch unwatch cookie 94512514132800] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.441+0000 7f0682f51980 20 remove_watcher() i=2 2024-04-27T03:18:10.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.441+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:81 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94512514135136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d555b0 con 0x7f0678036070 2024-04-27T03:18:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.441+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(81 notify.3 [watch unwatch cookie 94512514135136] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.441+0000 7f0682f51980 20 remove_watcher() i=3 2024-04-27T03:18:10.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.441+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94512514136192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4eb40 con 0x55f568d432f0 2024-04-27T03:18:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f068126d640 1 -- 172.21.15.110:0/188340642 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(82 notify.4 [watch unwatch cookie 94512514136192] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0678033030 con 0x55f568d432f0 2024-04-27T03:18:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0682f51980 20 remove_watcher() i=4 2024-04-27T03:18:10.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:83 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94512514127568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d555b0 con 0x55f568d48460 2024-04-27T03:18:10.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(83 notify.5 [watch unwatch cookie 94512514127568] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0682f51980 20 remove_watcher() i=5 2024-04-27T03:18:10.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94512514137808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4eb40 con 0x55f568d48460 2024-04-27T03:18:10.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0680a6c640 1 -- 172.21.15.110:0/188340642 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(84 notify.6 [watch unwatch cookie 94512514137808] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0670053030 con 0x55f568d48460 2024-04-27T03:18:10.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0682f51980 20 remove_watcher() i=6 2024-04-27T03:18:10.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.445+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:85 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94512514143952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f568d4d6d0 con 0x7f0678036070 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(85 notify.7 [watch unwatch cookie 94512514143952] v27'95 uv93 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f06680b3040 con 0x7f0678036070 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 20 remove_watcher() i=7 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d432f0 msgr2=0x55f568d456e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d432f0 0x55f568d456e0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568d3f010 msgr2=0x55f568d41480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568d3f010 0x55f568d41480 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0678036070 msgr2=0x7f0678021700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0678036070 0x7f0678021700 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f568d48460 msgr2=0x55f568d4a850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f568d48460 0x55f568d4a850 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f054c034600 msgr2=0x7f054c036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f054c034600 0x7f054c036ac0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f06700066b0 tx=0x7f0670034000 comp rx=0 tx=0).stop 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 msgr2=0x55f568d339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f068026b640 1 -- 172.21.15.110:0/188340642 reap_dead start 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d339b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f06780168b0 tx=0x7f06780498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 shutdown_connections 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/188340642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d2c4f0 0x55f568d339b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 >> 172.21.15.110:0/188340642 conn(0x55f568d28830 msgr2=0x55f568d25e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 shutdown_connections 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/188340642 wait complete. 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0558034650 msgr2=0x7f0558036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0558034650 0x7f0558036b10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55f568d0f750 tx=0x7f0670047f20 comp rx=0 tx=0).stop 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d1f320 msgr2=0x55f568d1f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d1f320 0x55f568d1f740 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f067800aab0 tx=0x7f067800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 shutdown_connections 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0558034650 0x7f0558036b10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/2513840917 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568d1f320 0x55f568d1f740 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 >> 172.21.15.110:0/2513840917 conn(0x55f568d14130 msgr2=0x55f568d14c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 shutdown_connections 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/2513840917 wait complete. 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d0c720 msgr2=0x55f568d0eb10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d0c720 0x55f568d0eb10 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568c14620 msgr2=0x55f568d0b3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568c14620 0x55f568d0b3f0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0568034650 msgr2=0x7f0568036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0568034650 0x7f0568036b10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f06680ad000 tx=0x7f06680a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:10.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 msgr2=0x55f568d04510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568d04510 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f06700012f0 tx=0x7f0670001260 comp rx=0 tx=0).stop 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 shutdown_connections 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f568d0c720 0x55f568d0eb10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f568c14620 0x55f568d0b3f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0568034650 0x7f0568036b10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 --2- 172.21.15.110:0/3075809533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f568b84890 0x55f568d04510 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 >> 172.21.15.110:0/3075809533 conn(0x55f568c0aa20 msgr2=0x55f568c0f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 shutdown_connections 2024-04-27T03:18:10.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.449+0000 7f0682f51980 1 -- 172.21.15.110:0/3075809533 wait complete. 2024-04-27T03:18:10.468 INFO:teuthology.orchestra.run.smithi110.stdout:{"bucket":"myfoo","tenant":"","versioning":"off","zonegroup":"3fb181a9-2f1b-4a6e-85a1-ca8305b0483c","placement_rule":"default-placement","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""},"id":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","marker":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","index_type":"Normal","index_generation":0,"num_shards":11,"object_lock_enabled":false,"mfa_enabled":false,"owner":"foo","ver":"0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1","master_ver":"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0","mtime":"2024-04-27T03:18:09.699113Z","creation_time":"2024-04-27T03:18:09.694744Z","max_marker":"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#","usage":{},"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}} 2024-04-27T03:18:10.468 DEBUG:tasks.util.rgw: json result: {'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:18:09.699113Z', 'creation_time': '2024-04-27T03:18:09.694744Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2024-04-27T03:18:10.468 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--uid', 'foo'] 2024-04-27T03:18:10.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--uid', 'foo'] 2024-04-27T03:18:10.468 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --uid foo 2024-04-27T03:18:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:10.546 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.537+0000 7f99319c1980 1 Processor -- start 2024-04-27T03:18:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.537+0000 7f99319c1980 1 -- start start 2024-04-27T03:18:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.537+0000 7f99319c1980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55883cca79c0 0x55883cca9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.537+0000 7f99319c1980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55883cb600d0 con 0x55883cca34b0 2024-04-27T03:18:10.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f99319c1980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55883cb55430 con 0x55883cca79c0 2024-04-27T03:18:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992ecdb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55883cca79c0 0x55883cca9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992ecdb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55883cca79c0 0x55883cca9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51572/0 (socket says 172.21.15.110:51572) 2024-04-27T03:18:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992f4dc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55883cca34b0 0x55883cca3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34784/0 (socket says 172.21.15.110:34784) 2024-04-27T03:18:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992ecdb640 1 -- 172.21.15.110:0/1407457194 learned_addr learned my addr 172.21.15.110:0/1407457194 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992ecdb640 1 -- 172.21.15.110:0/1407457194 >> v1:172.21.15.77:6789/0 conn(0x55883cca34b0 legacy=0x55883cca3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992ecdb640 1 -- 172.21.15.110:0/1407457194 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55883cb4e510 con 0x55883cca79c0 2024-04-27T03:18:10.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992ecdb640 1 --2- 172.21.15.110:0/1407457194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca79c0 0x55883cca9db0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f991800a5c0 tx=0x7f991801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72b30246c22574f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992e4da640 1 -- 172.21.15.110:0/1407457194 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99180a6070 con 0x55883cca79c0 2024-04-27T03:18:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992e4da640 1 -- 172.21.15.110:0/1407457194 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99180af070 con 0x55883cca79c0 2024-04-27T03:18:10.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.541+0000 7f992e4da640 1 -- 172.21.15.110:0/1407457194 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99180aa5c0 con 0x55883cca79c0 2024-04-27T03:18:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.545+0000 7f99319c1980 1 -- 172.21.15.110:0/1407457194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca79c0 msgr2=0x55883cca9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.545+0000 7f99319c1980 1 --2- 172.21.15.110:0/1407457194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca79c0 0x55883cca9db0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f991800a5c0 tx=0x7f991801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.545+0000 7f99319c1980 1 -- 172.21.15.110:0/1407457194 shutdown_connections 2024-04-27T03:18:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.545+0000 7f99319c1980 1 --2- 172.21.15.110:0/1407457194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca79c0 0x55883cca9db0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.545+0000 7f99319c1980 1 -- 172.21.15.110:0/1407457194 >> 172.21.15.110:0/1407457194 conn(0x55883cc16b90 msgr2=0x55883cc16fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.545+0000 7f99319c1980 1 -- 172.21.15.110:0/1407457194 shutdown_connections 2024-04-27T03:18:10.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.545+0000 7f99319c1980 1 -- 172.21.15.110:0/1407457194 wait complete. 2024-04-27T03:18:10.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.557+0000 7f99319c1980 1 Processor -- start 2024-04-27T03:18:10.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.557+0000 7f99319c1980 1 -- start start 2024-04-27T03:18:10.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.557+0000 7f99319c1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd8e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.557+0000 7f99319c1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55883cba42d0 con 0x55883cca34b0 2024-04-27T03:18:10.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.557+0000 7f992f4dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd8e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd8e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51578/0 (socket says 172.21.15.110:51578) 2024-04-27T03:18:10.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 -- 172.21.15.110:0/3849325924 learned_addr learned my addr 172.21.15.110:0/3849325924 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 -- 172.21.15.110:0/3849325924 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55883cb2aae0 con 0x55883cca34b0 2024-04-27T03:18:10.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 --2- 172.21.15.110:0/3849325924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd8e5d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f99200015f0 tx=0x7f9920001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6e5bb51cd5c653 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f981d60a640 1 -- 172.21.15.110:0/3849325924 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9920013420 con 0x55883cca34b0 2024-04-27T03:18:10.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f981d60a640 1 -- 172.21.15.110:0/3849325924 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99200135e0 con 0x55883cca34b0 2024-04-27T03:18:10.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f981d60a640 1 -- 172.21.15.110:0/3849325924 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9920013940 con 0x55883cca34b0 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 -- 172.21.15.110:0/3849325924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 msgr2=0x55883cd8e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 --2- 172.21.15.110:0/3849325924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd8e5d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f99200015f0 tx=0x7f9920001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 -- 172.21.15.110:0/3849325924 shutdown_connections 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 --2- 172.21.15.110:0/3849325924 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd8e5d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 -- 172.21.15.110:0/3849325924 >> 172.21.15.110:0/3849325924 conn(0x55883cc16b90 msgr2=0x55883cd8a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 -- 172.21.15.110:0/3849325924 shutdown_connections 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 -- 172.21.15.110:0/3849325924 wait complete. 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 Processor -- start 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 -- start start 2024-04-27T03:18:10.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd95610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f99319c1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55883ccacde0 con 0x55883cca34b0 2024-04-27T03:18:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd95610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd95610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51594/0 (socket says 172.21.15.110:51594) 2024-04-27T03:18:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 -- 172.21.15.110:0/1655490894 learned_addr learned my addr 172.21.15.110:0/1655490894 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55883cd95b50 con 0x55883cca34b0 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f992f4dc640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd95610 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9920006640 tx=0x7f9920000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.561+0000 7f981be07640 1 -- 172.21.15.110:0/1655490894 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9920022020 con 0x55883cca34b0 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55883cd95de0 con 0x55883cca34b0 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f981be07640 1 -- 172.21.15.110:0/1655490894 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9920006320 con 0x55883cca34b0 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f981be07640 1 -- 172.21.15.110:0/1655490894 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9920004af0 con 0x55883cca34b0 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55883cd98a60 con 0x55883cca34b0 2024-04-27T03:18:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f981be07640 1 -- 172.21.15.110:0/1655490894 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9920025020 con 0x55883cca34b0 2024-04-27T03:18:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f981be07640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9814034650 0x7f9814036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f981be07640 1 -- 172.21.15.110:0/1655490894 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9920018030 con 0x55883cca34b0 2024-04-27T03:18:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f992ecdb640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9814034650 0x7f9814036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cca79c0 0x55883cd9c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cd9cb60 con 0x55883cca79c0 2024-04-27T03:18:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f992fcdd640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cca79c0 0x55883cd9c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f992ecdb640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9814034650 0x7f9814036b10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f99180ae000 tx=0x7f99180a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f992fcdd640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cca79c0 0x55883cd9c540 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f992fcdd640 1 -- 172.21.15.110:0/1655490894 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f99280083f0 con 0x55883cca79c0 2024-04-27T03:18:10.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cd9d950 0x55883cd9fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cda0370 con 0x55883cd9d950 2024-04-27T03:18:10.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f992f4dc640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cd9d950 0x55883cd9fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.565+0000 7f992f4dc640 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cd9d950 0x55883cd9fd40 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992f4dc640 1 -- 172.21.15.110:0/1655490894 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f992001b070 con 0x55883cd9d950 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cda1c20 con 0x55883cca79c0 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992fcdd640 1 -- 172.21.15.110:0/1655490894 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f99280083f0 con 0x55883cca79c0 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cda0e50 con 0x55883cd9d950 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992f4dc640 1 -- 172.21.15.110:0/1655490894 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f992001b070 con 0x55883cd9d950 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cda1240 con 0x55883cca79c0 2024-04-27T03:18:10.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992fcdd640 1 -- 172.21.15.110:0/1655490894 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f99280083f0 con 0x55883cca79c0 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f99319c1980 1 Processor -- start 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f99319c1980 1 -- start start 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f99319c1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdb0520 0x55883cdb0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f99319c1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55883cba8310 con 0x55883cdb0520 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992fcdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdb0520 0x55883cdb0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992fcdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdb0520 0x55883cdb0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51604/0 (socket says 172.21.15.110:51604) 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992fcdd640 1 -- 172.21.15.110:0/3379681829 learned_addr learned my addr 172.21.15.110:0/3379681829 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992fcdd640 1 -- 172.21.15.110:0/3379681829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55883cdb0e80 con 0x55883cdb0520 2024-04-27T03:18:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f992fcdd640 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdb0520 0x55883cdb0940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f992800aab0 tx=0x7f992800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.569+0000 7f9819602640 1 -- 172.21.15.110:0/3379681829 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9928007d30 con 0x55883cdb0520 2024-04-27T03:18:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9819602640 1 -- 172.21.15.110:0/3379681829 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f992801d070 con 0x55883cdb0520 2024-04-27T03:18:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9819602640 1 -- 172.21.15.110:0/3379681829 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9928018440 con 0x55883cdb0520 2024-04-27T03:18:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55883cdb1920 con 0x55883cdb0520 2024-04-27T03:18:10.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55883cdb1df0 con 0x55883cdb0520 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9819602640 1 -- 172.21.15.110:0/3379681829 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9928013070 con 0x55883cdb0520 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9819602640 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9808034650 0x7f9808036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9819602640 1 -- 172.21.15.110:0/3379681829 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f992804b520 con 0x55883cdb0520 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f992f4dc640 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9808034650 0x7f9808036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f99319c1980 1 Processor -- start 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f99319c1980 1 -- start start 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f992f4dc640 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9808034650 0x7f9808036b10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55883cda0970 tx=0x7f992001d720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f99319c1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdbdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f99319c1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55883cd8ed20 con 0x55883cdbd720 2024-04-27T03:18:10.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f992fcdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdbdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f992fcdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdbdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51620/0 (socket says 172.21.15.110:51620) 2024-04-27T03:18:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f992fcdd640 1 -- 172.21.15.110:0/2209775572 learned_addr learned my addr 172.21.15.110:0/2209775572 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f992fcdd640 1 -- 172.21.15.110:0/2209775572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55883cdbe110 con 0x55883cdbd720 2024-04-27T03:18:10.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f992fcdd640 1 --2- 172.21.15.110:0/2209775572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdbdb40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9928049930 tx=0x7f9928018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a7b8d39812ffcb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9812ffd640 1 -- 172.21.15.110:0/2209775572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9928047070 con 0x55883cdbd720 2024-04-27T03:18:10.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9812ffd640 1 -- 172.21.15.110:0/2209775572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9928018bc0 con 0x55883cdbd720 2024-04-27T03:18:10.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.573+0000 7f9812ffd640 1 -- 172.21.15.110:0/2209775572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9928020360 con 0x55883cdbd720 2024-04-27T03:18:10.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- 172.21.15.110:0/2209775572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 msgr2=0x55883cdbdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 --2- 172.21.15.110:0/2209775572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdbdb40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9928049930 tx=0x7f9928018950 comp rx=0 tx=0).stop 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- 172.21.15.110:0/2209775572 shutdown_connections 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 --2- 172.21.15.110:0/2209775572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdbdb40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- 172.21.15.110:0/2209775572 >> 172.21.15.110:0/2209775572 conn(0x55883cdb9a60 msgr2=0x55883cdb7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- 172.21.15.110:0/2209775572 shutdown_connections 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- 172.21.15.110:0/2209775572 wait complete. 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 Processor -- start 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- start start 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdc4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55883cdbe830 con 0x55883cdbd720 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f992fcdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdc4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f992fcdd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdc4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51636/0 (socket says 172.21.15.110:51636) 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 learned_addr learned my addr 172.21.15.110:0/3850347730 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55883cdc5120 con 0x55883cdbd720 2024-04-27T03:18:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f992fcdd640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdc4be0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f99280168b0 tx=0x7f99280498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f98117fa640 1 -- 172.21.15.110:0/3850347730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9928031070 con 0x55883cdbd720 2024-04-27T03:18:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55883cdc53b0 con 0x55883cdbd720 2024-04-27T03:18:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55883cdc8090 con 0x55883cdbd720 2024-04-27T03:18:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f98117fa640 1 -- 172.21.15.110:0/3850347730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9928020900 con 0x55883cdbd720 2024-04-27T03:18:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f98117fa640 1 -- 172.21.15.110:0/3850347730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9928049a70 con 0x55883cdbd720 2024-04-27T03:18:10.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f98117fa640 1 -- 172.21.15.110:0/3850347730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9928034070 con 0x55883cdbd720 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f98117fa640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f8034650 0x7f97f8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.577+0000 7f98117fa640 1 -- 172.21.15.110:0/3850347730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f992800c0b0 con 0x55883cdbd720 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992f4dc640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f8034650 0x7f97f8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cdd0260 0x55883cdd2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd2e70 con 0x55883cdd0260 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992ecdb640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cdd0260 0x55883cdd2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992f4dc640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f8034650 0x7f97f8036b10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9920014fd0 tx=0x7f992003d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992ecdb640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cdd0260 0x55883cdd2720 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 realm 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd32a0 con 0x55883cdd0260 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd3b40 con 0x55883cdd0260 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cdd4560 0x55883cdd6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd70e0 con 0x55883cdd4560 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992fcdd640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cdd4560 0x55883cdd6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992fcdd640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cdd4560 0x55883cdd6970 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd7980 con 0x55883cdd0260 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 20 searching for the correct realm 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55883cdd96f0 0x55883cddbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd96f0 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.581+0000 7f992f4dc640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55883cdd96f0 0x55883cddbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992f4dc640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55883cdd96f0 0x55883cddbae0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992fcdd640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9928036070 0x7f9928021700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x7f9928036070 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9928036070 0x7f9928021700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9928036070 0x7f9928021700 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd96f0 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.585+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x7f9928036070 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x7f9928036070 2024-04-27T03:18:10.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd96f0 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.589+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd96f0 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd96f0 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x7f9928036070 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd0260 2024-04-27T03:18:10.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x7f9928036070 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd96f0 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x7f9928036070 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.593+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd96f0 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc0c0 con 0x55883cdd4560 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:10.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd8300 con 0x55883cdd0260 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd8780 con 0x55883cdd4560 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdd8be0 con 0x55883cdd0260 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 zone default found 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 4 Realm: () 2024-04-27T03:18:10.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 zonegroup default 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cddc9d0 con 0x7f9928036070 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.597+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cdde780 con 0x55883cdd96f0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cddec10 con 0x55883cdd96f0 2024-04-27T03:18:10.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 20 started sync module instance, tier type = 2024-04-27T03:18:10.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:10.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cdddbf0 con 0x55883cdd4560 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94043625086944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cddf520 con 0x55883cdd4560 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cddfd70 con 0x55883cdd0260 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94043625092384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cddff30 con 0x55883cdd0260 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.601+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde0a50 con 0x55883cdd4560 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94043625086944] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94043625092384] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94043625096480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde0a10 con 0x55883cdd4560 2024-04-27T03:18:10.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde12f0 con 0x7f9928036070 2024-04-27T03:18:10.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94043625098816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde0a10 con 0x7f9928036070 2024-04-27T03:18:10.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.605+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde1da0 con 0x55883cdd4560 2024-04-27T03:18:10.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94043625098816] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94043625096480] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94043625091184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde1da0 con 0x55883cdd4560 2024-04-27T03:18:10.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde2190 con 0x55883cdd96f0 2024-04-27T03:18:10.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94043625099872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde29e0 con 0x55883cdd96f0 2024-04-27T03:18:10.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde33e0 con 0x55883cdd96f0 2024-04-27T03:18:10.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.609+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94043625091184] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94043625101488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde37d0 con 0x55883cdd96f0 2024-04-27T03:18:10.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94043625099872] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde41d0 con 0x7f9928036070 2024-04-27T03:18:10.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94043625101488] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'96 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94043625107184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde3ce0 con 0x7f9928036070 2024-04-27T03:18:10.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=0 2024-04-27T03:18:10.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=1 2024-04-27T03:18:10.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=2 2024-04-27T03:18:10.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=3 2024-04-27T03:18:10.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=4 2024-04-27T03:18:10.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=5 2024-04-27T03:18:10.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=6 2024-04-27T03:18:10.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94043625107184] v27'97 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 20 add_watcher() i=7 2024-04-27T03:18:10.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:10.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883cde41f0 con 0x55883cdd0260 2024-04-27T03:18:10.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.613+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94043625104688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883cde4a50 con 0x55883cdd0260 2024-04-27T03:18:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94043625104688] v27'104 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f97e5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 5 note: GC not initialized 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f97e4fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 Filter name: none 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f975e7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce0a7d0 con 0x55883cdd0260 2024-04-27T03:18:10.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:10.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce0b1c0 con 0x55883cdd0260 2024-04-27T03:18:10.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:10.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce0c1d0 con 0x7f9928036070 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(64 foo.buckets [call out=183b] v0'0 uv14 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.617+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce09920 con 0x7f9928036070 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(65 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce0c190 con 0x7f9928036070 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(66 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 4.11 4:8a36b39c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce0bb70 con 0x55883cdd96f0 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 4.39 4:9c046854:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce10bf0 con 0x55883cdd96f0 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 4.15 4:aaf269a6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce134e0 con 0x55883cdd96f0 2024-04-27T03:18:10.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 4.24 4:241b82a3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce161b0 con 0x55883cdd0260 2024-04-27T03:18:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 4.e 4:709f8441:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce10bf0 con 0x7f9928036070 2024-04-27T03:18:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 4.3d 4:bc3f4761:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce1b030 con 0x55883cdd4560 2024-04-27T03:18:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 4.8 4:10a25087:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce1d500 con 0x55883cdd96f0 2024-04-27T03:18:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 4.1 4:836fd4f5:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce20020 con 0x7f9928036070 2024-04-27T03:18:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 4.2e 4:773ce9c4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce22f00 con 0x55883cdd96f0 2024-04-27T03:18:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 4.1 4:80e658b2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce257f0 con 0x7f9928036070 2024-04-27T03:18:10.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(70 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f99180aaa20 con 0x55883cdd0260 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(71 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:77 4.b 4:d2da04c3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55883ce22f40 con 0x7f9928036070 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(67 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(69 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(74 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f99180aaa20 con 0x7f9928036070 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(68 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(75 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(76 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f99180ee060 con 0x7f9928036070 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(72 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(77 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f99180ee060 con 0x7f9928036070 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f97e5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.621+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:78 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94043625104688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x55883cdd0260 2024-04-27T03:18:10.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.625+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(78 data_loggenerations_metadata [watch unwatch cookie 94043625104688] v27'105 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f99180ee060 con 0x55883cdd0260 2024-04-27T03:18:10.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.625+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:79 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94043625086944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x55883cdd4560 2024-04-27T03:18:10.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.625+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(79 notify.0 [watch unwatch cookie 94043625086944] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.625+0000 7f99319c1980 20 remove_watcher() i=0 2024-04-27T03:18:10.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.625+0000 7f99319c1980 2 removed watcher, disabling cache 2024-04-27T03:18:10.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.625+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:80 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94043625092384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x55883cdd0260 2024-04-27T03:18:10.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.629+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(80 notify.1 [watch unwatch cookie 94043625092384] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180ee060 con 0x55883cdd0260 2024-04-27T03:18:10.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.629+0000 7f99319c1980 20 remove_watcher() i=1 2024-04-27T03:18:10.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.629+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94043625096480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x55883cdd4560 2024-04-27T03:18:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.629+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(81 notify.2 [watch unwatch cookie 94043625096480] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.629+0000 7f99319c1980 20 remove_watcher() i=2 2024-04-27T03:18:10.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.629+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:82 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94043625098816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x7f9928036070 2024-04-27T03:18:10.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(82 notify.3 [watch unwatch cookie 94043625098816] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180ee060 con 0x7f9928036070 2024-04-27T03:18:10.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f99319c1980 20 remove_watcher() i=3 2024-04-27T03:18:10.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94043625091184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x55883cdd4560 2024-04-27T03:18:10.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f992fcdd640 1 -- 172.21.15.110:0/3850347730 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(83 notify.4 [watch unwatch cookie 94043625091184] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9928033030 con 0x55883cdd4560 2024-04-27T03:18:10.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f99319c1980 20 remove_watcher() i=4 2024-04-27T03:18:10.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94043625099872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x55883cdd96f0 2024-04-27T03:18:10.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(84 notify.5 [watch unwatch cookie 94043625099872] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f99319c1980 20 remove_watcher() i=5 2024-04-27T03:18:10.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.633+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:85 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94043625101488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x55883cdd96f0 2024-04-27T03:18:10.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f992f4dc640 1 -- 172.21.15.110:0/3850347730 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(85 notify.6 [watch unwatch cookie 94043625101488] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f992001b070 con 0x55883cdd96f0 2024-04-27T03:18:10.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 20 remove_watcher() i=6 2024-04-27T03:18:10.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:86 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94043625107184] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55883ce08560 con 0x7f9928036070 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(86 notify.7 [watch unwatch cookie 94043625107184] v27'98 uv96 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99180ee060 con 0x7f9928036070 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 20 remove_watcher() i=7 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cdd4560 msgr2=0x55883cdd6970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cdd4560 0x55883cdd6970 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cdd0260 msgr2=0x55883cdd2720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cdd0260 0x55883cdd2720 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9928036070 msgr2=0x7f9928021700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9928036070 0x7f9928021700 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55883cdd96f0 msgr2=0x55883cddbae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55883cdd96f0 0x55883cddbae0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f8034650 msgr2=0x7f97f8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f8034650 0x7f97f8036b10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9920014fd0 tx=0x7f992003d000 comp rx=0 tx=0).stop 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 msgr2=0x55883cdc4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdc4be0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f99280168b0 tx=0x7f99280498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f992ecdb640 1 -- 172.21.15.110:0/3850347730 reap_dead start 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 shutdown_connections 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3850347730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdbd720 0x55883cdc4be0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 >> 172.21.15.110:0/3850347730 conn(0x55883cdb9a60 msgr2=0x55883cdb70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 shutdown_connections 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3850347730 wait complete. 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9808034650 msgr2=0x7f9808036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9808034650 0x7f9808036b10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55883cda0970 tx=0x7f992001d720 comp rx=0 tx=0).stop 2024-04-27T03:18:10.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdb0520 msgr2=0x55883cdb0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdb0520 0x55883cdb0940 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f992800aab0 tx=0x7f992800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 shutdown_connections 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9808034650 0x7f9808036b10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/3379681829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cdb0520 0x55883cdb0940 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 >> 172.21.15.110:0/3379681829 conn(0x55883cda5330 msgr2=0x55883cda5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 shutdown_connections 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/3379681829 wait complete. 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cd9d950 msgr2=0x55883cd9fd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cd9d950 0x55883cd9fd40 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cca79c0 msgr2=0x55883cd9c540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cca79c0 0x55883cd9c540 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9814034650 msgr2=0x7f9814036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9814034650 0x7f9814036b10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f99180ae000 tx=0x7f99180a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:10.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 msgr2=0x55883cd95610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd95610 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9920006640 tx=0x7f9920000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 shutdown_connections 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55883cd9d950 0x55883cd9fd40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55883cca79c0 0x55883cd9c540 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9814034650 0x7f9814036b10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 --2- 172.21.15.110:0/1655490894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55883cca34b0 0x55883cd95610 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 >> 172.21.15.110:0/1655490894 conn(0x55883cc16b90 msgr2=0x55883cd88c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 shutdown_connections 2024-04-27T03:18:10.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.637+0000 7f99319c1980 1 -- 172.21.15.110:0/1655490894 wait complete. 2024-04-27T03:18:10.657 INFO:teuthology.orchestra.run.smithi110.stdout:[{"bucket":"myfoo","tenant":"","versioning":"off","zonegroup":"3fb181a9-2f1b-4a6e-85a1-ca8305b0483c","placement_rule":"default-placement","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""},"id":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","marker":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","index_type":"Normal","index_generation":0,"num_shards":11,"object_lock_enabled":false,"mfa_enabled":false,"owner":"foo","ver":"0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1","master_ver":"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0","mtime":"2024-04-27T03:18:09.699113Z","creation_time":"2024-04-27T03:18:09.694744Z","max_marker":"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#","usage":{},"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}}] 2024-04-27T03:18:10.657 DEBUG:tasks.util.rgw: json result: [{'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:18:09.699113Z', 'creation_time': '2024-04-27T03:18:09.694744Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}] 2024-04-27T03:18:10.667 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2024-04-27T03:18:10.667 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--bucket', 'myfoo'] 2024-04-27T03:18:10.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--bucket', 'myfoo'] 2024-04-27T03:18:10.668 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --bucket myfoo 2024-04-27T03:18:10.734 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:10.735 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:10.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76887a6980 1 Processor -- start 2024-04-27T03:18:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76887a6980 1 -- start start 2024-04-27T03:18:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76887a6980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559ff81ff850 0x559ff828df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76887a6980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x559ff814a0d0 con 0x559ff828e480 2024-04-27T03:18:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76887a6980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x559ff813f430 con 0x559ff81ff850 2024-04-27T03:18:10.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76862c1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559ff81ff850 0x559ff828df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76862c1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559ff81ff850 0x559ff828df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51650/0 (socket says 172.21.15.110:51650) 2024-04-27T03:18:10.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f7685ac0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x559ff828e480 0x559ff8290870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34788/0 (socket says 172.21.15.110:34788) 2024-04-27T03:18:10.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76862c1640 1 -- 172.21.15.110:0/12543425 learned_addr learned my addr 172.21.15.110:0/12543425 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76862c1640 1 -- 172.21.15.110:0/12543425 >> v1:172.21.15.77:6789/0 conn(0x559ff828e480 legacy=0x559ff8290870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76862c1640 1 -- 172.21.15.110:0/12543425 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff8138510 con 0x559ff81ff850 2024-04-27T03:18:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.729+0000 7f76862c1640 1 --2- 172.21.15.110:0/12543425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff828df40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f767800a5c0 tx=0x7f767801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c80114f85ecb96 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76852bf640 1 -- 172.21.15.110:0/12543425 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76780a6070 con 0x559ff81ff850 2024-04-27T03:18:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76852bf640 1 -- 172.21.15.110:0/12543425 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76780af070 con 0x559ff81ff850 2024-04-27T03:18:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76852bf640 1 -- 172.21.15.110:0/12543425 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76780aa5c0 con 0x559ff81ff850 2024-04-27T03:18:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76887a6980 1 -- 172.21.15.110:0/12543425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 msgr2=0x559ff828df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76887a6980 1 --2- 172.21.15.110:0/12543425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff828df40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f767800a5c0 tx=0x7f767801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76887a6980 1 -- 172.21.15.110:0/12543425 shutdown_connections 2024-04-27T03:18:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76887a6980 1 --2- 172.21.15.110:0/12543425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff828df40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76887a6980 1 -- 172.21.15.110:0/12543425 >> 172.21.15.110:0/12543425 conn(0x559ff8285a20 msgr2=0x559ff8287e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76887a6980 1 -- 172.21.15.110:0/12543425 shutdown_connections 2024-04-27T03:18:10.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.733+0000 7f76887a6980 1 -- 172.21.15.110:0/12543425 wait complete. 2024-04-27T03:18:10.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 Processor -- start 2024-04-27T03:18:10.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- start start 2024-04-27T03:18:10.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff8376400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559ff818e2d0 con 0x559ff81ff850 2024-04-27T03:18:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76862c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff8376400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76862c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff8376400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51652/0 (socket says 172.21.15.110:51652) 2024-04-27T03:18:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76862c1640 1 -- 172.21.15.110:0/4150976188 learned_addr learned my addr 172.21.15.110:0/4150976188 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76862c1640 1 -- 172.21.15.110:0/4150976188 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff8114ae0 con 0x559ff81ff850 2024-04-27T03:18:10.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76862c1640 1 --2- 172.21.15.110:0/4150976188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff8376400 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7678018f70 tx=0x7f7678003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63070ab8ba8730a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f7574608640 1 -- 172.21.15.110:0/4150976188 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76780aa9e0 con 0x559ff81ff850 2024-04-27T03:18:10.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f7574608640 1 -- 172.21.15.110:0/4150976188 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76780aab80 con 0x559ff81ff850 2024-04-27T03:18:10.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f7574608640 1 -- 172.21.15.110:0/4150976188 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76780aae60 con 0x559ff81ff850 2024-04-27T03:18:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- 172.21.15.110:0/4150976188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 msgr2=0x559ff8376400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 --2- 172.21.15.110:0/4150976188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff8376400 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7678018f70 tx=0x7f7678003030 comp rx=0 tx=0).stop 2024-04-27T03:18:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- 172.21.15.110:0/4150976188 shutdown_connections 2024-04-27T03:18:10.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 --2- 172.21.15.110:0/4150976188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff8376400 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- 172.21.15.110:0/4150976188 >> 172.21.15.110:0/4150976188 conn(0x559ff8285a20 msgr2=0x559ff8371f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- 172.21.15.110:0/4150976188 shutdown_connections 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- 172.21.15.110:0/4150976188 wait complete. 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 Processor -- start 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.749+0000 7f76887a6980 1 -- start start 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76887a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff837d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76887a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559ff8292aa0 con 0x559ff81ff850 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76862c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff837d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76862c1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff837d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51656/0 (socket says 172.21.15.110:51656) 2024-04-27T03:18:10.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76862c1640 1 -- 172.21.15.110:0/1960240779 learned_addr learned my addr 172.21.15.110:0/1960240779 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76862c1640 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff837d8c0 con 0x559ff81ff850 2024-04-27T03:18:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76862c1640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff837d380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f767801b170 tx=0x7f76780180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7572e05640 1 -- 172.21.15.110:0/1960240779 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76780bd7c0 con 0x559ff81ff850 2024-04-27T03:18:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559ff837db50 con 0x559ff81ff850 2024-04-27T03:18:10.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7572e05640 1 -- 172.21.15.110:0/1960240779 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76780bd960 con 0x559ff81ff850 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff8380730 con 0x559ff81ff850 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7572e05640 1 -- 172.21.15.110:0/1960240779 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76780bdc40 con 0x559ff81ff850 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7572e05640 1 -- 172.21.15.110:0/1960240779 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f76780ce030 con 0x559ff81ff850 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7572e05640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c034650 0x7f756c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7572e05640 1 -- 172.21.15.110:0/1960240779 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f76780cc020 con 0x559ff81ff850 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7685ac0640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c034650 0x7f756c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff828e480 0x559ff8384210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8384830 con 0x559ff828e480 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7686ac2640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff828e480 0x559ff8384210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7685ac0640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c034650 0x7f756c036b10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f767c001e50 tx=0x7f767c0013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.753+0000 7f7686ac2640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff828e480 0x559ff8384210 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f7686ac2640 1 -- 172.21.15.110:0/1960240779 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f768000a4d0 con 0x559ff828e480 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff8385600 0x559ff83879f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8388050 con 0x559ff8385600 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76862c1640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff8385600 0x559ff83879f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76862c1640 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff8385600 0x559ff83879f0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76862c1640 1 -- 172.21.15.110:0/1960240779 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff8385600 2024-04-27T03:18:10.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8389900 con 0x559ff828e480 2024-04-27T03:18:10.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f7686ac2640 1 -- 172.21.15.110:0/1960240779 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f768000a4d0 con 0x559ff828e480 2024-04-27T03:18:10.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8388b30 con 0x559ff8385600 2024-04-27T03:18:10.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76862c1640 1 -- 172.21.15.110:0/1960240779 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff8385600 2024-04-27T03:18:10.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8388f50 con 0x559ff828e480 2024-04-27T03:18:10.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f7686ac2640 1 -- 172.21.15.110:0/1960240779 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f768000a4d0 con 0x559ff828e480 2024-04-27T03:18:10.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76887a6980 1 Processor -- start 2024-04-27T03:18:10.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.757+0000 7f76887a6980 1 -- start start 2024-04-27T03:18:10.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76887a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff8398250 0x559ff8398670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76887a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559ff8192310 con 0x559ff8398250 2024-04-27T03:18:10.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f7686ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff8398250 0x559ff8398670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f7686ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff8398250 0x559ff8398670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51670/0 (socket says 172.21.15.110:51670) 2024-04-27T03:18:10.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f7686ac2640 1 -- 172.21.15.110:0/3951551807 learned_addr learned my addr 172.21.15.110:0/3951551807 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f7686ac2640 1 -- 172.21.15.110:0/3951551807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff8398bb0 con 0x559ff8398250 2024-04-27T03:18:10.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f7686ac2640 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff8398250 0x559ff8398670 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f768000dfc0 tx=0x7f76800178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f756bfff640 1 -- 172.21.15.110:0/3951551807 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76800157d0 con 0x559ff8398250 2024-04-27T03:18:10.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f756bfff640 1 -- 172.21.15.110:0/3951551807 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7680015970 con 0x559ff8398250 2024-04-27T03:18:10.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f756bfff640 1 -- 172.21.15.110:0/3951551807 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7680015c70 con 0x559ff8398250 2024-04-27T03:18:10.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559ff8399650 con 0x559ff8398250 2024-04-27T03:18:10.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff8399b20 con 0x559ff8398250 2024-04-27T03:18:10.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f756bfff640 1 -- 172.21.15.110:0/3951551807 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7680029440 con 0x559ff8398250 2024-04-27T03:18:10.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f756bfff640 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75600346a0 0x7f7560036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f756bfff640 1 -- 172.21.15.110:0/3951551807 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f768000e040 con 0x559ff8398250 2024-04-27T03:18:10.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76862c1640 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75600346a0 0x7f7560036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76887a6980 1 Processor -- start 2024-04-27T03:18:10.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76887a6980 1 -- start start 2024-04-27T03:18:10.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.761+0000 7f76862c1640 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75600346a0 0x7f7560036b60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x559ff8384dc0 tx=0x7f76780e38f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff83a5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559ff8376a90 con 0x559ff83a53e0 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff83a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff83a5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51684/0 (socket says 172.21.15.110:51684) 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 -- 172.21.15.110:0/3838391037 learned_addr learned my addr 172.21.15.110:0/3838391037 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 -- 172.21.15.110:0/3838391037 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff83a5dd0 con 0x559ff83a53e0 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 --2- 172.21.15.110:0/3838391037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff83a5800 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f768004a840 tx=0x7f768004aaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28e843e0ae0b844 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7569ffb640 1 -- 172.21.15.110:0/3838391037 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f768003c320 con 0x559ff83a53e0 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7569ffb640 1 -- 172.21.15.110:0/3838391037 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768004c110 con 0x559ff83a53e0 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7569ffb640 1 -- 172.21.15.110:0/3838391037 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76800283a0 con 0x559ff83a53e0 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- 172.21.15.110:0/3838391037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 msgr2=0x559ff83a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 --2- 172.21.15.110:0/3838391037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff83a5800 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f768004a840 tx=0x7f768004aaf0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- 172.21.15.110:0/3838391037 shutdown_connections 2024-04-27T03:18:10.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 --2- 172.21.15.110:0/3838391037 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff83a5800 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- 172.21.15.110:0/3838391037 >> 172.21.15.110:0/3838391037 conn(0x559ff83a1720 msgr2=0x559ff839f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- 172.21.15.110:0/3838391037 shutdown_connections 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- 172.21.15.110:0/3838391037 wait complete. 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 Processor -- start 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- start start 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff828b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f76887a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559ff83a6550 con 0x559ff83a53e0 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff828b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff828b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51690/0 (socket says 172.21.15.110:51690) 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 learned_addr learned my addr 172.21.15.110:0/2774161680 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.765+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff8289f60 con 0x559ff83a53e0 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7686ac2640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff828b840 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x559ff83a1e20 tx=0x7f768004a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7557fff640 1 -- 172.21.15.110:0/2774161680 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7680036070 con 0x559ff83a53e0 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559ff828a1f0 con 0x559ff83a53e0 2024-04-27T03:18:10.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff828a660 con 0x559ff83a53e0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7557fff640 1 -- 172.21.15.110:0/2774161680 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768003c320 con 0x559ff83a53e0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7557fff640 1 -- 172.21.15.110:0/2774161680 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7680041040 con 0x559ff83a53e0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7557fff640 1 -- 172.21.15.110:0/2774161680 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7680039070 con 0x559ff83a53e0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7557fff640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75500346a0 0x7f7550036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7557fff640 1 -- 172.21.15.110:0/2774161680 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7680014530 con 0x559ff83a53e0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76862c1640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75500346a0 0x7f7550036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff83ba180 0x559ff83bc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83bcd90 con 0x559ff83ba180 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7685ac0640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff83ba180 0x559ff83bc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76862c1640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75500346a0 0x7f7550036b60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7678005550 tx=0x7f76780ed000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7685ac0640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff83ba180 0x559ff83bc640 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f767c00f6a0 con 0x559ff83ba180 2024-04-27T03:18:10.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 realm 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83b9690 con 0x559ff83ba180 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f767c00f6a0 con 0x559ff83ba180 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83bd860 con 0x559ff83ba180 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f767c00f6a0 con 0x559ff83ba180 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff83be430 0x559ff83c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c1060 con 0x559ff83be430 2024-04-27T03:18:10.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.769+0000 7f7686ac2640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff83be430 0x559ff83c08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7686ac2640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff83be430 0x559ff83c08f0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c1900 con 0x559ff83ba180 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f767c00f6a0 con 0x559ff83ba180 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76887a6980 20 searching for the correct realm 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559ff83c3670 0x559ff83c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83c3670 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76862c1640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559ff83c3670 0x559ff83c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76862c1640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559ff83c3670 0x559ff83c5a60 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7686ac2640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7680014030 0x7f7680020840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x7f7680014030 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7685ac0640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7680014030 0x7f7680020840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7685ac0640 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7680014030 0x7f7680020840 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c00f6a0 con 0x7f7680014030 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.773+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c00f6a0 con 0x559ff83ba180 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83c3670 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x7f7680014030 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x7f7680014030 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x7f7680014030 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f767c02b040 con 0x7f7680014030 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.777+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83c3670 2024-04-27T03:18:10.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83c3670 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83c3670 2024-04-27T03:18:10.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x7f7680014030 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x7f7680014030 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83ba180 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c02b040 con 0x559ff83ba180 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x7f7680014030 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.781+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83c3670 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x7f7680014030 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83c3670 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6070 con 0x559ff83be430 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c22b0 con 0x559ff83ba180 2024-04-27T03:18:10.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.785+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c2730 con 0x559ff83be430 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c2b90 con 0x559ff83ba180 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 zone default found 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 4 Realm: () 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:10.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 zonegroup default 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c6980 con 0x7f7680014030 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c8390 con 0x559ff83c3670 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c87d0 con 0x559ff83c3670 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 started sync module instance, tier type = 2024-04-27T03:18:10.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:10.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.789+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c77f0 con 0x559ff83be430 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94145552874464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c9110 con 0x559ff83be430 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c9960 con 0x559ff83ba180 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94145552879888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83c9ac0 con 0x559ff83ba180 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83ca5c0 con 0x559ff83be430 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94145552874464] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.793+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94145552879888] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94145552883888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83ca580 con 0x559ff83be430 2024-04-27T03:18:10.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cae90 con 0x7f7680014030 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94145552886240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83ca580 con 0x7f7680014030 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cb910 con 0x559ff83be430 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94145552886240] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94145552883888] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94145552878688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cb420 con 0x559ff83be430 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.797+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cc1b0 con 0x559ff83c3670 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94145552891136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cb420 con 0x559ff83c3670 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83ccf30 con 0x559ff83c3670 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94145552878688] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94145552891136] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94145552893264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cd320 con 0x559ff83c3670 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cdbb0 con 0x7f7680014030 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94145552893264] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'99 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94145552894320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83cd7d0 con 0x7f7680014030 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 20 add_watcher() i=0 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 20 add_watcher() i=1 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 20 add_watcher() i=2 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 20 add_watcher() i=3 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 20 add_watcher() i=4 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 20 add_watcher() i=5 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.801+0000 7f76887a6980 20 add_watcher() i=6 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94145552894320] v27'100 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 20 add_watcher() i=7 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83d1090 con 0x559ff83ba180 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94145552897264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83ce550 con 0x559ff83ba180 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94145552897264] v27'106 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f753cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 5 note: GC not initialized 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f74b77fe640 20 reqs_thread_entry: start 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 20 Filter name: none 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f74b57fa640 20 reqs_thread_entry: start 2024-04-27T03:18:10.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.805+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f4770 con 0x7f7680014030 2024-04-27T03:18:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:10.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f50a0 con 0x7f7680014030 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f50e0 con 0x7f7680014030 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(64 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f6780 con 0x7f7680014030 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(65 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:10.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:66 4.11 4:8a36b39c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f5e20 con 0x559ff83c3670 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 4.39 4:9c046854:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83fb310 con 0x559ff83c3670 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 4.15 4:aaf269a6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83fdd40 con 0x559ff83c3670 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 4.24 4:241b82a3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8400680 con 0x559ff83ba180 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 4.e 4:709f8441:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8402ef0 con 0x7f7680014030 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 4.3d 4:bc3f4761:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8405a10 con 0x559ff83be430 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 4.8 4:10a25087:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff8408440 con 0x559ff83c3670 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 4.1 4:836fd4f5:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff840ae50 con 0x7f7680014030 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:74 4.2e 4:773ce9c4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff840d790 con 0x559ff83c3670 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:75 4.1 4:80e658b2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff83fe770 con 0x7f7680014030 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 4.b 4:d2da04c3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559ff84107c0 con 0x7f7680014030 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(67 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(70 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(68 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(72 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(66 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(74 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(73 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(76 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(71 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5 [call out=124b] v0'0 uv3 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.809+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(75 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.813+0000 7f753cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.813+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:77 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94145552897264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x559ff83ba180 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.813+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(77 data_loggenerations_metadata [watch unwatch cookie 94145552897264] v27'107 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.813+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94145552874464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x559ff83be430 2024-04-27T03:18:10.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(78 notify.0 [watch unwatch cookie 94145552874464] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f76887a6980 20 remove_watcher() i=0 2024-04-27T03:18:10.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f76887a6980 2 removed watcher, disabling cache 2024-04-27T03:18:10.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94145552879888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x559ff83ba180 2024-04-27T03:18:10.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(79 notify.1 [watch unwatch cookie 94145552879888] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x559ff83ba180 2024-04-27T03:18:10.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f76887a6980 20 remove_watcher() i=1 2024-04-27T03:18:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94145552883888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x559ff83be430 2024-04-27T03:18:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(80 notify.2 [watch unwatch cookie 94145552883888] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f76887a6980 20 remove_watcher() i=2 2024-04-27T03:18:10.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.817+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:81 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94145552886240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x7f7680014030 2024-04-27T03:18:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.821+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(81 notify.3 [watch unwatch cookie 94145552886240] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.821+0000 7f76887a6980 20 remove_watcher() i=3 2024-04-27T03:18:10.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.821+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94145552878688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x559ff83be430 2024-04-27T03:18:10.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.821+0000 7f7686ac2640 1 -- 172.21.15.110:0/2774161680 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(82 notify.4 [watch unwatch cookie 94145552878688] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7680038030 con 0x559ff83be430 2024-04-27T03:18:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.821+0000 7f76887a6980 20 remove_watcher() i=4 2024-04-27T03:18:10.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.821+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:83 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94145552891136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x559ff83c3670 2024-04-27T03:18:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(83 notify.5 [watch unwatch cookie 94145552891136] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 20 remove_watcher() i=5 2024-04-27T03:18:10.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94145552893264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x559ff83c3670 2024-04-27T03:18:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76862c1640 1 -- 172.21.15.110:0/2774161680 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(84 notify.6 [watch unwatch cookie 94145552893264] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76780c6d20 con 0x559ff83c3670 2024-04-27T03:18:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 20 remove_watcher() i=6 2024-04-27T03:18:10.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:85 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94145552894320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559ff83f2130 con 0x7f7680014030 2024-04-27T03:18:10.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(85 notify.7 [watch unwatch cookie 94145552894320] v27'101 uv99 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c03e040 con 0x7f7680014030 2024-04-27T03:18:10.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 20 remove_watcher() i=7 2024-04-27T03:18:10.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff83be430 msgr2=0x559ff83c08f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff83be430 0x559ff83c08f0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff83ba180 msgr2=0x559ff83bc640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff83ba180 0x559ff83bc640 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7680014030 msgr2=0x7f7680020840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7680014030 0x7f7680020840 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559ff83c3670 msgr2=0x559ff83c5a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559ff83c3670 0x559ff83c5a60 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75500346a0 msgr2=0x7f7550036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75500346a0 0x7f7550036b60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7678005550 tx=0x7f76780ed000 comp rx=0 tx=0).stop 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 msgr2=0x559ff828b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f7685ac0640 1 -- 172.21.15.110:0/2774161680 reap_dead start 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff828b840 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x559ff83a1e20 tx=0x7f768004a7d0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 shutdown_connections 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/2774161680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff83a53e0 0x559ff828b840 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 >> 172.21.15.110:0/2774161680 conn(0x559ff83a1720 msgr2=0x559ff839ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 shutdown_connections 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/2774161680 wait complete. 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75600346a0 msgr2=0x7f7560036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75600346a0 0x7f7560036b60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x559ff8384dc0 tx=0x7f76780e38f0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff8398250 msgr2=0x559ff8398670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.825+0000 7f76887a6980 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff8398250 0x559ff8398670 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f768000dfc0 tx=0x7f76800178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 shutdown_connections 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f75600346a0 0x7f7560036b60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/3951551807 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff8398250 0x559ff8398670 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 >> 172.21.15.110:0/3951551807 conn(0x559ff838d060 msgr2=0x559ff838db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 shutdown_connections 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/3951551807 wait complete. 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff8385600 msgr2=0x559ff83879f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff8385600 0x559ff83879f0 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff828e480 msgr2=0x559ff8384210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff828e480 0x559ff8384210 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c034650 msgr2=0x7f756c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c034650 0x7f756c036b10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f767c001e50 tx=0x7f767c0013b0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 msgr2=0x559ff837d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff837d380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f767801b170 tx=0x7f76780180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 shutdown_connections 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559ff8385600 0x559ff83879f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559ff828e480 0x559ff8384210 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c034650 0x7f756c036b10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 --2- 172.21.15.110:0/1960240779 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559ff81ff850 0x559ff837d380 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 >> 172.21.15.110:0/1960240779 conn(0x559ff8285a20 msgr2=0x559ff8370aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 shutdown_connections 2024-04-27T03:18:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.829+0000 7f76887a6980 1 -- 172.21.15.110:0/1960240779 wait complete. 2024-04-27T03:18:10.846 INFO:teuthology.orchestra.run.smithi110.stdout:{"bucket":"myfoo","tenant":"","versioning":"off","zonegroup":"3fb181a9-2f1b-4a6e-85a1-ca8305b0483c","placement_rule":"default-placement","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""},"id":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","marker":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","index_type":"Normal","index_generation":0,"num_shards":11,"object_lock_enabled":false,"mfa_enabled":false,"owner":"foo","ver":"0#1,1#1,2#1,3#1,4#1,5#2,6#1,7#1,8#1,9#1,10#1","master_ver":"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0","mtime":"2024-04-27T03:18:09.699113Z","creation_time":"2024-04-27T03:18:09.694744Z","max_marker":"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#","usage":{"rgw.main":{"size":3,"size_actual":4096,"size_utilized":3,"size_kb":1,"size_kb_actual":4,"size_kb_utilized":1,"num_objects":1}},"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}} 2024-04-27T03:18:10.847 DEBUG:tasks.util.rgw: json result: {'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#2,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:18:09.699113Z', 'creation_time': '2024-04-27T03:18:09.694744Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 3, 'size_actual': 4096, 'size_utilized': 3, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2024-04-27T03:18:10.847 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'stats', '--uid', 'foo', '--sync-stats'] 2024-04-27T03:18:10.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'stats', '--uid', 'foo', '--sync-stats'] 2024-04-27T03:18:10.847 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user stats --uid foo --sync-stats 2024-04-27T03:18:10.923 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:10.923 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:10.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f6971158980 1 Processor -- start 2024-04-27T03:18:10.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f6971158980 1 -- start start 2024-04-27T03:18:10.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f6971158980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56304b435a90 0x56304b437e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f6971158980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x56304b2ee0d0 con 0x56304b3a2fd0 2024-04-27T03:18:10.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f6971158980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x56304b2e3430 con 0x56304b435a90 2024-04-27T03:18:10.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f696e472640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56304b435a90 0x56304b437e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f696e472640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56304b435a90 0x56304b437e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51706/0 (socket says 172.21.15.110:51706) 2024-04-27T03:18:10.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f696ec73640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x56304b3a2fd0 0x56304b434370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34802/0 (socket says 172.21.15.110:34802) 2024-04-27T03:18:10.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f696e472640 1 -- 172.21.15.110:0/1640829180 learned_addr learned my addr 172.21.15.110:0/1640829180 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f696e472640 1 -- 172.21.15.110:0/1640829180 >> v1:172.21.15.77:6789/0 conn(0x56304b3a2fd0 legacy=0x56304b434370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.917+0000 7f696e472640 1 -- 172.21.15.110:0/1640829180 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304b2dc510 con 0x56304b435a90 2024-04-27T03:18:10.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f696e472640 1 --2- 172.21.15.110:0/1640829180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b435a90 0x56304b437e80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f696400a5c0 tx=0x7f696401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5eab1eb0f0ce43 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f696dc71640 1 -- 172.21.15.110:0/1640829180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69640a6070 con 0x56304b435a90 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f696dc71640 1 -- 172.21.15.110:0/1640829180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69640af070 con 0x56304b435a90 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f696dc71640 1 -- 172.21.15.110:0/1640829180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69640aa5c0 con 0x56304b435a90 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f6971158980 1 -- 172.21.15.110:0/1640829180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b435a90 msgr2=0x56304b437e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f6971158980 1 --2- 172.21.15.110:0/1640829180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b435a90 0x56304b437e80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f696400a5c0 tx=0x7f696401b340 comp rx=0 tx=0).stop 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f6971158980 1 -- 172.21.15.110:0/1640829180 shutdown_connections 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f6971158980 1 --2- 172.21.15.110:0/1640829180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b435a90 0x56304b437e80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f6971158980 1 -- 172.21.15.110:0/1640829180 >> 172.21.15.110:0/1640829180 conn(0x56304b429a50 msgr2=0x56304b42be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f6971158980 1 -- 172.21.15.110:0/1640829180 shutdown_connections 2024-04-27T03:18:10.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.921+0000 7f6971158980 1 -- 172.21.15.110:0/1640829180 wait complete. 2024-04-27T03:18:10.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 Processor -- start 2024-04-27T03:18:10.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 -- start start 2024-04-27T03:18:10.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b51c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56304b3322d0 con 0x56304b3a2fd0 2024-04-27T03:18:10.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f696ec73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b51c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f696ec73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b51c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51720/0 (socket says 172.21.15.110:51720) 2024-04-27T03:18:10.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f696ec73640 1 -- 172.21.15.110:0/4051941597 learned_addr learned my addr 172.21.15.110:0/4051941597 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f696ec73640 1 -- 172.21.15.110:0/4051941597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304b2b8ae0 con 0x56304b3a2fd0 2024-04-27T03:18:10.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f696ec73640 1 --2- 172.21.15.110:0/4051941597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b51c620 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69600015f0 tx=0x7f6960001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1e9d96df7e3332 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f685ce09640 1 -- 172.21.15.110:0/4051941597 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6960013420 con 0x56304b3a2fd0 2024-04-27T03:18:10.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f685ce09640 1 -- 172.21.15.110:0/4051941597 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69600135e0 con 0x56304b3a2fd0 2024-04-27T03:18:10.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f685ce09640 1 -- 172.21.15.110:0/4051941597 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6960013940 con 0x56304b3a2fd0 2024-04-27T03:18:10.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 -- 172.21.15.110:0/4051941597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 msgr2=0x56304b51c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 --2- 172.21.15.110:0/4051941597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b51c620 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69600015f0 tx=0x7f6960001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 -- 172.21.15.110:0/4051941597 shutdown_connections 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 --2- 172.21.15.110:0/4051941597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b51c620 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 -- 172.21.15.110:0/4051941597 >> 172.21.15.110:0/4051941597 conn(0x56304b429a50 msgr2=0x56304b5181b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 -- 172.21.15.110:0/4051941597 shutdown_connections 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 -- 172.21.15.110:0/4051941597 wait complete. 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.937+0000 7f6971158980 1 Processor -- start 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f6971158980 1 -- start start 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f6971158980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b523660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f6971158980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56304b43ae20 con 0x56304b3a2fd0 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696ec73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b523660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696ec73640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b523660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51734/0 (socket says 172.21.15.110:51734) 2024-04-27T03:18:10.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696ec73640 1 -- 172.21.15.110:0/1872176048 learned_addr learned my addr 172.21.15.110:0/1872176048 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696ec73640 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304b523ba0 con 0x56304b3a2fd0 2024-04-27T03:18:10.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696ec73640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b523660 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6960006640 tx=0x7f6960000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f685b606640 1 -- 172.21.15.110:0/1872176048 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6960022020 con 0x56304b3a2fd0 2024-04-27T03:18:10.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f685b606640 1 -- 172.21.15.110:0/1872176048 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6960006320 con 0x56304b3a2fd0 2024-04-27T03:18:10.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56304b523e30 con 0x56304b3a2fd0 2024-04-27T03:18:10.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f685b606640 1 -- 172.21.15.110:0/1872176048 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6960004af0 con 0x56304b3a2fd0 2024-04-27T03:18:10.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56304b526ab0 con 0x56304b3a2fd0 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f685b606640 1 -- 172.21.15.110:0/1872176048 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6960025020 con 0x56304b3a2fd0 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f685b606640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f68540346a0 0x7f6854036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f685b606640 1 -- 172.21.15.110:0/1872176048 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f6960018030 con 0x56304b3a2fd0 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696e472640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f68540346a0 0x7f6854036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b435a90 0x56304b52a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b52ab90 con 0x56304b435a90 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696f474640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b435a90 0x56304b52a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.941+0000 7f696e472640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f68540346a0 0x7f6854036b60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f69640ae000 tx=0x7f69640a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696f474640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b435a90 0x56304b52a590 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696f474640 1 -- 172.21.15.110:0/1872176048 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f69680083f0 con 0x56304b435a90 2024-04-27T03:18:10.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b52b960 0x56304b52dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b52e390 con 0x56304b52b960 2024-04-27T03:18:10.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696ec73640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b52b960 0x56304b52dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696ec73640 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b52b960 0x56304b52dd50 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696ec73640 1 -- 172.21.15.110:0/1872176048 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f696001b070 con 0x56304b52b960 2024-04-27T03:18:10.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b52fc40 con 0x56304b435a90 2024-04-27T03:18:10.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696f474640 1 -- 172.21.15.110:0/1872176048 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f69680083f0 con 0x56304b435a90 2024-04-27T03:18:10.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b52ee70 con 0x56304b52b960 2024-04-27T03:18:10.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696ec73640 1 -- 172.21.15.110:0/1872176048 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f696001b070 con 0x56304b52b960 2024-04-27T03:18:10.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b52f260 con 0x56304b435a90 2024-04-27T03:18:10.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f696f474640 1 -- 172.21.15.110:0/1872176048 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f69680083f0 con 0x56304b435a90 2024-04-27T03:18:10.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 Processor -- start 2024-04-27T03:18:10.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 -- start start 2024-04-27T03:18:10.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.945+0000 7f6971158980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b53e540 0x56304b53e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6971158980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56304b336310 con 0x56304b53e540 2024-04-27T03:18:10.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f696f474640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b53e540 0x56304b53e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f696f474640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b53e540 0x56304b53e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51740/0 (socket says 172.21.15.110:51740) 2024-04-27T03:18:10.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f696f474640 1 -- 172.21.15.110:0/808290259 learned_addr learned my addr 172.21.15.110:0/808290259 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f696f474640 1 -- 172.21.15.110:0/808290259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304b53eea0 con 0x56304b53e540 2024-04-27T03:18:10.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f696f474640 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b53e540 0x56304b53e960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f696800aab0 tx=0x7f696800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6858e01640 1 -- 172.21.15.110:0/808290259 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6968007d30 con 0x56304b53e540 2024-04-27T03:18:10.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6858e01640 1 -- 172.21.15.110:0/808290259 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f696801d070 con 0x56304b53e540 2024-04-27T03:18:10.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56304b53f940 con 0x56304b53e540 2024-04-27T03:18:10.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6858e01640 1 -- 172.21.15.110:0/808290259 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6968018440 con 0x56304b53e540 2024-04-27T03:18:10.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56304b53fe10 con 0x56304b53e540 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6858e01640 1 -- 172.21.15.110:0/808290259 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6968013070 con 0x56304b53e540 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6858e01640 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6848034650 0x7f6848036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6858e01640 1 -- 172.21.15.110:0/808290259 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f696804b530 con 0x56304b53e540 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f696ec73640 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6848034650 0x7f6848036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6971158980 1 Processor -- start 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6971158980 1 -- start start 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f696ec73640 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6848034650 0x7f6848036b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x56304b52e990 tx=0x7f6960004d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6971158980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b54bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.949+0000 7f6971158980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56304b51cd70 con 0x56304b54b6f0 2024-04-27T03:18:10.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b54bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b54bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51744/0 (socket says 172.21.15.110:51744) 2024-04-27T03:18:10.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 -- 172.21.15.110:0/145151034 learned_addr learned my addr 172.21.15.110:0/145151034 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 -- 172.21.15.110:0/145151034 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304b54c0e0 con 0x56304b54b6f0 2024-04-27T03:18:10.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 --2- 172.21.15.110:0/145151034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b54bb10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6968049970 tx=0x7f696803a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c511e6192097e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f68527fc640 1 -- 172.21.15.110:0/145151034 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f696803f020 con 0x56304b54b6f0 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f68527fc640 1 -- 172.21.15.110:0/145151034 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f696803b310 con 0x56304b54b6f0 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f68527fc640 1 -- 172.21.15.110:0/145151034 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6968022460 con 0x56304b54b6f0 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- 172.21.15.110:0/145151034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 msgr2=0x56304b54bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 --2- 172.21.15.110:0/145151034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b54bb10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6968049970 tx=0x7f696803a110 comp rx=0 tx=0).stop 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- 172.21.15.110:0/145151034 shutdown_connections 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 --2- 172.21.15.110:0/145151034 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b54bb10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- 172.21.15.110:0/145151034 >> 172.21.15.110:0/145151034 conn(0x56304b547a80 msgr2=0x56304b545ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- 172.21.15.110:0/145151034 shutdown_connections 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- 172.21.15.110:0/145151034 wait complete. 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 Processor -- start 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- start start 2024-04-27T03:18:10.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b552c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56304b54c850 con 0x56304b54b6f0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b552c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b552c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51760/0 (socket says 172.21.15.110:51760) 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 learned_addr learned my addr 172.21.15.110:0/1000138468 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56304b553160 con 0x56304b54b6f0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f696f474640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b552c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x56304b548180 tx=0x7f69680498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6850ff9640 1 -- 172.21.15.110:0/1000138468 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f696802f020 con 0x56304b54b6f0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56304b5533f0 con 0x56304b54b6f0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56304b5560d0 con 0x56304b54b6f0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6850ff9640 1 -- 172.21.15.110:0/1000138468 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6968020c80 con 0x56304b54b6f0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.953+0000 7f6850ff9640 1 -- 172.21.15.110:0/1000138468 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f696803f040 con 0x56304b54b6f0 2024-04-27T03:18:10.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6850ff9640 1 -- 172.21.15.110:0/1000138468 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6968032020 con 0x56304b54b6f0 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6850ff9640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6838034650 0x7f6838036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6850ff9640 1 -- 172.21.15.110:0/1000138468 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f696804ad90 con 0x56304b54b6f0 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696ec73640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6838034650 0x7f6838036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b55e1e0 0x56304b560680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b560dd0 con 0x56304b55e1e0 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696e472640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b55e1e0 0x56304b560680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696ec73640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6838034650 0x7f6838036b10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f696001efd0 tx=0x7f6960037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696e472640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b55e1e0 0x56304b560680 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 realm 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b561200 con 0x56304b55e1e0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b561aa0 con 0x56304b55e1e0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b5624c0 0x56304b5648d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b565040 con 0x56304b5624c0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696f474640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b5624c0 0x56304b5648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.957+0000 7f696f474640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b5624c0 0x56304b5648d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f6971158980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5658e0 con 0x56304b55e1e0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f6971158980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f6971158980 20 searching for the correct realm 2024-04-27T03:18:10.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56304b567650 0x56304b569a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b567650 2024-04-27T03:18:10.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696ec73640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56304b567650 0x56304b569a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696ec73640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56304b567650 0x56304b569a40 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696f474640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6968034020 0x7f69680137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x7f6968034020 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696e472640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6968034020 0x7f69680137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696e472640 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6968034020 0x7f69680137e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.961+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b567650 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x7f6968034020 2024-04-27T03:18:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x7f6968034020 2024-04-27T03:18:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.965+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b567650 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b567650 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b567650 2024-04-27T03:18:10.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x7f6968034020 2024-04-27T03:18:10.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b55e1e0 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x7f6968034020 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.969+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b567650 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x7f6968034020 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b567650 2024-04-27T03:18:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a020 con 0x56304b5624c0 2024-04-27T03:18:10.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:10.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:10.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:10.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:10.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:10.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b566260 con 0x56304b55e1e0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.973+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5666b0 con 0x56304b5624c0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b566ad0 con 0x56304b55e1e0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 zone default found 2024-04-27T03:18:10.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 4 Realm: () 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 zonegroup default 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b56a950 con 0x7f6968034020 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b56c700 con 0x56304b567650 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:10.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b56cb70 con 0x56304b567650 2024-04-27T03:18:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 started sync module instance, tier type = 2024-04-27T03:18:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:10.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.977+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56bb70 con 0x56304b5624c0 2024-04-27T03:18:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94765422395232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56b7d0 con 0x56304b5624c0 2024-04-27T03:18:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56daa0 con 0x56304b55e1e0 2024-04-27T03:18:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94765422401008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56da10 con 0x56304b55e1e0 2024-04-27T03:18:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56e9e0 con 0x56304b5624c0 2024-04-27T03:18:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94765422395232] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94765422401008] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.981+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94765422403216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56e9a0 con 0x56304b5624c0 2024-04-27T03:18:10.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56f280 con 0x7f6968034020 2024-04-27T03:18:10.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94765422407120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56e9a0 con 0x7f6968034020 2024-04-27T03:18:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56fd20 con 0x56304b5624c0 2024-04-27T03:18:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94765422403216] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94765422407120] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94765422408176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b56fce0 con 0x56304b5624c0 2024-04-27T03:18:10.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.985+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b570110 con 0x56304b567650 2024-04-27T03:18:10.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94765422404784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b5709b0 con 0x56304b567650 2024-04-27T03:18:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b571370 con 0x56304b567650 2024-04-27T03:18:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94765422408176] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:10.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94765422404784] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94765422409744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b571370 con 0x56304b567650 2024-04-27T03:18:10.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b571c50 con 0x7f6968034020 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94765422409744] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'102 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94765422417760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b571370 con 0x7f6968034020 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 20 add_watcher() i=0 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 20 add_watcher() i=1 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 20 add_watcher() i=2 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 20 add_watcher() i=3 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 20 add_watcher() i=4 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 20 add_watcher() i=5 2024-04-27T03:18:10.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.989+0000 7f6971158980 20 add_watcher() i=6 2024-04-27T03:18:11.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94765422417760] v27'103 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 20 add_watcher() i=7 2024-04-27T03:18:11.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:11.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5750d0 con 0x56304b55e1e0 2024-04-27T03:18:11.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:11.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94765422416016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b572620 con 0x56304b55e1e0 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94765422416016] v27'108 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f68257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 5 note: GC not initialized 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f679ffff640 20 reqs_thread_entry: start 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 20 Filter name: none 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f679dffb640 20 reqs_thread_entry: start 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.993+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5983e0 con 0x56304b55e1e0 2024-04-27T03:18:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b599930 con 0x7f6968034020 2024-04-27T03:18:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 foo.buckets [call out=183b] v0'0 uv14 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b599930 con 0x7f6968034020 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(64 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:65 4.11 4:8a36b39c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b59b360 con 0x56304b567650 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:66 4.39 4:9c046854:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b59e010 con 0x56304b567650 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 4.15 4:aaf269a6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5a0d40 con 0x56304b567650 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 4.24 4:241b82a3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b59e010 con 0x56304b55e1e0 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 4.e 4:709f8441:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5a5b00 con 0x7f6968034020 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 4.3d 4:bc3f4761:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5a5b00 con 0x56304b5624c0 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 4.8 4:10a25087:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5aac50 con 0x56304b567650 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 4.1 4:836fd4f5:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5ad560 con 0x7f6968034020 2024-04-27T03:18:11.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 4.2e 4:773ce9c4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b0290 con 0x56304b567650 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 4.1 4:80e658b2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b3000 con 0x7f6968034020 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:75 4.b 4:d2da04c3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b5d60 con 0x7f6968034020 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(65 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(66 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(68 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(67 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(69 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(72 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(75 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(70 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5 [call out=124b] v0'0 uv3 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(74 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(71 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:11.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(73 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:11.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:10.997+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b0290 con 0x7f6968034020 2024-04-27T03:18:11.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(76 foo.buckets [call] v27'15 uv15 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:77 8.9 8:96d98480:users.uid::foo.buckets:head [call user.complete_stats_sync in=37b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b0290 con 0x7f6968034020 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(77 foo.buckets [call] v27'16 uv16 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f6971158980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:78 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b0290 con 0x56304b55e1e0 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(78 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f6971158980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:79 8.9 8:96d98480:users.uid::foo.buckets:head [call user.get_header in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b0290 con 0x7f6968034020 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(79 foo.buckets [call out=58b] v0'0 uv16 ondisk = 0) ==== 155+0+58 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f68257e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.001+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:80 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94765422416016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b5b0290 con 0x56304b55e1e0 2024-04-27T03:18:11.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.005+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(80 data_loggenerations_metadata [watch unwatch cookie 94765422416016] v27'109 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:11.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.005+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94765422395232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x56304b5624c0 2024-04-27T03:18:11.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.005+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(81 notify.0 [watch unwatch cookie 94765422395232] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:11.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.005+0000 7f6971158980 20 remove_watcher() i=0 2024-04-27T03:18:11.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.005+0000 7f6971158980 2 removed watcher, disabling cache 2024-04-27T03:18:11.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.005+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:82 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94765422401008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x56304b55e1e0 2024-04-27T03:18:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(82 notify.1 [watch unwatch cookie 94765422401008] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x56304b55e1e0 2024-04-27T03:18:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f6971158980 20 remove_watcher() i=1 2024-04-27T03:18:11.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94765422403216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x56304b5624c0 2024-04-27T03:18:11.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(83 notify.2 [watch unwatch cookie 94765422403216] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:11.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f6971158980 20 remove_watcher() i=2 2024-04-27T03:18:11.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:84 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94765422407120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x7f6968034020 2024-04-27T03:18:11.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(84 notify.3 [watch unwatch cookie 94765422407120] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f6971158980 20 remove_watcher() i=3 2024-04-27T03:18:11.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.009+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94765422408176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x56304b5624c0 2024-04-27T03:18:11.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.013+0000 7f696f474640 1 -- 172.21.15.110:0/1000138468 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(85 notify.4 [watch unwatch cookie 94765422408176] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6968041050 con 0x56304b5624c0 2024-04-27T03:18:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.013+0000 7f6971158980 20 remove_watcher() i=4 2024-04-27T03:18:11.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.013+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94765422404784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x56304b567650 2024-04-27T03:18:11.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.013+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 notify.5 [watch unwatch cookie 94765422404784] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:11.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.013+0000 7f6971158980 20 remove_watcher() i=5 2024-04-27T03:18:11.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.013+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94765422409744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x56304b567650 2024-04-27T03:18:11.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f696ec73640 1 -- 172.21.15.110:0/1000138468 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 notify.6 [watch unwatch cookie 94765422409744] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f696001b070 con 0x56304b567650 2024-04-27T03:18:11.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 20 remove_watcher() i=6 2024-04-27T03:18:11.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:88 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94765422417760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56304b588f00 con 0x7f6968034020 2024-04-27T03:18:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 <== osd.2 v2:172.21.15.110:6808/3575017494 22 ==== osd_op_reply(88 notify.7 [watch unwatch cookie 94765422417760] v27'104 uv102 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f69640aaa20 con 0x7f6968034020 2024-04-27T03:18:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 20 remove_watcher() i=7 2024-04-27T03:18:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b5624c0 msgr2=0x56304b5648d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b5624c0 0x56304b5648d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b55e1e0 msgr2=0x56304b560680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b55e1e0 0x56304b560680 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6968034020 msgr2=0x7f69680137e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6968034020 0x7f69680137e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56304b567650 msgr2=0x56304b569a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56304b567650 0x56304b569a40 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6838034650 msgr2=0x7f6838036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6838034650 0x7f6838036b10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f696001efd0 tx=0x7f6960037000 comp rx=0 tx=0).stop 2024-04-27T03:18:11.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f696e472640 1 -- 172.21.15.110:0/1000138468 reap_dead start 2024-04-27T03:18:11.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 msgr2=0x56304b552c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b552c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x56304b548180 tx=0x7f69680498d0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 shutdown_connections 2024-04-27T03:18:11.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1000138468 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b54b6f0 0x56304b552c20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 >> 172.21.15.110:0/1000138468 conn(0x56304b547a80 msgr2=0x56304b54cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 shutdown_connections 2024-04-27T03:18:11.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1000138468 wait complete. 2024-04-27T03:18:11.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6848034650 msgr2=0x7f6848036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6848034650 0x7f6848036b10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x56304b52e990 tx=0x7f6960004d60 comp rx=0 tx=0).stop 2024-04-27T03:18:11.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b53e540 msgr2=0x56304b53e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b53e540 0x56304b53e960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f696800aab0 tx=0x7f696800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:11.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 shutdown_connections 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6848034650 0x7f6848036b10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/808290259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b53e540 0x56304b53e960 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 >> 172.21.15.110:0/808290259 conn(0x56304b533350 msgr2=0x56304b533e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 shutdown_connections 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/808290259 wait complete. 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b52b960 msgr2=0x56304b52dd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b52b960 0x56304b52dd50 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b435a90 msgr2=0x56304b52a590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.017+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b435a90 0x56304b52a590 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f68540346a0 msgr2=0x7f6854036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f68540346a0 0x7f6854036b60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f69640ae000 tx=0x7f69640a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:11.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 msgr2=0x56304b523660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b523660 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6960006640 tx=0x7f6960000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 shutdown_connections 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56304b52b960 0x56304b52dd50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56304b435a90 0x56304b52a590 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f68540346a0 0x7f6854036b60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 --2- 172.21.15.110:0/1872176048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56304b3a2fd0 0x56304b523660 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 >> 172.21.15.110:0/1872176048 conn(0x56304b429a50 msgr2=0x56304b516cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 shutdown_connections 2024-04-27T03:18:11.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.021+0000 7f6971158980 1 -- 172.21.15.110:0/1872176048 wait complete. 2024-04-27T03:18:11.040 INFO:teuthology.orchestra.run.smithi110.stdout:{"stats":{"size":3,"size_actual":4096,"size_kb":1,"size_kb_actual":4,"num_objects":1},"last_stats_sync":"2024-04-27T03:18:11.001744Z","last_stats_update":"2024-04-27T03:18:11.000354Z"} 2024-04-27T03:18:11.040 DEBUG:tasks.util.rgw: json result: {'stats': {'size': 3, 'size_actual': 4096, 'size_kb': 1, 'size_kb_actual': 4, 'num_objects': 1}, 'last_stats_sync': '2024-04-27T03:18:11.001744Z', 'last_stats_update': '2024-04-27T03:18:11.000354Z'} 2024-04-27T03:18:11.047 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:11.048 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'unlink', '--uid', 'foo', '--bucket', 'myfoo'] 2024-04-27T03:18:11.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'unlink', '--uid', 'foo', '--bucket', 'myfoo'] 2024-04-27T03:18:11.048 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket unlink --uid foo --bucket myfoo 2024-04-27T03:18:11.115 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:11.115 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:11.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b21b88980 1 Processor -- start 2024-04-27T03:18:11.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b21b88980 1 -- start start 2024-04-27T03:18:11.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b21b88980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x565192d335f0 0x565192dc72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b21b88980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x565192c7d0d0 con 0x565192d34930 2024-04-27T03:18:11.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b21b88980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x565192c72430 con 0x565192d335f0 2024-04-27T03:18:11.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b1eea2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x565192d335f0 0x565192dc72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b1eea2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x565192d335f0 0x565192dc72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51766/0 (socket says 172.21.15.110:51766) 2024-04-27T03:18:11.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b1f6a3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x565192d34930 0x565192d32ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34814/0 (socket says 172.21.15.110:34814) 2024-04-27T03:18:11.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b1eea2640 1 -- 172.21.15.110:0/1433291947 learned_addr learned my addr 172.21.15.110:0/1433291947 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b1eea2640 1 -- 172.21.15.110:0/1433291947 >> v1:172.21.15.77:6789/0 conn(0x565192d34930 legacy=0x565192d32ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b1eea2640 1 -- 172.21.15.110:0/1433291947 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x565192c6b510 con 0x565192d335f0 2024-04-27T03:18:11.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.109+0000 7f3b1e6a1640 1 -- 172.21.15.110:0/1433291947 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157986928 0 0) 0x565192c7d0d0 con 0x565192d34930 2024-04-27T03:18:11.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b1eea2640 1 --2- 172.21.15.110:0/1433291947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192dc72b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400a5c0 tx=0x7f3b1401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb3bf02fab7454 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b1e6a1640 1 -- 172.21.15.110:0/1433291947 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b140a6070 con 0x565192d335f0 2024-04-27T03:18:11.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b1e6a1640 1 -- 172.21.15.110:0/1433291947 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b140af070 con 0x565192d335f0 2024-04-27T03:18:11.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b1e6a1640 1 -- 172.21.15.110:0/1433291947 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b140aa5c0 con 0x565192d335f0 2024-04-27T03:18:11.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b21b88980 1 -- 172.21.15.110:0/1433291947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 msgr2=0x565192dc72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b21b88980 1 --2- 172.21.15.110:0/1433291947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192dc72b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400a5c0 tx=0x7f3b1401b340 comp rx=0 tx=0).stop 2024-04-27T03:18:11.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b21b88980 1 -- 172.21.15.110:0/1433291947 shutdown_connections 2024-04-27T03:18:11.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b21b88980 1 --2- 172.21.15.110:0/1433291947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192dc72b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b21b88980 1 -- 172.21.15.110:0/1433291947 >> 172.21.15.110:0/1433291947 conn(0x565192db9b70 msgr2=0x565192dbbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b21b88980 1 -- 172.21.15.110:0/1433291947 shutdown_connections 2024-04-27T03:18:11.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.113+0000 7f3b21b88980 1 -- 172.21.15.110:0/1433291947 wait complete. 2024-04-27T03:18:11.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b21b88980 1 Processor -- start 2024-04-27T03:18:11.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b21b88980 1 -- start start 2024-04-27T03:18:11.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b21b88980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eab890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b21b88980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565192cc12d0 con 0x565192d335f0 2024-04-27T03:18:11.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b1f6a3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b1f6a3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eab890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51776/0 (socket says 172.21.15.110:51776) 2024-04-27T03:18:11.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/2363999277 learned_addr learned my addr 172.21.15.110:0/2363999277 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/2363999277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565192c47ae0 con 0x565192d335f0 2024-04-27T03:18:11.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/2363999277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eab890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000b250 tx=0x7f3b1000b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a87308118e80fdd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3a0d60a640 1 -- 172.21.15.110:0/2363999277 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1000d640 con 0x565192d335f0 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3a0d60a640 1 -- 172.21.15.110:0/2363999277 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b10008070 con 0x565192d335f0 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.129+0000 7f3a0d60a640 1 -- 172.21.15.110:0/2363999277 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1000d990 con 0x565192d335f0 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- 172.21.15.110:0/2363999277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 msgr2=0x565192eab890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2363999277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eab890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000b250 tx=0x7f3b1000b710 comp rx=0 tx=0).stop 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- 172.21.15.110:0/2363999277 shutdown_connections 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2363999277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eab890 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- 172.21.15.110:0/2363999277 >> 172.21.15.110:0/2363999277 conn(0x565192db9b70 msgr2=0x565192ea7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- 172.21.15.110:0/2363999277 shutdown_connections 2024-04-27T03:18:11.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- 172.21.15.110:0/2363999277 wait complete. 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 Processor -- start 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- start start 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eb2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565192dca2e0 con 0x565192d335f0 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b1f6a3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eb2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b1f6a3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eb2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51790/0 (socket says 172.21.15.110:51790) 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/2313212157 learned_addr learned my addr 172.21.15.110:0/2313212157 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565192eb2e70 con 0x565192d335f0 2024-04-27T03:18:11.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eb2930 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3b10000c10 tx=0x7f3b10000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3a0be07640 1 -- 172.21.15.110:0/2313212157 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1001a7c0 con 0x565192d335f0 2024-04-27T03:18:11.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3a0be07640 1 -- 172.21.15.110:0/2313212157 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1001a960 con 0x565192d335f0 2024-04-27T03:18:11.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x565192eb3100 con 0x565192d335f0 2024-04-27T03:18:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3a0be07640 1 -- 172.21.15.110:0/2313212157 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1001ac60 con 0x565192d335f0 2024-04-27T03:18:11.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.133+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565192eb5d80 con 0x565192d335f0 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3a0be07640 1 -- 172.21.15.110:0/2313212157 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f3b10017070 con 0x565192d335f0 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3a0be07640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3a04034650 0x7f3a04036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3a0be07640 1 -- 172.21.15.110:0/2313212157 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f3b10015020 con 0x565192d335f0 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1eea2640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3a04034650 0x7f3a04036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192d34930 0x565192eb9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192eb9ef0 con 0x565192d34930 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192d34930 0x565192eb9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1eea2640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3a04034650 0x7f3a04036b10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b140ae000 tx=0x7f3b140a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192d34930 0x565192eb9880 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1fea4640 1 -- 172.21.15.110:0/2313212157 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192d34930 2024-04-27T03:18:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ebad10 0x565192ebd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ebd740 con 0x565192ebad10 2024-04-27T03:18:11.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ebad10 0x565192ebd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ebad10 0x565192ebd100 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/2313212157 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ebad10 2024-04-27T03:18:11.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ebeff0 con 0x565192d34930 2024-04-27T03:18:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b1fea4640 1 -- 172.21.15.110:0/2313212157 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192d34930 2024-04-27T03:18:11.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.137+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ebe220 con 0x565192ebad10 2024-04-27T03:18:11.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/2313212157 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ebad10 2024-04-27T03:18:11.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ebe610 con 0x565192d34930 2024-04-27T03:18:11.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b1fea4640 1 -- 172.21.15.110:0/2313212157 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192d34930 2024-04-27T03:18:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b21b88980 1 Processor -- start 2024-04-27T03:18:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b21b88980 1 -- start start 2024-04-27T03:18:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b21b88980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192ecd910 0x565192ecdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b21b88980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565192cc5310 con 0x565192ecd910 2024-04-27T03:18:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b1fea4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192ecd910 0x565192ecdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b1fea4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192ecd910 0x565192ecdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51794/0 (socket says 172.21.15.110:51794) 2024-04-27T03:18:11.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b1fea4640 1 -- 172.21.15.110:0/3520583766 learned_addr learned my addr 172.21.15.110:0/3520583766 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b1fea4640 1 -- 172.21.15.110:0/3520583766 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565192ece270 con 0x565192ecd910 2024-04-27T03:18:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192ecd910 0x565192ecdd30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3b180056d0 tx=0x7f3b18017820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3a09602640 1 -- 172.21.15.110:0/3520583766 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b18012840 con 0x565192ecd910 2024-04-27T03:18:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3a09602640 1 -- 172.21.15.110:0/3520583766 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b180129e0 con 0x565192ecd910 2024-04-27T03:18:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x565192eced10 con 0x565192ecd910 2024-04-27T03:18:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3a09602640 1 -- 172.21.15.110:0/3520583766 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b18012ce0 con 0x565192ecd910 2024-04-27T03:18:11.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.141+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565192ecf1e0 con 0x565192ecd910 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3a09602640 1 -- 172.21.15.110:0/3520583766 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f3b18020440 con 0x565192ecd910 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3a09602640 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39f8034650 0x7f39f8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3a09602640 1 -- 172.21.15.110:0/3520583766 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f3b1800e040 con 0x565192ecd910 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39f8034650 0x7f39f8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b21b88980 1 Processor -- start 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b21b88980 1 -- start start 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39f8034650 0x7f39f8036b10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x565192ebdd40 tx=0x7f3b100014a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b21b88980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192edaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b21b88980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565192eabfe0 con 0x565192edaac0 2024-04-27T03:18:11.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b1fea4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192edaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b1fea4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192edaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51802/0 (socket says 172.21.15.110:51802) 2024-04-27T03:18:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b1fea4640 1 -- 172.21.15.110:0/1098169282 learned_addr learned my addr 172.21.15.110:0/1098169282 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b1fea4640 1 -- 172.21.15.110:0/1098169282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565192edb4b0 con 0x565192edaac0 2024-04-27T03:18:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/1098169282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192edaee0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3b1804a6a0 tx=0x7f3b18014410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4150019ed1f59b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3a02ffd640 1 -- 172.21.15.110:0/1098169282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1801f3a0 con 0x565192edaac0 2024-04-27T03:18:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3a02ffd640 1 -- 172.21.15.110:0/1098169282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1801f540 con 0x565192edaac0 2024-04-27T03:18:11.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.145+0000 7f3a02ffd640 1 -- 172.21.15.110:0/1098169282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b18020880 con 0x565192edaac0 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- 172.21.15.110:0/1098169282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 msgr2=0x565192edaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 --2- 172.21.15.110:0/1098169282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192edaee0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3b1804a6a0 tx=0x7f3b18014410 comp rx=0 tx=0).stop 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- 172.21.15.110:0/1098169282 shutdown_connections 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 --2- 172.21.15.110:0/1098169282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192edaee0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- 172.21.15.110:0/1098169282 >> 172.21.15.110:0/1098169282 conn(0x565192ed6e50 msgr2=0x565192ed4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- 172.21.15.110:0/1098169282 shutdown_connections 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- 172.21.15.110:0/1098169282 wait complete. 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 Processor -- start 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- start start 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192ee1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x565192edbc20 con 0x565192edaac0 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b1fea4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192ee1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b1fea4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192ee1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51808/0 (socket says 172.21.15.110:51808) 2024-04-27T03:18:11.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 learned_addr learned my addr 172.21.15.110:0/847596736 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565192ee2530 con 0x565192edaac0 2024-04-27T03:18:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192ee1ff0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3b18014720 tx=0x7f3b1804a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3a017fa640 1 -- 172.21.15.110:0/847596736 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b1801e930 con 0x565192edaac0 2024-04-27T03:18:11.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x565192ee27c0 con 0x565192edaac0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3a017fa640 1 -- 172.21.15.110:0/847596736 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b18044040 con 0x565192edaac0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3a017fa640 1 -- 172.21.15.110:0/847596736 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b18013c60 con 0x565192edaac0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565192ee54a0 con 0x565192edaac0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3a017fa640 1 -- 172.21.15.110:0/847596736 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f3b180133b0 con 0x565192edaac0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3a017fa640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39e8034650 0x7f39e8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.149+0000 7f3a017fa640 1 -- 172.21.15.110:0/847596736 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f3b18034c30 con 0x565192edaac0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39e8034650 0x7f39e8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192eed710 0x565192eefbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef0320 con 0x565192eed710 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1eea2640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192eed710 0x565192eefbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39e8034650 0x7f39e8036b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000e930 tx=0x7f3b10044000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1eea2640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192eed710 0x565192eefbd0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 realm 2024-04-27T03:18:11.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef0750 con 0x565192eed710 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef1020 con 0x565192eed710 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ef1a40 0x565192ef3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef45e0 con 0x565192ef1a40 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ef1a40 0x565192ef3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ef1a40 0x565192ef3e70 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.153+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef4e30 con 0x565192eed710 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b21b88980 20 searching for the correct realm 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x565192ef6ba0 0x565192ef8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef6ba0 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x565192ef6ba0 0x565192ef8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1f6a3640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x565192ef6ba0 0x565192ef8f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1fea4640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3b18014030 0x7f3b1800b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x7f3b18014030 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3b18014030 0x7f3b1800b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3b18014030 0x7f3b1800b8b0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef6ba0 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.157+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x7f3b18014030 2024-04-27T03:18:11.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x7f3b18014030 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef6ba0 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.161+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef6ba0 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef6ba0 2024-04-27T03:18:11.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x7f3b18014030 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192eed710 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x7f3b18014030 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef6ba0 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x7f3b18014030 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.165+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef6ba0 2024-04-27T03:18:11.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9610 con 0x565192ef1a40 2024-04-27T03:18:11.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:11.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:11.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:11.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:11.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:11.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:11.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef57b0 con 0x565192eed710 2024-04-27T03:18:11.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef6050 con 0x565192ef1a40 2024-04-27T03:18:11.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.169+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef53f0 con 0x565192eed710 2024-04-27T03:18:11.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:11.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 zone default found 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 4 Realm: () 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 zonegroup default 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192ef9f20 con 0x7f3b18014030 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192efc150 con 0x565192ef6ba0 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192efc4f0 con 0x565192ef6ba0 2024-04-27T03:18:11.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 started sync module instance, tier type = 2024-04-27T03:18:11.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:11.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efb5e0 con 0x565192ef1a40 2024-04-27T03:18:11.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94908357523920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efcd40 con 0x565192ef1a40 2024-04-27T03:18:11.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.173+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efd590 con 0x565192eed710 2024-04-27T03:18:11.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94908357528896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efd650 con 0x565192eed710 2024-04-27T03:18:11.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efe170 con 0x565192ef1a40 2024-04-27T03:18:11.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94908357523920] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94908357528896] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94908357532736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efe130 con 0x565192ef1a40 2024-04-27T03:18:11.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.177+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efea10 con 0x7f3b18014030 2024-04-27T03:18:11.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94908357535072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192efe130 con 0x7f3b18014030 2024-04-27T03:18:11.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192eff4c0 con 0x565192ef1a40 2024-04-27T03:18:11.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94908357532736] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94908357535072] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94908357536128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192eff4c0 con 0x565192ef1a40 2024-04-27T03:18:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192eff8b0 con 0x565192ef6ba0 2024-04-27T03:18:11.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94908357527696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f00100 con 0x565192ef6ba0 2024-04-27T03:18:11.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.181+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f00b00 con 0x565192ef6ba0 2024-04-27T03:18:11.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94908357536128] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94908357527696] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94908357537744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f00ef0 con 0x565192ef6ba0 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f01780 con 0x7f3b18014030 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'105 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94908357545712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f01370 con 0x7f3b18014030 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=0 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=1 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=2 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=3 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=4 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=5 2024-04-27T03:18:11.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94908357537744] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=6 2024-04-27T03:18:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94908357545712] v27'106 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 20 add_watcher() i=7 2024-04-27T03:18:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.185+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192f00610 con 0x565192eed710 2024-04-27T03:18:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94908357548832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f00610 con 0x565192eed710 2024-04-27T03:18:11.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94908357548832] v27'110 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f39d5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:11.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:11.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 5 note: GC not initialized 2024-04-27T03:18:11.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f39d4fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 20 Filter name: none 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f394e7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.189+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192f28440 con 0x565192eed710 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192f2a520 con 0x7f3b18014030 2024-04-27T03:18:11.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv1 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192f2bd00 con 0x7f3b18014030 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(64 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:11.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache get: name=default.rgw.meta+users.uid+foo : hit (requested=0x13, cached=0x13) 2024-04-27T03:18:11.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f2f420 con 0x7f3b18014030 2024-04-27T03:18:11.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(65 foo.buckets [call] v27'17 uv17 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x13, cached=0x11) 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192f2e580 con 0x7f3b18014030 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(66 myfoo [call out=48b,read 0~165 out=165b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 233+0+288 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x13 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.193+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 8.9 8:91b71841:root::myfoo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f2f140 con 0x7f3b18014030 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(67 myfoo [delete,create,call,call,writefull 0~165] v27'18 uv18 ondisk = 0) ==== 317+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b21b88980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b21b88980 10 moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b21b88980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94908357733328 in=350b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x565192f30ca0 con 0x565192ef6ba0 2024-04-27T03:18:11.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== watch-notify(notify (1) cookie 94908357537744 notify 115964116999 ret 0) ==== 380+0+0 (crc 0 0 0) 0x7f3b1005e040 con 0x565192ef6ba0 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3a02ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116999 cookie 94908357537744 notifier 4471 bl.length()=338 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3a02ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3a02ffd640 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(68 notify.6 [notify cookie 94908357733328 out=8b] v0'0 uv105 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3a02ffd640 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f39e4003eb0 con 0x565192ef6ba0 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(69 notify.6 [notify-ack] v0'0 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== watch-notify(notify_complete (2) cookie 94908357733328 notify 115964116999 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f3b10038030 con 0x565192ef6ba0 2024-04-27T03:18:11.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f39d5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.197+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94908357548832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x565192eed710 2024-04-27T03:18:11.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94908357548832] v27'111 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94908357523920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x565192ef1a40 2024-04-27T03:18:11.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94908357523920] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b21b88980 20 remove_watcher() i=0 2024-04-27T03:18:11.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b21b88980 2 removed watcher, disabling cache 2024-04-27T03:18:11.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94908357528896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x565192eed710 2024-04-27T03:18:11.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94908357528896] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x565192eed710 2024-04-27T03:18:11.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b21b88980 20 remove_watcher() i=1 2024-04-27T03:18:11.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.201+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94908357532736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x565192ef1a40 2024-04-27T03:18:11.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.205+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94908357532736] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.205+0000 7f3b21b88980 20 remove_watcher() i=2 2024-04-27T03:18:11.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.205+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94908357535072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x7f3b18014030 2024-04-27T03:18:11.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.205+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94908357535072] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.205+0000 7f3b21b88980 20 remove_watcher() i=3 2024-04-27T03:18:11.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.205+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94908357536128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x565192ef1a40 2024-04-27T03:18:11.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b1fea4640 1 -- 172.21.15.110:0/847596736 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94908357536128] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1800a4d0 con 0x565192ef1a40 2024-04-27T03:18:11.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b21b88980 20 remove_watcher() i=4 2024-04-27T03:18:11.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94908357527696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x565192ef6ba0 2024-04-27T03:18:11.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94908357527696] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b21b88980 20 remove_watcher() i=5 2024-04-27T03:18:11.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94908357537744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x565192ef6ba0 2024-04-27T03:18:11.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b1f6a3640 1 -- 172.21.15.110:0/847596736 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94908357537744] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b1002e070 con 0x565192ef6ba0 2024-04-27T03:18:11.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b21b88980 20 remove_watcher() i=6 2024-04-27T03:18:11.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.209+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94908357545712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x565192f26170 con 0x7f3b18014030 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94908357545712] v27'107 uv105 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f3b140aa960 con 0x7f3b18014030 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 20 remove_watcher() i=7 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ef1a40 msgr2=0x565192ef3e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ef1a40 0x565192ef3e70 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192eed710 msgr2=0x565192eefbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192eed710 0x565192eefbd0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3b18014030 msgr2=0x7f3b1800b8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f3b18014030 0x7f3b1800b8b0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x565192ef6ba0 msgr2=0x565192ef8f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x565192ef6ba0 0x565192ef8f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39e8034650 msgr2=0x7f39e8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39e8034650 0x7f39e8036b10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000e930 tx=0x7f3b10044000 comp rx=0 tx=0).stop 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b1eea2640 1 -- 172.21.15.110:0/847596736 reap_dead start 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 msgr2=0x565192ee1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192ee1ff0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3b18014720 tx=0x7f3b1804a600 comp rx=0 tx=0).stop 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 shutdown_connections 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/847596736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192edaac0 0x565192ee1ff0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 >> 172.21.15.110:0/847596736 conn(0x565192ed6e50 msgr2=0x565192edc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 shutdown_connections 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/847596736 wait complete. 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39f8034650 msgr2=0x7f39f8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39f8034650 0x7f39f8036b10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x565192ebdd40 tx=0x7f3b100014a0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192ecd910 msgr2=0x565192ecdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192ecd910 0x565192ecdd30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3b180056d0 tx=0x7f3b18017820 comp rx=0 tx=0).stop 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 shutdown_connections 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f39f8034650 0x7f39f8036b10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/3520583766 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192ecd910 0x565192ecdd30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 >> 172.21.15.110:0/3520583766 conn(0x565192ec2720 msgr2=0x565192ec3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 shutdown_connections 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/3520583766 wait complete. 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ebad10 msgr2=0x565192ebd100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ebad10 0x565192ebd100 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192d34930 msgr2=0x565192eb9880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192d34930 0x565192eb9880 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3a04034650 msgr2=0x7f3a04036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3a04034650 0x7f3a04036b10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3b140ae000 tx=0x7f3b140a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 msgr2=0x565192eb2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eb2930 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3b10000c10 tx=0x7f3b10000ee0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 shutdown_connections 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x565192ebad10 0x565192ebd100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x565192d34930 0x565192eb9880 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3a04034650 0x7f3a04036b10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 --2- 172.21.15.110:0/2313212157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x565192d335f0 0x565192eb2930 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 >> 172.21.15.110:0/2313212157 conn(0x565192db9b70 msgr2=0x565192ea5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 shutdown_connections 2024-04-27T03:18:11.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.213+0000 7f3b21b88980 1 -- 172.21.15.110:0/2313212157 wait complete. 2024-04-27T03:18:11.232 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2024-04-27T03:18:11.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2024-04-27T03:18:11.232 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid fud --display-name Fud --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh --max-buckets 1 2024-04-27T03:18:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:11.320 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c488b980 1 Processor -- start 2024-04-27T03:18:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c488b980 1 -- start start 2024-04-27T03:18:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c488b980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b859646a0 0x556b85966a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c488b980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x556b8581f0d0 con 0x556b858d4890 2024-04-27T03:18:11.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c488b980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x556b85814430 con 0x556b859646a0 2024-04-27T03:18:11.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c1ba5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b859646a0 0x556b85966a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c23a6640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x556b858d4890 0x556b85962f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34828/0 (socket says 172.21.15.110:34828) 2024-04-27T03:18:11.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c1ba5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b859646a0 0x556b85966a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51820/0 (socket says 172.21.15.110:51820) 2024-04-27T03:18:11.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c23a6640 1 -- 172.21.15.110:0/550224137 learned_addr learned my addr 172.21.15.110:0/550224137 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:11.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c1ba5640 1 -- 172.21.15.110:0/550224137 >> v1:172.21.15.77:6789/0 conn(0x556b858d4890 legacy=0x556b85962f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.313+0000 7f54c1ba5640 1 -- 172.21.15.110:0/550224137 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b8580d510 con 0x556b859646a0 2024-04-27T03:18:11.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c1ba5640 1 --2- 172.21.15.110:0/550224137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b859646a0 0x556b85966a90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f54b800a480 tx=0x7f54b801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c2c7c88e8d2394 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c13a4640 1 -- 172.21.15.110:0/550224137 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b80b0070 con 0x556b859646a0 2024-04-27T03:18:11.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c13a4640 1 -- 172.21.15.110:0/550224137 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b80ab540 con 0x556b859646a0 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c13a4640 1 -- 172.21.15.110:0/550224137 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b80ab840 con 0x556b859646a0 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c488b980 1 -- 172.21.15.110:0/550224137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b859646a0 msgr2=0x556b85966a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c488b980 1 --2- 172.21.15.110:0/550224137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b859646a0 0x556b85966a90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f54b800a480 tx=0x7f54b801b280 comp rx=0 tx=0).stop 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c488b980 1 -- 172.21.15.110:0/550224137 shutdown_connections 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c488b980 1 --2- 172.21.15.110:0/550224137 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b859646a0 0x556b85966a90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c488b980 1 -- 172.21.15.110:0/550224137 >> 172.21.15.110:0/550224137 conn(0x556b8595aa60 msgr2=0x556b8595ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c488b980 1 -- 172.21.15.110:0/550224137 shutdown_connections 2024-04-27T03:18:11.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.317+0000 7f54c488b980 1 -- 172.21.15.110:0/550224137 wait complete. 2024-04-27T03:18:11.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c488b980 1 Processor -- start 2024-04-27T03:18:11.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c488b980 1 -- start start 2024-04-27T03:18:11.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c488b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a4b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c488b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b858632d0 con 0x556b858d4890 2024-04-27T03:18:11.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c23a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a4b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c23a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a4b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51834/0 (socket says 172.21.15.110:51834) 2024-04-27T03:18:11.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c23a6640 1 -- 172.21.15.110:0/2159746994 learned_addr learned my addr 172.21.15.110:0/2159746994 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c23a6640 1 -- 172.21.15.110:0/2159746994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b857e9ae0 con 0x556b858d4890 2024-04-27T03:18:11.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f54c23a6640 1 --2- 172.21.15.110:0/2159746994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a4b5e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f54b40017b0 tx=0x7f54b4001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e209dedce3e8fb3b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f53b0608640 1 -- 172.21.15.110:0/2159746994 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4016070 con 0x556b858d4890 2024-04-27T03:18:11.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f53b0608640 1 -- 172.21.15.110:0/2159746994 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b4004530 con 0x556b858d4890 2024-04-27T03:18:11.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.333+0000 7f53b0608640 1 -- 172.21.15.110:0/2159746994 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4004870 con 0x556b858d4890 2024-04-27T03:18:11.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- 172.21.15.110:0/2159746994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 msgr2=0x556b85a4b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 --2- 172.21.15.110:0/2159746994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a4b5e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f54b40017b0 tx=0x7f54b4001c70 comp rx=0 tx=0).stop 2024-04-27T03:18:11.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- 172.21.15.110:0/2159746994 shutdown_connections 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 --2- 172.21.15.110:0/2159746994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a4b5e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- 172.21.15.110:0/2159746994 >> 172.21.15.110:0/2159746994 conn(0x556b8595aa60 msgr2=0x556b85a47170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- 172.21.15.110:0/2159746994 shutdown_connections 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- 172.21.15.110:0/2159746994 wait complete. 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 Processor -- start 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- start start 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b85967b70 con 0x556b858d4890 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c23a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c23a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51838/0 (socket says 172.21.15.110:51838) 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c23a6640 1 -- 172.21.15.110:0/1990813253 learned_addr learned my addr 172.21.15.110:0/1990813253 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c23a6640 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b85a52b30 con 0x556b858d4890 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c23a6640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a525f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000f90 tx=0x7f54b4001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f53aee05640 1 -- 172.21.15.110:0/1990813253 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4020070 con 0x556b858d4890 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b85a52dc0 con 0x556b858d4890 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f53aee05640 1 -- 172.21.15.110:0/1990813253 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b4011400 con 0x556b858d4890 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f53aee05640 1 -- 172.21.15.110:0/1990813253 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b4016040 con 0x556b858d4890 2024-04-27T03:18:11.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b85a55a40 con 0x556b858d4890 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f53aee05640 1 -- 172.21.15.110:0/1990813253 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f54b4023070 con 0x556b858d4890 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f53aee05640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a8034650 0x7f53a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f53aee05640 1 -- 172.21.15.110:0/1990813253 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f54b4049210 con 0x556b858d4890 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.337+0000 7f54c1ba5640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a8034650 0x7f53a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b859646a0 0x556b85a59690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a59bd0 con 0x556b859646a0 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b859646a0 0x556b85a59690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c1ba5640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a8034650 0x7f53a8036b10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54b80ad000 tx=0x7f54b80a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b859646a0 0x556b85a59690 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c2ba7640 1 -- 172.21.15.110:0/1990813253 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54bc0083f0 con 0x556b859646a0 2024-04-27T03:18:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a5a680 0x556b85a5cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a5d290 con 0x556b85a5a680 2024-04-27T03:18:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c23a6640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a5a680 0x556b85a5cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c23a6640 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a5a680 0x556b85a5cb20 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c23a6640 1 -- 172.21.15.110:0/1990813253 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a5a680 2024-04-27T03:18:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a5eb40 con 0x556b859646a0 2024-04-27T03:18:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c2ba7640 1 -- 172.21.15.110:0/1990813253 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f54bc0083f0 con 0x556b859646a0 2024-04-27T03:18:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a5dd70 con 0x556b85a5a680 2024-04-27T03:18:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c23a6640 1 -- 172.21.15.110:0/1990813253 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a5a680 2024-04-27T03:18:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.341+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a5e160 con 0x556b859646a0 2024-04-27T03:18:11.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c2ba7640 1 -- 172.21.15.110:0/1990813253 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f54bc0083f0 con 0x556b859646a0 2024-04-27T03:18:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c488b980 1 Processor -- start 2024-04-27T03:18:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c488b980 1 -- start start 2024-04-27T03:18:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c488b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a6d440 0x556b85a6d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c488b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b85867310 con 0x556b85a6d440 2024-04-27T03:18:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c2ba7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a6d440 0x556b85a6d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c2ba7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a6d440 0x556b85a6d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51850/0 (socket says 172.21.15.110:51850) 2024-04-27T03:18:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c2ba7640 1 -- 172.21.15.110:0/2569230611 learned_addr learned my addr 172.21.15.110:0/2569230611 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c2ba7640 1 -- 172.21.15.110:0/2569230611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b85a6dda0 con 0x556b85a6d440 2024-04-27T03:18:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a6d440 0x556b85a6d860 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00aab0 tx=0x7f54bc00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f53a7fff640 1 -- 172.21.15.110:0/2569230611 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc007d30 con 0x556b85a6d440 2024-04-27T03:18:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b85a6e840 con 0x556b85a6d440 2024-04-27T03:18:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f53a7fff640 1 -- 172.21.15.110:0/2569230611 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc01d070 con 0x556b85a6d440 2024-04-27T03:18:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f53a7fff640 1 -- 172.21.15.110:0/2569230611 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc018440 con 0x556b85a6d440 2024-04-27T03:18:11.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b85a6ed10 con 0x556b85a6d440 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f53a7fff640 1 -- 172.21.15.110:0/2569230611 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f54bc013070 con 0x556b85a6d440 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f53a7fff640 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f53a7fff640 1 -- 172.21.15.110:0/2569230611 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f54bc04b520 con 0x556b85a6d440 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.345+0000 7f54c23a6640 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 Processor -- start 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- start start 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c23a6640 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x556b85a5d890 tx=0x7f54b4031040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85a7aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b85a4bd30 con 0x556b85a7a5f0 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85a7aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85a7aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51862/0 (socket says 172.21.15.110:51862) 2024-04-27T03:18:11.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 -- 172.21.15.110:0/2901639347 learned_addr learned my addr 172.21.15.110:0/2901639347 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 -- 172.21.15.110:0/2901639347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b85a7afe0 con 0x556b85a7a5f0 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/2901639347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85a7aa10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f54bc049960 tx=0x7f54bc0186a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988207071f3146a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f53a5ffb640 1 -- 172.21.15.110:0/2901639347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc0225e0 con 0x556b85a7a5f0 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f53a5ffb640 1 -- 172.21.15.110:0/2901639347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc018bc0 con 0x556b85a7a5f0 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f53a5ffb640 1 -- 172.21.15.110:0/2901639347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc020360 con 0x556b85a7a5f0 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- 172.21.15.110:0/2901639347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 msgr2=0x556b85a7aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 --2- 172.21.15.110:0/2901639347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85a7aa10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f54bc049960 tx=0x7f54bc0186a0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- 172.21.15.110:0/2901639347 shutdown_connections 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 --2- 172.21.15.110:0/2901639347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85a7aa10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- 172.21.15.110:0/2901639347 >> 172.21.15.110:0/2901639347 conn(0x556b85a76980 msgr2=0x556b85a749e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- 172.21.15.110:0/2901639347 shutdown_connections 2024-04-27T03:18:11.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- 172.21.15.110:0/2901639347 wait complete. 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 Processor -- start 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- start start 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85960880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c488b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b85a7b750 con 0x556b85a7a5f0 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85960880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85960880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51868/0 (socket says 172.21.15.110:51868) 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 learned_addr learned my addr 172.21.15.110:0/3042780005 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.349+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b8595efa0 con 0x556b85a7a5f0 2024-04-27T03:18:11.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85960880 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f54bc0186e0 tx=0x7f54bc0498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f5393fff640 1 -- 172.21.15.110:0/3042780005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc031070 con 0x556b85a7a5f0 2024-04-27T03:18:11.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b8595f230 con 0x556b85a7a5f0 2024-04-27T03:18:11.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f5393fff640 1 -- 172.21.15.110:0/3042780005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc0225e0 con 0x556b85a7a5f0 2024-04-27T03:18:11.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f5393fff640 1 -- 172.21.15.110:0/3042780005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc020a30 con 0x556b85a7a5f0 2024-04-27T03:18:11.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b8595f700 con 0x556b85a7a5f0 2024-04-27T03:18:11.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f5393fff640 1 -- 172.21.15.110:0/3042780005 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f54bc034070 con 0x556b85a7a5f0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f5393fff640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f5393fff640 1 -- 172.21.15.110:0/3042780005 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f54bc037c90 con 0x556b85a7a5f0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c23a6640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b85a8f360 0x556b85a91820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a91f70 con 0x556b85a8f360 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c1ba5640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b85a8f360 0x556b85a91820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c23a6640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54b4047620 tx=0x7f54b403b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c1ba5640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b85a8f360 0x556b85a91820 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 realm 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a92340 con 0x556b85a8f360 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a92c10 con 0x556b85a8f360 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a93630 0x556b85a95a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.353+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a961b0 con 0x556b85a93630 2024-04-27T03:18:11.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a93630 0x556b85a95a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a93630 0x556b85a95a40 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c488b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a96a00 con 0x556b85a8f360 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c488b980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c488b980 20 searching for the correct realm 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b85a98770 0x556b85a9ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a98770 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c23a6640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b85a98770 0x556b85a9ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c23a6640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b85a98770 0x556b85a9ab60 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c2ba7640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc03b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x7f54bc036070 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.357+0000 7f54c1ba5640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc03b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc03b840 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a98770 2024-04-27T03:18:11.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x7f54bc036070 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.361+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x7f54bc036070 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a98770 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a98770 2024-04-27T03:18:11.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a98770 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.365+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x7f54bc036070 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a8f360 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x7f54bc036070 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a98770 2024-04-27T03:18:11.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x7f54bc036070 2024-04-27T03:18:11.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a98770 2024-04-27T03:18:11.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.369+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9b1e0 con 0x556b85a93630 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a97380 con 0x556b85a8f360 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a96f90 con 0x556b85a93630 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a97c70 con 0x556b85a8f360 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 zone default found 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 4 Realm: () 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 zonegroup default 2024-04-27T03:18:11.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9baf0 con 0x7f54bc036070 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9d740 con 0x556b85a98770 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9dbf0 con 0x556b85a98770 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 started sync module instance, tier type = 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:11.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.373+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9d270 con 0x556b85a93630 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93920292357856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9e050 con 0x556b85a93630 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9ea90 con 0x556b85a8f360 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93920292365664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9ea50 con 0x556b85a8f360 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9f9c0 con 0x556b85a93630 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93920292357856] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.377+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93920292365664] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93920292368000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9f9c0 con 0x556b85a93630 2024-04-27T03:18:11.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9fdf0 con 0x7f54bc036070 2024-04-27T03:18:11.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93920292359088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85a9fdb0 con 0x7f54bc036070 2024-04-27T03:18:11.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa0cd0 con 0x556b85a93630 2024-04-27T03:18:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93920292368000] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93920292359088] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93920292372960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa0cd0 con 0x556b85a93630 2024-04-27T03:18:11.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.381+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa10c0 con 0x556b85a98770 2024-04-27T03:18:11.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93920292369616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa1910 con 0x556b85a98770 2024-04-27T03:18:11.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa2310 con 0x556b85a98770 2024-04-27T03:18:11.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93920292372960] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93920292369616] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93920292374528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa2700 con 0x556b85a98770 2024-04-27T03:18:11.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa2f90 con 0x7f54bc036070 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93920292374528] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'108 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93920292380192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa2bb0 con 0x7f54bc036070 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 20 add_watcher() i=0 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 20 add_watcher() i=1 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 20 add_watcher() i=2 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 20 add_watcher() i=3 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 20 add_watcher() i=4 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 20 add_watcher() i=5 2024-04-27T03:18:11.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.385+0000 7f54c488b980 20 add_watcher() i=6 2024-04-27T03:18:11.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93920292380192] v27'109 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c488b980 20 add_watcher() i=7 2024-04-27T03:18:11.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c488b980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:11.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa6470 con 0x556b85a8f360 2024-04-27T03:18:11.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93920292382928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85aa3930 con 0x556b85a8f360 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93920292382928] v27'112 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f5378fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c488b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c488b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f54c488b980 5 note: GC not initialized 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.389+0000 7f52f37fe640 20 reqs_thread_entry: start 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 20 Filter name: none 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f52f17fa640 20 reqs_thread_entry: start 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac9c80 con 0x7f54bc036070 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 fud [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac82c0 con 0x556b85a98770 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 p5YnriCv1nAtykxBrupQ [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 206+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2024-04-27T03:18:11.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.e 8:75f48285:users.uid::fud:head [delete,create,call version.set in=58b,writefull 0~380 in=380b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac8b60 con 0x7f54bc036070 2024-04-27T03:18:11.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(64 fud [delete,create,call,writefull 0~380] v27'1 uv1 ondisk = 0) ==== 273+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2024-04-27T03:18:11.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:11.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2024-04-27T03:18:11.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:65 7.3 7:c609908c:::notify.5:head [notify cookie 93920292538032 in=568b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85acd720 con 0x556b85a98770 2024-04-27T03:18:11.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== watch-notify(notify (1) cookie 93920292369616 notify 115964117000 ret 0) ==== 598+0+0 (crc 0 0 0) 0x556b85a55a40 con 0x556b85a98770 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f53a5ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117000 cookie 93920292369616 notifier 4478 bl.length()=556 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f53a5ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f53a5ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(65 notify.5 [notify cookie 93920292538032 out=8b] v0'0 uv108 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f53a5ffb640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:66 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f5388003eb0 con 0x556b85a98770 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.393+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(66 notify.5 [notify-ack] v0'0 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== watch-notify(notify_complete (2) cookie 93920292538032 notify 115964117000 ret 0) ==== 42+0+48 (crc 0 0 0) 0x556b85a52dc0 con 0x556b85a98770 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac86b0 con 0x556b85a98770 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(67 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~7] v27'5 uv5 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c488b980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c488b980 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c488b980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:18:11.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify cookie 93920292538032 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556b85acd720 con 0x556b85a93630 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 93920292368000 notify 115964117010 ret 0) ==== 219+0+0 (crc 0 0 0) 0x556b8595f700 con 0x556b85a93630 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f53a5ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117010 cookie 93920292368000 notifier 4478 bl.length()=177 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f53a5ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f53a5ffb640 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f53a5ffb640 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f5388003eb0 con 0x556b85a93630 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [notify cookie 93920292538032 out=8b] v0'0 uv108 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.2 [notify-ack] v0'0 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 93920292538032 notify 115964117010 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f54bc063040 con 0x556b85a93630 2024-04-27T03:18:11.408 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"fud","display_name":"Fud","email":"","suspended":0,"max_buckets":1,"subusers":[],"keys":[{"user":"fud","access_key":"p5YnriCv1nAtykxBrupQ","secret_key":"Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh","active":true,"create_date":"2024-04-27T03:18:11.394675Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:11.394663Z","tags":[],"group_ids":[]} 2024-04-27T03:18:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f5378fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.397+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93920292382928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x556b85a8f360 2024-04-27T03:18:11.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.401+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 93920292382928] v27'113 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.401+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93920292357856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x556b85a93630 2024-04-27T03:18:11.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 93920292357856] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c488b980 20 remove_watcher() i=0 2024-04-27T03:18:11.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c488b980 2 removed watcher, disabling cache 2024-04-27T03:18:11.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93920292365664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x556b85a8f360 2024-04-27T03:18:11.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 93920292365664] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x556b85a8f360 2024-04-27T03:18:11.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c488b980 20 remove_watcher() i=1 2024-04-27T03:18:11.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93920292368000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x556b85a93630 2024-04-27T03:18:11.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 93920292368000] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c488b980 20 remove_watcher() i=2 2024-04-27T03:18:11.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.405+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93920292359088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x7f54bc036070 2024-04-27T03:18:11.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.409+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 93920292359088] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.409+0000 7f54c488b980 20 remove_watcher() i=3 2024-04-27T03:18:11.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.409+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93920292372960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x556b85a93630 2024-04-27T03:18:11.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.409+0000 7f54c2ba7640 1 -- 172.21.15.110:0/3042780005 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 93920292372960] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556b85a93630 2024-04-27T03:18:11.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.409+0000 7f54c488b980 20 remove_watcher() i=4 2024-04-27T03:18:11.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.409+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93920292369616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x556b85a98770 2024-04-27T03:18:11.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 93920292369616] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 20 remove_watcher() i=5 2024-04-27T03:18:11.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93920292374528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x556b85a98770 2024-04-27T03:18:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c23a6640 1 -- 172.21.15.110:0/3042780005 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 93920292374528] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b402c070 con 0x556b85a98770 2024-04-27T03:18:11.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 20 remove_watcher() i=6 2024-04-27T03:18:11.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93920292380192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556b85ac7920 con 0x7f54bc036070 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 93920292380192] v27'110 uv108 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b80b3040 con 0x7f54bc036070 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 20 remove_watcher() i=7 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a93630 msgr2=0x556b85a95a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a93630 0x556b85a95a40 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b85a8f360 msgr2=0x556b85a91820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b85a8f360 0x556b85a91820 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 msgr2=0x7f54bc03b840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc03b840 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b85a98770 msgr2=0x556b85a9ab60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b85a98770 0x556b85a9ab60 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 msgr2=0x7f538c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f54b4047620 tx=0x7f54b403b000 comp rx=0 tx=0).stop 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 msgr2=0x556b85960880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85960880 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f54bc0186e0 tx=0x7f54bc0498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c1ba5640 1 -- 172.21.15.110:0/3042780005 reap_dead start 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 shutdown_connections 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/3042780005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a7a5f0 0x556b85960880 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 >> 172.21.15.110:0/3042780005 conn(0x556b85a76980 msgr2=0x556b85a7be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 shutdown_connections 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/3042780005 wait complete. 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 msgr2=0x7f539c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x556b85a5d890 tx=0x7f54b4031040 comp rx=0 tx=0).stop 2024-04-27T03:18:11.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a6d440 msgr2=0x556b85a6d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a6d440 0x556b85a6d860 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00aab0 tx=0x7f54bc00af70 comp rx=0 tx=0).stop 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 shutdown_connections 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 --2- 172.21.15.110:0/2569230611 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b85a6d440 0x556b85a6d860 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 >> 172.21.15.110:0/2569230611 conn(0x556b85a62250 msgr2=0x556b85a62d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 shutdown_connections 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.413+0000 7f54c488b980 1 -- 172.21.15.110:0/2569230611 wait complete. 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a5a680 msgr2=0x556b85a5cb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a5a680 0x556b85a5cb20 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b859646a0 msgr2=0x556b85a59690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b859646a0 0x556b85a59690 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a8034650 msgr2=0x7f53a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a8034650 0x7f53a8036b10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f54b80ad000 tx=0x7f54b80a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:11.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 msgr2=0x556b85a525f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a525f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000f90 tx=0x7f54b4001260 comp rx=0 tx=0).stop 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 shutdown_connections 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b85a5a680 0x556b85a5cb20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b859646a0 0x556b85a59690 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a8034650 0x7f53a8036b10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 --2- 172.21.15.110:0/1990813253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b858d4890 0x556b85a525f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 >> 172.21.15.110:0/1990813253 conn(0x556b8595aa60 msgr2=0x556b85a45c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 shutdown_connections 2024-04-27T03:18:11.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.417+0000 7f54c488b980 1 -- 172.21.15.110:0/1990813253 wait complete. 2024-04-27T03:18:11.435 DEBUG:tasks.util.rgw: json result: {'user_id': 'fud', 'display_name': 'Fud', 'email': '', 'suspended': 0, 'max_buckets': 1, 'subusers': [], 'keys': [{'user': 'fud', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True, 'create_date': '2024-04-27T03:18:11.394675Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:11.394663Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:11.440 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2024-04-27T03:18:11.445 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:11.445 INFO:tasks.util.rgw:rgwadmin: client.0 : ['metadata', 'get', 'bucket:myfoo'] 2024-04-27T03:18:11.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'metadata', 'get', 'bucket:myfoo'] 2024-04-27T03:18:11.445 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph metadata get bucket:myfoo 2024-04-27T03:18:11.508 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:11.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f99109f5980 1 Processor -- start 2024-04-27T03:18:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f99109f5980 1 -- start start 2024-04-27T03:18:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f99109f5980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f012dfd150 0x55f012dfd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f99109f5980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55f012cbd0d0 con 0x55f012dfda70 2024-04-27T03:18:11.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f99109f5980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55f012cb2430 con 0x55f012dfd150 2024-04-27T03:18:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f990e510640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f012dfd150 0x55f012dfd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f990e510640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f012dfd150 0x55f012dfd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51872/0 (socket says 172.21.15.110:51872) 2024-04-27T03:18:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f990e510640 1 -- 172.21.15.110:0/2025650437 learned_addr learned my addr 172.21.15.110:0/2025650437 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f990dd0f640 1 --1- 172.21.15.110:0/2025650437 >> v1:172.21.15.77:6789/0 conn(0x55f012dfda70 0x55f012dfe480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34838/0 (socket says 172.21.15.110:34838) 2024-04-27T03:18:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f990e510640 1 -- 172.21.15.110:0/2025650437 >> v1:172.21.15.77:6789/0 conn(0x55f012dfda70 legacy=0x55f012dfe480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.501+0000 7f990e510640 1 -- 172.21.15.110:0/2025650437 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f012cab510 con 0x55f012dfd150 2024-04-27T03:18:11.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f990e510640 1 --2- 172.21.15.110:0/2025650437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012dfd530 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f990000a4e0 tx=0x7f990001af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19849679b9544918 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f990d50e640 1 -- 172.21.15.110:0/2025650437 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99000a3e60 con 0x55f012dfd150 2024-04-27T03:18:11.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f990d50e640 1 -- 172.21.15.110:0/2025650437 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99000a7070 con 0x55f012dfd150 2024-04-27T03:18:11.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f990d50e640 1 -- 172.21.15.110:0/2025650437 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99000ab540 con 0x55f012dfd150 2024-04-27T03:18:11.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f99109f5980 1 -- 172.21.15.110:0/2025650437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 msgr2=0x55f012dfd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f99109f5980 1 --2- 172.21.15.110:0/2025650437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012dfd530 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f990000a4e0 tx=0x7f990001af20 comp rx=0 tx=0).stop 2024-04-27T03:18:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f99109f5980 1 -- 172.21.15.110:0/2025650437 shutdown_connections 2024-04-27T03:18:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f99109f5980 1 --2- 172.21.15.110:0/2025650437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012dfd530 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f99109f5980 1 -- 172.21.15.110:0/2025650437 >> 172.21.15.110:0/2025650437 conn(0x55f012df8a20 msgr2=0x55f012dfae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f99109f5980 1 -- 172.21.15.110:0/2025650437 shutdown_connections 2024-04-27T03:18:11.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.505+0000 7f99109f5980 1 -- 172.21.15.110:0/2025650437 wait complete. 2024-04-27T03:18:11.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f99109f5980 1 Processor -- start 2024-04-27T03:18:11.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f99109f5980 1 -- start start 2024-04-27T03:18:11.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f99109f5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012ee7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f99109f5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f012d012d0 con 0x55f012dfd150 2024-04-27T03:18:11.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f990e510640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012ee7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f990e510640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012ee7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51874/0 (socket says 172.21.15.110:51874) 2024-04-27T03:18:11.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f990e510640 1 -- 172.21.15.110:0/1272779657 learned_addr learned my addr 172.21.15.110:0/1272779657 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f990e510640 1 -- 172.21.15.110:0/1272779657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f012c87ae0 con 0x55f012dfd150 2024-04-27T03:18:11.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.521+0000 7f990e510640 1 --2- 172.21.15.110:0/1272779657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012ee7930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9900018b50 tx=0x7f99000033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2f3f6f1f207c5f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f97fc608640 1 -- 172.21.15.110:0/1272779657 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99000ab540 con 0x55f012dfd150 2024-04-27T03:18:11.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f97fc608640 1 -- 172.21.15.110:0/1272779657 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99000ab6e0 con 0x55f012dfd150 2024-04-27T03:18:11.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f97fc608640 1 -- 172.21.15.110:0/1272779657 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99000ab9e0 con 0x55f012dfd150 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- 172.21.15.110:0/1272779657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 msgr2=0x55f012ee7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 --2- 172.21.15.110:0/1272779657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012ee7930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9900018b50 tx=0x7f99000033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- 172.21.15.110:0/1272779657 shutdown_connections 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 --2- 172.21.15.110:0/1272779657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012ee7930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- 172.21.15.110:0/1272779657 >> 172.21.15.110:0/1272779657 conn(0x55f012df8a20 msgr2=0x55f012ee34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- 172.21.15.110:0/1272779657 shutdown_connections 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- 172.21.15.110:0/1272779657 wait complete. 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 Processor -- start 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- start start 2024-04-27T03:18:11.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012eee850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f012d71f40 con 0x55f012dfd150 2024-04-27T03:18:11.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f990e510640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012eee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f990e510640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012eee850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51886/0 (socket says 172.21.15.110:51886) 2024-04-27T03:18:11.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f990e510640 1 -- 172.21.15.110:0/70313485 learned_addr learned my addr 172.21.15.110:0/70313485 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f990e510640 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f012eeed90 con 0x55f012dfd150 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f990e510640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012eee850 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f990001b7c0 tx=0x7f9900017cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f97fae05640 1 -- 172.21.15.110:0/70313485 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99000c3020 con 0x55f012dfd150 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f97fae05640 1 -- 172.21.15.110:0/70313485 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99000bf7b0 con 0x55f012dfd150 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f012eef020 con 0x55f012dfd150 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f97fae05640 1 -- 172.21.15.110:0/70313485 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99000bfab0 con 0x55f012dfd150 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.525+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f012ef1c00 con 0x55f012dfd150 2024-04-27T03:18:11.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f97fae05640 1 -- 172.21.15.110:0/70313485 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f99000c6020 con 0x55f012dfd150 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f97fae05640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f4034650 0x7f97f4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f97fae05640 1 -- 172.21.15.110:0/70313485 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f99000ba040 con 0x55f012dfd150 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990dd0f640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f4034650 0x7f97f4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012dfda70 0x55f012ef5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012ef5d90 con 0x55f012dfda70 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990ed11640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012dfda70 0x55f012ef5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990dd0f640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f4034650 0x7f97f4036b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b410 tx=0x7f98f8001580 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990ed11640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012dfda70 0x55f012ef5850 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990ed11640 1 -- 172.21.15.110:0/70313485 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f99080083f0 con 0x55f012dfda70 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012ef6b20 0x55f012ef8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012ef94b0 con 0x55f012ef6b20 2024-04-27T03:18:11.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990e510640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012ef6b20 0x55f012ef8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990e510640 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012ef6b20 0x55f012ef8f10 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f990e510640 1 -- 172.21.15.110:0/70313485 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f99000c8930 con 0x55f012ef6b20 2024-04-27T03:18:11.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.529+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012efad60 con 0x55f012dfda70 2024-04-27T03:18:11.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990ed11640 1 -- 172.21.15.110:0/70313485 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f99080083f0 con 0x55f012dfda70 2024-04-27T03:18:11.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012ef9f90 con 0x55f012ef6b20 2024-04-27T03:18:11.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990e510640 1 -- 172.21.15.110:0/70313485 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f99000c8930 con 0x55f012ef6b20 2024-04-27T03:18:11.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012efa380 con 0x55f012dfda70 2024-04-27T03:18:11.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990ed11640 1 -- 172.21.15.110:0/70313485 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f99080083f0 con 0x55f012dfda70 2024-04-27T03:18:11.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 Processor -- start 2024-04-27T03:18:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 -- start start 2024-04-27T03:18:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f09660 0x55f012f09a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f012d05310 con 0x55f012f09660 2024-04-27T03:18:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990ed11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f09660 0x55f012f09a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990ed11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f09660 0x55f012f09a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51892/0 (socket says 172.21.15.110:51892) 2024-04-27T03:18:11.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990ed11640 1 -- 172.21.15.110:0/2097812346 learned_addr learned my addr 172.21.15.110:0/2097812346 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990ed11640 1 -- 172.21.15.110:0/2097812346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f012f09fc0 con 0x55f012f09660 2024-04-27T03:18:11.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f990ed11640 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f09660 0x55f012f09a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f990800aab0 tx=0x7f990800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f97f3fff640 1 -- 172.21.15.110:0/2097812346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9908007d30 con 0x55f012f09660 2024-04-27T03:18:11.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f97f3fff640 1 -- 172.21.15.110:0/2097812346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f990801d070 con 0x55f012f09660 2024-04-27T03:18:11.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f012f0aa60 con 0x55f012f09660 2024-04-27T03:18:11.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f97f3fff640 1 -- 172.21.15.110:0/2097812346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9908018440 con 0x55f012f09660 2024-04-27T03:18:11.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.533+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f012f0af30 con 0x55f012f09660 2024-04-27T03:18:11.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f97f3fff640 1 -- 172.21.15.110:0/2097812346 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9908013070 con 0x55f012f09660 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f97f3fff640 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97e80346a0 0x7f97e8036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f97f3fff640 1 -- 172.21.15.110:0/2097812346 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f990804b5f0 con 0x55f012f09660 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f990e510640 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97e80346a0 0x7f97e8036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 Processor -- start 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 -- start start 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f16c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f012ee7fc0 con 0x55f012f16860 2024-04-27T03:18:11.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f990e510640 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97e80346a0 0x7f97e8036b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55f012ef9ab0 tx=0x7f99000da040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f990ed11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f16c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f990ed11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f16c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51904/0 (socket says 172.21.15.110:51904) 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f990ed11640 1 -- 172.21.15.110:0/54919781 learned_addr learned my addr 172.21.15.110:0/54919781 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f990ed11640 1 -- 172.21.15.110:0/54919781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f012f17230 con 0x55f012f16860 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f990ed11640 1 --2- 172.21.15.110:0/54919781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f16c60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9908049970 tx=0x7f990803a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3debe7ab9dbebe89 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f97f1ffb640 1 -- 172.21.15.110:0/54919781 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f990803f020 con 0x55f012f16860 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f97f1ffb640 1 -- 172.21.15.110:0/54919781 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f990803b310 con 0x55f012f16860 2024-04-27T03:18:11.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f97f1ffb640 1 -- 172.21.15.110:0/54919781 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9908022460 con 0x55f012f16860 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 -- 172.21.15.110:0/54919781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 msgr2=0x55f012f16c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 --2- 172.21.15.110:0/54919781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f16c60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9908049970 tx=0x7f990803a110 comp rx=0 tx=0).stop 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 -- 172.21.15.110:0/54919781 shutdown_connections 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 --2- 172.21.15.110:0/54919781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f16c60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 -- 172.21.15.110:0/54919781 >> 172.21.15.110:0/54919781 conn(0x55f012f12ba0 msgr2=0x55f012f10c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 -- 172.21.15.110:0/54919781 shutdown_connections 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.537+0000 7f99109f5980 1 -- 172.21.15.110:0/54919781 wait complete. 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 Processor -- start 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 -- start start 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f1dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f012f179d0 con 0x55f012f16860 2024-04-27T03:18:11.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f990ed11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f1dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f990ed11640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f1dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51914/0 (socket says 172.21.15.110:51914) 2024-04-27T03:18:11.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 learned_addr learned my addr 172.21.15.110:0/609356502 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f012f1e280 con 0x55f012f16860 2024-04-27T03:18:11.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f990ed11640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f1dd40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x55f012f132a0 tx=0x7f99080498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f97dffff640 1 -- 172.21.15.110:0/609356502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f990802f020 con 0x55f012f16860 2024-04-27T03:18:11.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f012f1e510 con 0x55f012f16860 2024-04-27T03:18:11.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f97dffff640 1 -- 172.21.15.110:0/609356502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9908020c80 con 0x55f012f16860 2024-04-27T03:18:11.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f97dffff640 1 -- 172.21.15.110:0/609356502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f990803f040 con 0x55f012f16860 2024-04-27T03:18:11.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f012f211f0 con 0x55f012f16860 2024-04-27T03:18:11.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f97dffff640 1 -- 172.21.15.110:0/609356502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9908032020 con 0x55f012f16860 2024-04-27T03:18:11.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f97dffff640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d8034650 0x7f97d8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f97dffff640 1 -- 172.21.15.110:0/609356502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9908035c40 con 0x55f012f16860 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f990e510640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d8034650 0x7f97d8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012f29330 0x55f012f2b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f2bf20 con 0x55f012f29330 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.541+0000 7f990dd0f640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012f29330 0x55f012f2b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990e510640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d8034650 0x7f97d8036b10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f990001af60 tx=0x7f99000ae000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990dd0f640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012f29330 0x55f012f2b7d0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98f800f7e0 con 0x55f012f29330 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 realm 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f28840 con 0x55f012f29330 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98f800f7e0 con 0x55f012f29330 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f2c9f0 con 0x55f012f29330 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98f800f7e0 con 0x55f012f29330 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012f2d5c0 0x55f012f2fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f301f0 con 0x55f012f2d5c0 2024-04-27T03:18:11.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990ed11640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012f2d5c0 0x55f012f2fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990ed11640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012f2d5c0 0x55f012f2fa80 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f30a90 con 0x55f012f29330 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f98f800f7e0 con 0x55f012f29330 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:11.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 20 searching for the correct realm 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f012f32800 0x55f012f34bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f32800 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990e510640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f012f32800 0x55f012f34bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.545+0000 7f990e510640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f012f32800 0x55f012f34bf0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990ed11640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9908056040 0x7f99080481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x7f9908056040 2024-04-27T03:18:11.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9908056040 0x7f99080481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9908056040 0x7f99080481f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f800f7e0 con 0x7f9908056040 2024-04-27T03:18:11.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f800f7e0 con 0x55f012f29330 2024-04-27T03:18:11.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f32800 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.549+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x7f9908056040 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x7f9908056040 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x7f9908056040 2024-04-27T03:18:11.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f98f802b040 con 0x7f9908056040 2024-04-27T03:18:11.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f32800 2024-04-27T03:18:11.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f32800 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.553+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f32800 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x7f9908056040 2024-04-27T03:18:11.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x7f9908056040 2024-04-27T03:18:11.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f29330 2024-04-27T03:18:11.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f802b040 con 0x55f012f29330 2024-04-27T03:18:11.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x7f9908056040 2024-04-27T03:18:11.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f32800 2024-04-27T03:18:11.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x7f9908056040 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.557+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f32800 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f012f351d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f31440 con 0x55f012f29330 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f318c0 con 0x55f012f2d5c0 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f31d20 con 0x55f012f29330 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 zone default found 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 4 Realm: () 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 zonegroup default 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f35b00 con 0x7f9908056040 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f374e0 con 0x55f012f32800 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.561+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f379c0 con 0x55f012f32800 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 20 started sync module instance, tier type = 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:11.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f36970 con 0x55f012f2d5c0 2024-04-27T03:18:11.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94489598455136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f382d0 con 0x55f012f2d5c0 2024-04-27T03:18:11.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f38b20 con 0x55f012f29330 2024-04-27T03:18:11.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94489598460624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f38c60 con 0x55f012f29330 2024-04-27T03:18:11.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.565+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f39780 con 0x55f012f2d5c0 2024-04-27T03:18:11.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94489598455136] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94489598460624] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94489598461680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f39740 con 0x55f012f2d5c0 2024-04-27T03:18:11.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3a020 con 0x7f9908056040 2024-04-27T03:18:11.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94489598466928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f39740 con 0x7f9908056040 2024-04-27T03:18:11.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3aad0 con 0x55f012f2d5c0 2024-04-27T03:18:11.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94489598466928] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.569+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94489598461680] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94489598464592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3aad0 con 0x55f012f2d5c0 2024-04-27T03:18:11.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3aec0 con 0x55f012f32800 2024-04-27T03:18:11.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94489598467984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3b710 con 0x55f012f32800 2024-04-27T03:18:11.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3c110 con 0x55f012f32800 2024-04-27T03:18:11.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94489598464592] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94489598467984] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94489598472800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3bc20 con 0x55f012f32800 2024-04-27T03:18:11.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.573+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3cb20 con 0x7f9908056040 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94489598472800] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'111 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94489598469600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3cb20 con 0x7f9908056040 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=0 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=1 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=2 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=3 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=4 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=5 2024-04-27T03:18:11.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=6 2024-04-27T03:18:11.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94489598469600] v27'112 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 20 add_watcher() i=7 2024-04-27T03:18:11.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:11.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3fdb0 con 0x55f012f29330 2024-04-27T03:18:11.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.577+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94489598478304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f3d0b0 con 0x55f012f29330 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94489598478304] v27'114 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f97c4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 5 note: GC not initialized 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f973f7fe640 20 reqs_thread_entry: start 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:11.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 20 Filter name: none 2024-04-27T03:18:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f973d7fa640 20 reqs_thread_entry: start 2024-04-27T03:18:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f012f63120 con 0x7f9908056040 2024-04-27T03:18:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 myfoo [call out=48b,stat out=16b,read 0~165 out=165b,getxattrs out=75b] v0'0 uv18 ondisk = 0) ==== 275+0+304 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f97c4fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.581+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94489598478304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x55f012f29330 2024-04-27T03:18:11.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.585+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94489598478304] v27'115 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.585+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94489598455136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x55f012f2d5c0 2024-04-27T03:18:11.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.585+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94489598455136] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.585+0000 7f99109f5980 20 remove_watcher() i=0 2024-04-27T03:18:11.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.585+0000 7f99109f5980 2 removed watcher, disabling cache 2024-04-27T03:18:11.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.585+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94489598460624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x55f012f29330 2024-04-27T03:18:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.585+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94489598460624] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x55f012f29330 2024-04-27T03:18:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f99109f5980 20 remove_watcher() i=1 2024-04-27T03:18:11.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94489598461680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x55f012f2d5c0 2024-04-27T03:18:11.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94489598461680] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f99109f5980 20 remove_watcher() i=2 2024-04-27T03:18:11.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94489598466928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f38780 con 0x7f9908056040 2024-04-27T03:18:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94489598466928] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f99109f5980 20 remove_watcher() i=3 2024-04-27T03:18:11.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.589+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94489598464592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x55f012f2d5c0 2024-04-27T03:18:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f990ed11640 1 -- 172.21.15.110:0/609356502 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94489598464592] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9908053040 con 0x55f012f2d5c0 2024-04-27T03:18:11.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f99109f5980 20 remove_watcher() i=4 2024-04-27T03:18:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94489598467984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x55f012f32800 2024-04-27T03:18:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94489598467984] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f99109f5980 20 remove_watcher() i=5 2024-04-27T03:18:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94489598472800] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x55f012f32800 2024-04-27T03:18:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f990e510640 1 -- 172.21.15.110:0/609356502 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94489598472800] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99000eb040 con 0x55f012f32800 2024-04-27T03:18:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f99109f5980 20 remove_watcher() i=6 2024-04-27T03:18:11.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94489598469600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f012f60f40 con 0x7f9908056040 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.593+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94489598469600] v27'113 uv111 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f98f803e040 con 0x7f9908056040 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 20 remove_watcher() i=7 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012f2d5c0 msgr2=0x55f012f2fa80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012f2d5c0 0x55f012f2fa80 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012f29330 msgr2=0x55f012f2b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012f29330 0x55f012f2b7d0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9908056040 msgr2=0x7f99080481f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9908056040 0x7f99080481f0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f012f32800 msgr2=0x55f012f34bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f012f32800 0x55f012f34bf0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d8034650 msgr2=0x7f97d8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97d8034650 0x7f97d8036b10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f990001af60 tx=0x7f99000ae000 comp rx=0 tx=0).stop 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 msgr2=0x55f012f1dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f990dd0f640 1 -- 172.21.15.110:0/609356502 reap_dead start 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f1dd40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x55f012f132a0 tx=0x7f99080498d0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 shutdown_connections 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/609356502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f16860 0x55f012f1dd40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 >> 172.21.15.110:0/609356502 conn(0x55f012f12ba0 msgr2=0x55f012f10200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 shutdown_connections 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/609356502 wait complete. 2024-04-27T03:18:11.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97e80346a0 msgr2=0x7f97e8036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97e80346a0 0x7f97e8036b60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55f012ef9ab0 tx=0x7f99000da040 comp rx=0 tx=0).stop 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f09660 msgr2=0x55f012f09a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f09660 0x55f012f09a80 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f990800aab0 tx=0x7f990800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 shutdown_connections 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97e80346a0 0x7f97e8036b60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/2097812346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012f09660 0x55f012f09a80 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 >> 172.21.15.110:0/2097812346 conn(0x55f012efe470 msgr2=0x55f012efef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 shutdown_connections 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/2097812346 wait complete. 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012ef6b20 msgr2=0x55f012ef8f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012ef6b20 0x55f012ef8f10 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012dfda70 msgr2=0x55f012ef5850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012dfda70 0x55f012ef5850 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f4034650 msgr2=0x7f97f4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f4034650 0x7f97f4036b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98f800b410 tx=0x7f98f8001580 comp rx=0 tx=0).stop 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 msgr2=0x55f012eee850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012eee850 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f990001b7c0 tx=0x7f9900017cc0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 shutdown_connections 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f012ef6b20 0x55f012ef8f10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f012dfda70 0x55f012ef5850 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f97f4034650 0x7f97f4036b10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 --2- 172.21.15.110:0/70313485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f012dfd150 0x55f012eee850 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 >> 172.21.15.110:0/70313485 conn(0x55f012df8a20 msgr2=0x55f012ee1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 shutdown_connections 2024-04-27T03:18:11.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.597+0000 7f99109f5980 1 -- 172.21.15.110:0/70313485 wait complete. 2024-04-27T03:18:11.615 INFO:teuthology.orchestra.run.smithi110.stdout:{"key":"bucket:myfoo","ver":{"tag":"_SbMQ0tae_-8Qb48fIa1uR7x","ver":2},"mtime":"2024-04-27T03:18:11.196356Z","data":{"bucket":{"name":"myfoo","marker":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","bucket_id":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","tenant":"","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""}},"owner":"foo","creation_time":"2024-04-27T03:18:09.694744Z","linked":false,"has_bucket_info":false}} 2024-04-27T03:18:11.615 DEBUG:tasks.util.rgw: json result: {'key': 'bucket:myfoo', 'ver': {'tag': '_SbMQ0tae_-8Qb48fIa1uR7x', 'ver': 2}, 'mtime': '2024-04-27T03:18:11.196356Z', 'data': {'bucket': {'name': 'myfoo', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'bucket_id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'tenant': '', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}}, 'owner': 'foo', 'creation_time': '2024-04-27T03:18:09.694744Z', 'linked': False, 'has_bucket_info': False}} 2024-04-27T03:18:11.615 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--uid', 'fud', '--bucket', 'myfoo', '--bucket-id', '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1'] 2024-04-27T03:18:11.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--uid', 'fud', '--bucket', 'myfoo', '--bucket-id', '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1'] 2024-04-27T03:18:11.616 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --uid fud --bucket myfoo --bucket-id 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:11.690 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:11.691 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.681+0000 7f2a7d186980 1 Processor -- start 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7d186980 1 -- start start 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7d186980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55affeb89230 0x55affeb89610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7d186980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55affea480d0 con 0x55affeb89b50 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7d186980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55affea3d430 con 0x55affeb89230 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7aca1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55affeb89230 0x55affeb89610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7aca1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55affeb89230 0x55affeb89610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51928/0 (socket says 172.21.15.110:51928) 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3893779504 learned_addr learned my addr 172.21.15.110:0/3893779504 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3893779504 >> v1:172.21.15.77:6789/0 conn(0x55affeb89b50 legacy=0x55affeb8a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3893779504 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affea36510 con 0x55affeb89230 2024-04-27T03:18:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/3893779504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affeb89610 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00a5c0 tx=0x7f2a6c01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d50fb1cca16622 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a79c9f640 1 -- 172.21.15.110:0/3893779504 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a6c0a6070 con 0x55affeb89230 2024-04-27T03:18:11.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a79c9f640 1 -- 172.21.15.110:0/3893779504 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c0af070 con 0x55affeb89230 2024-04-27T03:18:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.685+0000 7f2a79c9f640 1 -- 172.21.15.110:0/3893779504 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a6c0aa5c0 con 0x55affeb89230 2024-04-27T03:18:11.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.689+0000 7f2a7d186980 1 -- 172.21.15.110:0/3893779504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 msgr2=0x55affeb89610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.689+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3893779504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affeb89610 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00a5c0 tx=0x7f2a6c01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.689+0000 7f2a7d186980 1 -- 172.21.15.110:0/3893779504 shutdown_connections 2024-04-27T03:18:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.689+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3893779504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affeb89610 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.689+0000 7f2a7d186980 1 -- 172.21.15.110:0/3893779504 >> 172.21.15.110:0/3893779504 conn(0x55affeb84b00 msgr2=0x55affeb86f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.689+0000 7f2a7d186980 1 -- 172.21.15.110:0/3893779504 shutdown_connections 2024-04-27T03:18:11.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.689+0000 7f2a7d186980 1 -- 172.21.15.110:0/3893779504 wait complete. 2024-04-27T03:18:11.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7d186980 1 Processor -- start 2024-04-27T03:18:11.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7d186980 1 -- start start 2024-04-27T03:18:11.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7d186980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec74b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7d186980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55affea8c2d0 con 0x55affeb89230 2024-04-27T03:18:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7aca1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec74b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7aca1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec74b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51938/0 (socket says 172.21.15.110:51938) 2024-04-27T03:18:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7aca1640 1 -- 172.21.15.110:0/2978773317 learned_addr learned my addr 172.21.15.110:0/2978773317 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7aca1640 1 -- 172.21.15.110:0/2978773317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affea12ae0 con 0x55affeb89230 2024-04-27T03:18:11.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.701+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/2978773317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec74b70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00a5c0 tx=0x7f2a6c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b18cad2c51e61b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2968e09640 1 -- 172.21.15.110:0/2978773317 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a6c0aa9b0 con 0x55affeb89230 2024-04-27T03:18:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2968e09640 1 -- 172.21.15.110:0/2978773317 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c0aab50 con 0x55affeb89230 2024-04-27T03:18:11.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2968e09640 1 -- 172.21.15.110:0/2978773317 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a6c0aae50 con 0x55affeb89230 2024-04-27T03:18:11.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- 172.21.15.110:0/2978773317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 msgr2=0x55affec74b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 --2- 172.21.15.110:0/2978773317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec74b70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00a5c0 tx=0x7f2a6c003030 comp rx=0 tx=0).stop 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- 172.21.15.110:0/2978773317 shutdown_connections 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 --2- 172.21.15.110:0/2978773317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec74b70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- 172.21.15.110:0/2978773317 >> 172.21.15.110:0/2978773317 conn(0x55affeb84b00 msgr2=0x55affec70700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- 172.21.15.110:0/2978773317 shutdown_connections 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- 172.21.15.110:0/2978773317 wait complete. 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 Processor -- start 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- start start 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec7baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55affeb93280 con 0x55affeb89230 2024-04-27T03:18:11.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7aca1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec7baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7aca1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec7baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51952/0 (socket says 172.21.15.110:51952) 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7aca1640 1 -- 172.21.15.110:0/867843057 learned_addr learned my addr 172.21.15.110:0/867843057 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7aca1640 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affec7c030 con 0x55affeb89230 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec7baf0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00a5c0 tx=0x7f2a6c0180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2967606640 1 -- 172.21.15.110:0/867843057 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a6c0c6750 con 0x55affeb89230 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2967606640 1 -- 172.21.15.110:0/867843057 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6c0c68f0 con 0x55affeb89230 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2967606640 1 -- 172.21.15.110:0/867843057 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a6c0c6bd0 con 0x55affeb89230 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55affec7c2c0 con 0x55affeb89230 2024-04-27T03:18:11.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.705+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55affec7eef0 con 0x55affeb89230 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2967606640 1 -- 172.21.15.110:0/867843057 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2a6c0ce020 con 0x55affeb89230 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2967606640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2960034650 0x7f2960036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2967606640 1 -- 172.21.15.110:0/867843057 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2a6c0cc020 con 0x55affeb89230 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7a4a0640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2960034650 0x7f2960036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affeb89b50 0x55affec829d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affec82ff0 con 0x55affeb89b50 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affeb89b50 0x55affec829d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7a4a0640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2960034650 0x7f2960036b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a70000f70 tx=0x7f2a70001420 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affeb89b50 0x55affec829d0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/867843057 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2a7400a4d0 con 0x55affeb89b50 2024-04-27T03:18:11.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affec83de0 0x55affec861d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affec86830 con 0x55affec83de0 2024-04-27T03:18:11.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affec83de0 0x55affec861d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affec83de0 0x55affec861d0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7aca1640 1 -- 172.21.15.110:0/867843057 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affec83de0 2024-04-27T03:18:11.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affec880e0 con 0x55affeb89b50 2024-04-27T03:18:11.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/867843057 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f2a7400a4d0 con 0x55affeb89b50 2024-04-27T03:18:11.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.709+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affec87310 con 0x55affec83de0 2024-04-27T03:18:11.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7aca1640 1 -- 172.21.15.110:0/867843057 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affec83de0 2024-04-27T03:18:11.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affec87700 con 0x55affeb89b50 2024-04-27T03:18:11.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/867843057 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f2a7400a4d0 con 0x55affeb89b50 2024-04-27T03:18:11.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7d186980 1 Processor -- start 2024-04-27T03:18:11.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7d186980 1 -- start start 2024-04-27T03:18:11.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7d186980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affec96a20 0x55affec96e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7d186980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55affea90310 con 0x55affec96a20 2024-04-27T03:18:11.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7b4a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affec96a20 0x55affec96e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7b4a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affec96a20 0x55affec96e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51966/0 (socket says 172.21.15.110:51966) 2024-04-27T03:18:11.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3076037591 learned_addr learned my addr 172.21.15.110:0/3076037591 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3076037591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affec97380 con 0x55affec96a20 2024-04-27T03:18:11.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affec96a20 0x55affec96e40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400dfc0 tx=0x7f2a740178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2964e01640 1 -- 172.21.15.110:0/3076037591 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a740157d0 con 0x55affec96a20 2024-04-27T03:18:11.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2964e01640 1 -- 172.21.15.110:0/3076037591 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a74015970 con 0x55affec96a20 2024-04-27T03:18:11.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55affec97e20 con 0x55affec96a20 2024-04-27T03:18:11.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2964e01640 1 -- 172.21.15.110:0/3076037591 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a74015c70 con 0x55affec96a20 2024-04-27T03:18:11.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.713+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55affec982f0 con 0x55affec96a20 2024-04-27T03:18:11.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2964e01640 1 -- 172.21.15.110:0/3076037591 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2a74029440 con 0x55affec96a20 2024-04-27T03:18:11.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2964e01640 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29540346a0 0x7f2954036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2964e01640 1 -- 172.21.15.110:0/3076037591 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2a7400e040 con 0x55affec96a20 2024-04-27T03:18:11.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29540346a0 0x7f2954036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 Processor -- start 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- start start 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29540346a0 0x7f2954036b60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55affec86e30 tx=0x7f2a6c0d0d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affeca4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55affec75200 con 0x55affeca3bf0 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affeca4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affeca4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51970/0 (socket says 172.21.15.110:51970) 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/4218355767 learned_addr learned my addr 172.21.15.110:0/4218355767 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/4218355767 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affeca45e0 con 0x55affeca3bf0 2024-04-27T03:18:11.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/4218355767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affeca4010 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2a74015f40 tx=0x7f2a7403c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58ceaf6f6fd97f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f295e7fc640 1 -- 172.21.15.110:0/4218355767 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a740283a0 con 0x55affeca3bf0 2024-04-27T03:18:11.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f295e7fc640 1 -- 172.21.15.110:0/4218355767 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a74028540 con 0x55affeca3bf0 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f295e7fc640 1 -- 172.21.15.110:0/4218355767 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a7404c240 con 0x55affeca3bf0 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- 172.21.15.110:0/4218355767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 msgr2=0x55affeca4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 --2- 172.21.15.110:0/4218355767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affeca4010 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f2a74015f40 tx=0x7f2a7403c280 comp rx=0 tx=0).stop 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- 172.21.15.110:0/4218355767 shutdown_connections 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 --2- 172.21.15.110:0/4218355767 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affeca4010 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- 172.21.15.110:0/4218355767 >> 172.21.15.110:0/4218355767 conn(0x55affec9ff30 msgr2=0x55affec9df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- 172.21.15.110:0/4218355767 shutdown_connections 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- 172.21.15.110:0/4218355767 wait complete. 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 Processor -- start 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- start start 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affecab040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7d186980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55affeca4d40 con 0x55affeca3bf0 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affecab040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affecab040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51980/0 (socket says 172.21.15.110:51980) 2024-04-27T03:18:11.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.717+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 learned_addr learned my addr 172.21.15.110:0/3033253947 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affecab580 con 0x55affeca3bf0 2024-04-27T03:18:11.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affecab040 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2a74012750 tx=0x7f2a7404a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f295cff9640 1 -- 172.21.15.110:0/3033253947 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a74036070 con 0x55affeca3bf0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55affecab810 con 0x55affeca3bf0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f295cff9640 1 -- 172.21.15.110:0/3033253947 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a74049460 con 0x55affeca3bf0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55affecae4f0 con 0x55affeca3bf0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f295cff9640 1 -- 172.21.15.110:0/3033253947 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a74041040 con 0x55affeca3bf0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f295cff9640 1 -- 172.21.15.110:0/3033253947 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2a74039070 con 0x55affeca3bf0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f295cff9640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2944034650 0x7f2944036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f295cff9640 1 -- 172.21.15.110:0/3033253947 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2a7403d610 con 0x55affeca3bf0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2944034650 0x7f2944036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affecb66e0 0x55affecb8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecb92d0 con 0x55affecb66e0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7a4a0640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affecb66e0 0x55affecb8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2944034650 0x7f2944036b10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c01b170 tx=0x7f2a6c0e8000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7a4a0640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affecb66e0 0x55affecb8b80 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2a7000d040 con 0x55affecb66e0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 20 realm 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecb5bf0 con 0x55affecb66e0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2a7000d040 con 0x55affecb66e0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.721+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecb9dd0 con 0x55affecb66e0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2a7000d040 con 0x55affecb66e0 2024-04-27T03:18:11.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affecbabb0 0x55affecbcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecbd670 con 0x55affecbabb0 2024-04-27T03:18:11.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affecbabb0 0x55affecbcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affecbabb0 0x55affecbcfa0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecbdec0 con 0x55affecb66e0 2024-04-27T03:18:11.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f2a7000d040 con 0x55affecb66e0 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 20 searching for the correct realm 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55affecbfc30 0x55affecc2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbfc30 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55affecbfc30 0x55affecc2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7aca1640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55affecbfc30 0x55affecc2020 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7b4a2640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2a74014030 0x7f2a7401f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x7f2a74014030 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.725+0000 7f2a7a4a0640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2a74014030 0x7f2a7401f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2a74014030 0x7f2a7401f720 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7000d040 con 0x7f2a74014030 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7000d040 con 0x55affecb66e0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbfc30 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x7f2a74014030 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x7f2a74014030 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.729+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x7f2a74014030 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f2a7002a040 con 0x7f2a74014030 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbfc30 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbfc30 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbfc30 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x7f2a74014030 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x7f2a74014030 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecb66e0 2024-04-27T03:18:11.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7002a040 con 0x55affecb66e0 2024-04-27T03:18:11.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.733+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x7f2a74014030 2024-04-27T03:18:11.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbfc30 2024-04-27T03:18:11.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x7f2a74014030 2024-04-27T03:18:11.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbfc30 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55affecc2650 con 0x55affecbabb0 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:11.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecbe870 con 0x55affecb66e0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x55affecb66e0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.737+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecbf110 con 0x55affecbabb0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecbe4b0 con 0x55affecb66e0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f2a7003d040 con 0x55affecb66e0 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 zone default found 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 4 Realm: () 2024-04-27T03:18:11.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 zonegroup default 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecbed60 con 0x7f2a74014030 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecc4eb0 con 0x55affecbfc30 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecc5250 con 0x55affecbfc30 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 started sync module instance, tier type = 2024-04-27T03:18:11.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:11.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc42f0 con 0x55affecbabb0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94214382438160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc3f50 con 0x55affecbabb0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.741+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc60c0 con 0x55affecb66e0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.745+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x55affecb66e0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.745+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94214382443536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc64e0 con 0x55affecb66e0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.745+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc6ee0 con 0x55affecbabb0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.745+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94214382438160] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.745+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94214382443536] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x55affecb66e0 2024-04-27T03:18:11.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94214382441968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc6030 con 0x55affecbabb0 2024-04-27T03:18:11.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc7780 con 0x7f2a74014030 2024-04-27T03:18:11.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94214382449360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc6030 con 0x7f2a74014030 2024-04-27T03:18:11.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc8250 con 0x55affecbabb0 2024-04-27T03:18:11.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94214382449360] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94214382441968] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94214382447088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc8210 con 0x55affecbabb0 2024-04-27T03:18:11.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc8640 con 0x55affecbfc30 2024-04-27T03:18:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94214382450416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc8e90 con 0x55affecbfc30 2024-04-27T03:18:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.749+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc9890 con 0x55affecbfc30 2024-04-27T03:18:11.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94214382447088] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94214382450416] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94214382452032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc89e0 con 0x55affecbfc30 2024-04-27T03:18:11.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecca130 con 0x7f2a74014030 2024-04-27T03:18:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94214382452032] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'114 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94214382457760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecc89e0 con 0x7f2a74014030 2024-04-27T03:18:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=0 2024-04-27T03:18:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=1 2024-04-27T03:18:11.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=2 2024-04-27T03:18:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=3 2024-04-27T03:18:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=4 2024-04-27T03:18:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=5 2024-04-27T03:18:11.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=6 2024-04-27T03:18:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94214382457760] v27'115 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 20 add_watcher() i=7 2024-04-27T03:18:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affeccd280 con 0x55affecb66e0 2024-04-27T03:18:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f2a7003d040 con 0x55affecb66e0 2024-04-27T03:18:11.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.753+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94214382460032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecca740 con 0x55affecb66e0 2024-04-27T03:18:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94214382460032] v27'116 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x55affecb66e0 2024-04-27T03:18:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2931fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 5 note: GC not initialized 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2930fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 Filter name: none 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2929ffb640 20 reqs_thread_entry: start 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecf0ae0 con 0x7f2a74014030 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 fud [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2024-04-27T03:18:11.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecf2bc0 con 0x7f2a74014030 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv18 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecf45f0 con 0x7f2a74014030 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(64 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv3 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:11.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 10 cache get: name=default.rgw.meta+users.uid+fud : hit (requested=0x13, cached=0x13) 2024-04-27T03:18:11.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.757+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecf8c40 con 0x7f2a74014030 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(65 foo.buckets [call] v27'19 uv17 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~379 in=379b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecfeb00 con 0x7f2a74014030 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(66 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [delete,create,call,call,writefull 0~379,setxattr (123)] v27'4 uv4 ondisk = 0) ==== 416+0+0 (crc 0 0 0) 0x7f2a7003d040 con 0x7f2a74014030 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 10 moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [notify cookie 94214382632608 in=764b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecf9070 con 0x55affecbabb0 2024-04-27T03:18:11.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94214382441968 notify 115964117011 ret 0) ==== 794+0+0 (crc 0 0 0) 0x55affecae4f0 con 0x55affecbabb0 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f295e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117011 cookie 94214382441968 notifier 4492 bl.length()=752 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f295e7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f295e7fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f295e7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f295e7fc640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f2940003eb0 con 0x55affecbabb0 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [notify cookie 94214382632608 out=8b] v0'0 uv114 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.2 [notify-ack] v0'0 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94214382632608 notify 115964117011 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55affecab810 con 0x55affecbabb0 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_sources= new_sources= 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential sources added=[] removed=[] 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_dests= new_dests= 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential dests added=[] removed=[] 2024-04-27T03:18:11.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.761+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 8.1 8:810902b1:users.uid::fud.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecf3ea0 con 0x55affecb66e0 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 fud.buckets [call] v27'1 uv1 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f2a70049040 con 0x55affecb66e0 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 8.9 8:91b71841:root::myfoo:head [delete,create,call version.inc in=38b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecf3ea0 con 0x7f2a74014030 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(70 myfoo [delete,create,call,writefull 0~165] v27'20 uv20 ondisk = 0) ==== 275+0+0 (crc 0 0 0) 0x7f2a70049040 con 0x7f2a74014030 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7d186980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7d186980 10 moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7d186980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94214382632608 in=326b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55affecfd810 con 0x55affecbfc30 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== watch-notify(notify (1) cookie 94214382452032 notify 115964117001 ret 0) ==== 356+0+0 (crc 0 0 0) 0x55affec7c2c0 con 0x55affecbfc30 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f295dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117001 cookie 94214382452032 notifier 4492 bl.length()=314 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f295dffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2024-04-27T03:18:11.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f295dffb640 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:11.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f295dffb640 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f28a4003410 con 0x55affecbfc30 2024-04-27T03:18:11.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(71 notify.6 [notify cookie 94214382632608 out=8b] v0'0 uv114 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(72 notify.6 [notify-ack] v0'0 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== watch-notify(notify_complete (2) cookie 94214382632608 notify 115964117001 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55affec7c030 con 0x55affecbfc30 2024-04-27T03:18:11.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2931fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.765+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:73 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94214382460032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x55affecb66e0 2024-04-27T03:18:11.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.769+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(73 data_loggenerations_metadata [watch unwatch cookie 94214382460032] v27'117 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f2a70049040 con 0x55affecb66e0 2024-04-27T03:18:11.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.769+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94214382438160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x55affecbabb0 2024-04-27T03:18:11.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.769+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(74 notify.0 [watch unwatch cookie 94214382438160] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.769+0000 7f2a7d186980 20 remove_watcher() i=0 2024-04-27T03:18:11.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.769+0000 7f2a7d186980 2 removed watcher, disabling cache 2024-04-27T03:18:11.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.769+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94214382443536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x55affecb66e0 2024-04-27T03:18:11.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 notify.1 [watch unwatch cookie 94214382443536] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a70049040 con 0x55affecb66e0 2024-04-27T03:18:11.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7d186980 20 remove_watcher() i=1 2024-04-27T03:18:11.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:76 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94214382441968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x55affecbabb0 2024-04-27T03:18:11.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(76 notify.2 [watch unwatch cookie 94214382441968] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7d186980 20 remove_watcher() i=2 2024-04-27T03:18:11.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:77 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94214382449360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x7f2a74014030 2024-04-27T03:18:11.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(77 notify.3 [watch unwatch cookie 94214382449360] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a70049040 con 0x7f2a74014030 2024-04-27T03:18:11.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7d186980 20 remove_watcher() i=3 2024-04-27T03:18:11.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.773+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94214382447088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x55affecbabb0 2024-04-27T03:18:11.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.777+0000 7f2a7b4a2640 1 -- 172.21.15.110:0/3033253947 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(78 notify.4 [watch unwatch cookie 94214382447088] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a74038030 con 0x55affecbabb0 2024-04-27T03:18:11.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.777+0000 7f2a7d186980 20 remove_watcher() i=4 2024-04-27T03:18:11.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.777+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94214382450416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x55affecbfc30 2024-04-27T03:18:11.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.777+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(79 notify.5 [watch unwatch cookie 94214382450416] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.777+0000 7f2a7d186980 20 remove_watcher() i=5 2024-04-27T03:18:11.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.777+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:80 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94214382452032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x55affecbfc30 2024-04-27T03:18:11.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7aca1640 1 -- 172.21.15.110:0/3033253947 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(80 notify.6 [watch unwatch cookie 94214382452032] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a6c0c6d70 con 0x55affecbfc30 2024-04-27T03:18:11.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 20 remove_watcher() i=6 2024-04-27T03:18:11.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:81 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94214382457760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55affecee810 con 0x7f2a74014030 2024-04-27T03:18:11.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(81 notify.7 [watch unwatch cookie 94214382457760] v27'116 uv114 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2a70049040 con 0x7f2a74014030 2024-04-27T03:18:11.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 20 remove_watcher() i=7 2024-04-27T03:18:11.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affecbabb0 msgr2=0x55affecbcfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affecbabb0 0x55affecbcfa0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affecb66e0 msgr2=0x55affecb8b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affecb66e0 0x55affecb8b80 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2a74014030 msgr2=0x7f2a7401f720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2a74014030 0x7f2a7401f720 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55affecbfc30 msgr2=0x55affecc2020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55affecbfc30 0x55affecc2020 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2944034650 msgr2=0x7f2944036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2944034650 0x7f2944036b10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c01b170 tx=0x7f2a6c0e8000 comp rx=0 tx=0).stop 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 msgr2=0x55affecab040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7a4a0640 1 -- 172.21.15.110:0/3033253947 reap_dead start 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affecab040 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2a74012750 tx=0x7f2a7404a850 comp rx=0 tx=0).stop 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 shutdown_connections 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3033253947 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeca3bf0 0x55affecab040 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 >> 172.21.15.110:0/3033253947 conn(0x55affec9ff30 msgr2=0x55affec9d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 shutdown_connections 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3033253947 wait complete. 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29540346a0 msgr2=0x7f2954036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29540346a0 0x7f2954036b60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x55affec86e30 tx=0x7f2a6c0d0d30 comp rx=0 tx=0).stop 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affec96a20 msgr2=0x55affec96e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affec96a20 0x55affec96e40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2a7400dfc0 tx=0x7f2a740178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 shutdown_connections 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29540346a0 0x7f2954036b60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/3076037591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affec96a20 0x55affec96e40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 >> 172.21.15.110:0/3076037591 conn(0x55affec8b830 msgr2=0x55affec8c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 shutdown_connections 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/3076037591 wait complete. 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affec83de0 msgr2=0x55affec861d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affec83de0 0x55affec861d0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affeb89b50 msgr2=0x55affec829d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affeb89b50 0x55affec829d0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2960034650 msgr2=0x7f2960036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2960034650 0x7f2960036b10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a70000f70 tx=0x7f2a70001420 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 msgr2=0x55affec7baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec7baf0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00a5c0 tx=0x7f2a6c0180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 shutdown_connections 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55affec83de0 0x55affec861d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55affeb89b50 0x55affec829d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2960034650 0x7f2960036b10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 --2- 172.21.15.110:0/867843057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55affeb89230 0x55affec7baf0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 >> 172.21.15.110:0/867843057 conn(0x55affeb84b00 msgr2=0x55affec6f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 shutdown_connections 2024-04-27T03:18:11.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.781+0000 7f2a7d186980 1 -- 172.21.15.110:0/867843057 wait complete. 2024-04-27T03:18:11.799 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud'] 2024-04-27T03:18:11.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud'] 2024-04-27T03:18:11.799 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud 2024-04-27T03:18:11.876 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:11.877 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:11.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91d0765980 1 Processor -- start 2024-04-27T03:18:11.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91d0765980 1 -- start start 2024-04-27T03:18:11.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91d0765980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55e4b71ca850 0x55e4b7258f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91d0765980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55e4b71150d0 con 0x55e4b7259480 2024-04-27T03:18:11.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91d0765980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55e4b710a430 con 0x55e4b71ca850 2024-04-27T03:18:11.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91ce280640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55e4b71ca850 0x55e4b7258f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91ce280640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55e4b71ca850 0x55e4b7258f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51990/0 (socket says 172.21.15.110:51990) 2024-04-27T03:18:11.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91ce280640 1 -- 172.21.15.110:0/831588791 learned_addr learned my addr 172.21.15.110:0/831588791 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91ce280640 1 -- 172.21.15.110:0/831588791 >> v1:172.21.15.77:6789/0 conn(0x55e4b7259480 legacy=0x55e4b725b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.869+0000 7f91ce280640 1 -- 172.21.15.110:0/831588791 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4b7103510 con 0x55e4b71ca850 2024-04-27T03:18:11.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91ce280640 1 --2- 172.21.15.110:0/831588791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7258f40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f91c000a4e0 tx=0x7f91c001b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af308f8ffd82f084 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91cd27e640 1 -- 172.21.15.110:0/831588791 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c00aa540 con 0x55e4b71ca850 2024-04-27T03:18:11.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91cd27e640 1 -- 172.21.15.110:0/831588791 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c00aa6e0 con 0x55e4b71ca850 2024-04-27T03:18:11.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91cd27e640 1 -- 172.21.15.110:0/831588791 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c00aaa40 con 0x55e4b71ca850 2024-04-27T03:18:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91d0765980 1 -- 172.21.15.110:0/831588791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 msgr2=0x55e4b7258f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91d0765980 1 --2- 172.21.15.110:0/831588791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7258f40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f91c000a4e0 tx=0x7f91c001b090 comp rx=0 tx=0).stop 2024-04-27T03:18:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91d0765980 1 -- 172.21.15.110:0/831588791 shutdown_connections 2024-04-27T03:18:11.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91d0765980 1 --2- 172.21.15.110:0/831588791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7258f40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91d0765980 1 -- 172.21.15.110:0/831588791 >> 172.21.15.110:0/831588791 conn(0x55e4b7250a20 msgr2=0x55e4b7252e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91d0765980 1 -- 172.21.15.110:0/831588791 shutdown_connections 2024-04-27T03:18:11.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.873+0000 7f91d0765980 1 -- 172.21.15.110:0/831588791 wait complete. 2024-04-27T03:18:11.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 Processor -- start 2024-04-27T03:18:11.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- start start 2024-04-27T03:18:11.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7341450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e4b71592d0 con 0x55e4b71ca850 2024-04-27T03:18:11.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7341450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7341450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52006/0 (socket says 172.21.15.110:52006) 2024-04-27T03:18:11.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 -- 172.21.15.110:0/1671732909 learned_addr learned my addr 172.21.15.110:0/1671732909 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 -- 172.21.15.110:0/1671732909 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4b70dfae0 con 0x55e4b71ca850 2024-04-27T03:18:11.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 --2- 172.21.15.110:0/1671732909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7341450 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f91c0018cc0 tx=0x7f91c00033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0834bc6d5d6b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f90bc608640 1 -- 172.21.15.110:0/1671732909 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c00b9070 con 0x55e4b71ca850 2024-04-27T03:18:11.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f90bc608640 1 -- 172.21.15.110:0/1671732909 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c00aaa80 con 0x55e4b71ca850 2024-04-27T03:18:11.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f90bc608640 1 -- 172.21.15.110:0/1671732909 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c00aad60 con 0x55e4b71ca850 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- 172.21.15.110:0/1671732909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 msgr2=0x55e4b7341450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 --2- 172.21.15.110:0/1671732909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7341450 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f91c0018cc0 tx=0x7f91c00033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- 172.21.15.110:0/1671732909 shutdown_connections 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 --2- 172.21.15.110:0/1671732909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7341450 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- 172.21.15.110:0/1671732909 >> 172.21.15.110:0/1671732909 conn(0x55e4b7250a20 msgr2=0x55e4b733cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- 172.21.15.110:0/1671732909 shutdown_connections 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- 172.21.15.110:0/1671732909 wait complete. 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 Processor -- start 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- start start 2024-04-27T03:18:11.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7348370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91d0765980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e4b725daa0 con 0x55e4b71ca850 2024-04-27T03:18:11.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7348370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7348370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52018/0 (socket says 172.21.15.110:52018) 2024-04-27T03:18:11.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.889+0000 7f91ce280640 1 -- 172.21.15.110:0/3054443012 learned_addr learned my addr 172.21.15.110:0/3054443012 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91ce280640 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4b73488b0 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91ce280640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7348370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91c001aec0 tx=0x7f91c0017e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f90bae05640 1 -- 172.21.15.110:0/3054443012 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c00b9070 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e4b7348b40 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f90bae05640 1 -- 172.21.15.110:0/3054443012 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c00c7410 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f90bae05640 1 -- 172.21.15.110:0/3054443012 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c00c76f0 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4b734b770 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f90bae05640 1 -- 172.21.15.110:0/3054443012 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f91c00b3040 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f90bae05640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90b4034650 0x7f90b4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f90bae05640 1 -- 172.21.15.110:0/3054443012 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f91c00c4070 con 0x55e4b71ca850 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91cda7f640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90b4034650 0x7f90b4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7259480 0x55e4b734f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b734f820 con 0x55e4b7259480 2024-04-27T03:18:11.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91cea81640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7259480 0x55e4b734f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91cda7f640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90b4034650 0x7f90b4036b10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f91b8000f70 tx=0x7f91b8001420 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91cea81640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7259480 0x55e4b734f250 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91cea81640 1 -- 172.21.15.110:0/3054443012 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b7259480 2024-04-27T03:18:11.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73505f0 0x55e4b73529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7353020 con 0x55e4b73505f0 2024-04-27T03:18:11.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.893+0000 7f91ce280640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73505f0 0x55e4b73529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91ce280640 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73505f0 0x55e4b73529e0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91ce280640 1 -- 172.21.15.110:0/3054443012 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b73505f0 2024-04-27T03:18:11.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73548d0 con 0x55e4b7259480 2024-04-27T03:18:11.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91cea81640 1 -- 172.21.15.110:0/3054443012 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b7259480 2024-04-27T03:18:11.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7353b00 con 0x55e4b73505f0 2024-04-27T03:18:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91ce280640 1 -- 172.21.15.110:0/3054443012 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b73505f0 2024-04-27T03:18:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7353f20 con 0x55e4b7259480 2024-04-27T03:18:11.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91cea81640 1 -- 172.21.15.110:0/3054443012 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b7259480 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91d0765980 1 Processor -- start 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91d0765980 1 -- start start 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91d0765980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73631e0 0x55e4b7363600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91d0765980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e4b715d310 con 0x55e4b73631e0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91cea81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73631e0 0x55e4b7363600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91cea81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73631e0 0x55e4b7363600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52034/0 (socket says 172.21.15.110:52034) 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91cea81640 1 -- 172.21.15.110:0/740101641 learned_addr learned my addr 172.21.15.110:0/740101641 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.897+0000 7f91cea81640 1 -- 172.21.15.110:0/740101641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4b7363b40 con 0x55e4b73631e0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91cea81640 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73631e0 0x55e4b7363600 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f91c800aab0 tx=0x7f91c800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b3fff640 1 -- 172.21.15.110:0/740101641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c8007d30 con 0x55e4b73631e0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b3fff640 1 -- 172.21.15.110:0/740101641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c801d070 con 0x55e4b73631e0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e4b73645e0 con 0x55e4b73631e0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b3fff640 1 -- 172.21.15.110:0/740101641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c8018440 con 0x55e4b73631e0 2024-04-27T03:18:11.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4b7364ab0 con 0x55e4b73631e0 2024-04-27T03:18:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b3fff640 1 -- 172.21.15.110:0/740101641 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f91c8013070 con 0x55e4b73631e0 2024-04-27T03:18:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b3fff640 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90a8034650 0x7f90a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b3fff640 1 -- 172.21.15.110:0/740101641 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f91c804b4f0 con 0x55e4b73631e0 2024-04-27T03:18:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91ce280640 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90a8034650 0x7f90a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91d0765980 1 Processor -- start 2024-04-27T03:18:11.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91d0765980 1 -- start start 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91ce280640 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90a8034650 0x7f90a8036b10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55e4b734fdb0 tx=0x7f91c00aa670 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91d0765980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91d0765980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e4b7341ae0 con 0x55e4b73703b0 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91cea81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91cea81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52042/0 (socket says 172.21.15.110:52042) 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91cea81640 1 -- 172.21.15.110:0/756151898 learned_addr learned my addr 172.21.15.110:0/756151898 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91cea81640 1 -- 172.21.15.110:0/756151898 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4b7370da0 con 0x55e4b73703b0 2024-04-27T03:18:11.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f91cea81640 1 --2- 172.21.15.110:0/756151898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73707d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f91c8049900 tx=0x7f91c8018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9978614f769a3872 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b1ffb640 1 -- 172.21.15.110:0/756151898 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c8040070 con 0x55e4b73703b0 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b1ffb640 1 -- 172.21.15.110:0/756151898 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c8018bc0 con 0x55e4b73703b0 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.901+0000 7f90b1ffb640 1 -- 172.21.15.110:0/756151898 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c8020360 con 0x55e4b73703b0 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/756151898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 msgr2=0x55e4b73707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 --2- 172.21.15.110:0/756151898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73707d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f91c8049900 tx=0x7f91c8018950 comp rx=0 tx=0).stop 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/756151898 shutdown_connections 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 --2- 172.21.15.110:0/756151898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73707d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/756151898 >> 172.21.15.110:0/756151898 conn(0x55e4b736c6f0 msgr2=0x55e4b736a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/756151898 shutdown_connections 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/756151898 wait complete. 2024-04-27T03:18:11.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 Processor -- start 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- start start 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73777f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e4b73714c0 con 0x55e4b73703b0 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91cea81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73777f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91cea81640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73777f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52046/0 (socket says 172.21.15.110:52046) 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 learned_addr learned my addr 172.21.15.110:0/392122084 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4b7377d30 con 0x55e4b73703b0 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91cea81640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73777f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f91c8049900 tx=0x7f91c8049890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f909ffff640 1 -- 172.21.15.110:0/392122084 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c8030020 con 0x55e4b73703b0 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e4b7377fc0 con 0x55e4b73703b0 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4b737aca0 con 0x55e4b73703b0 2024-04-27T03:18:11.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f909ffff640 1 -- 172.21.15.110:0/392122084 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c80206f0 con 0x55e4b73703b0 2024-04-27T03:18:11.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f909ffff640 1 -- 172.21.15.110:0/392122084 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f91c8040030 con 0x55e4b73703b0 2024-04-27T03:18:11.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f909ffff640 1 -- 172.21.15.110:0/392122084 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f91c8033020 con 0x55e4b73703b0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f909ffff640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9098034650 0x7f9098036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f909ffff640 1 -- 172.21.15.110:0/392122084 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f91c8041070 con 0x55e4b73703b0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91ce280640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9098034650 0x7f9098036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7382eb0 0x55e4b7385370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.905+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7385ac0 con 0x55e4b7382eb0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cda7f640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7382eb0 0x55e4b7385370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91ce280640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9098034650 0x7f9098036b10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f91c00e2f80 tx=0x7f91c00e7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cda7f640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7382eb0 0x55e4b7385370 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f91b800d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 realm 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7382380 con 0x55e4b7382eb0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f91b800d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73865c0 con 0x55e4b7382eb0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f91b800d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73873a0 0x55e4b7389790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7389e60 con 0x55e4b73873a0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cea81640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73873a0 0x55e4b7389790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cea81640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73873a0 0x55e4b7389790 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738a700 con 0x55e4b7382eb0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f91b800d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 20 searching for the correct realm 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e4b738c470 0x55e4b738e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b738c470 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91ce280640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e4b738c470 0x55e4b738e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91ce280640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e4b738c470 0x55e4b738e860 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.909+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cea81640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f91c8035020 0x7f91c802c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:11.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x7f91c8035020 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f91c8035020 0x7f91c802c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f91c8035020 0x7f91c802c620 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b800d040 con 0x7f91c8035020 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b800d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b738c470 2024-04-27T03:18:11.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.913+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x7f91c8035020 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x7f91c8035020 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x7f91c8035020 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f91b802a040 con 0x7f91c8035020 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b738c470 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b738c470 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b738c470 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x7f91c8035020 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.917+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x7f91c8035020 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b7382eb0 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b802a040 con 0x55e4b7382eb0 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x7f91c8035020 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b738c470 2024-04-27T03:18:11.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x7f91c8035020 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b738c470 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738ee90 con 0x55e4b73873a0 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:11.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.921+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738b080 con 0x55e4b7382eb0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738b920 con 0x55e4b73873a0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738acc0 con 0x55e4b7382eb0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 zone default found 2024-04-27T03:18:11.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 4 Realm: () 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 zonegroup default 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b738b540 con 0x7f91c8035020 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7391220 con 0x55e4b738c470 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:11.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73915c0 con 0x55e4b738c470 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 started sync module instance, tier type = 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.925+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73906c0 con 0x55e4b73873a0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94440814873264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7390320 con 0x55e4b73873a0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73923b0 con 0x55e4b7382eb0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94440814878368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7392820 con 0x55e4b7382eb0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73932e0 con 0x55e4b73873a0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94440814873264] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94440814878368] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94440814882192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7392370 con 0x55e4b73873a0 2024-04-27T03:18:11.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.929+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7393b80 con 0x7f91c8035020 2024-04-27T03:18:11.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94440814884560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7392370 con 0x7f91c8035020 2024-04-27T03:18:11.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7394600 con 0x55e4b73873a0 2024-04-27T03:18:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94440814884560] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94440814882192] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94440814877168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7394110 con 0x55e4b73873a0 2024-04-27T03:18:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7394ed0 con 0x55e4b738c470 2024-04-27T03:18:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94440814889504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7394110 con 0x55e4b738c470 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.933+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7395c10 con 0x55e4b738c470 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94440814877168] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94440814889504] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94440814885616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7395720 con 0x55e4b738c470 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73964b0 con 0x7f91c8035020 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'117 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94440814895104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73968d0 con 0x7f91c8035020 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=0 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=1 2024-04-27T03:18:11.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=2 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=3 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=4 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=5 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94440814885616] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=6 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94440814895104] v27'118 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 20 add_watcher() i=7 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73995b0 con 0x55e4b7382eb0 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.937+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94440814891632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73968d0 con 0x55e4b7382eb0 2024-04-27T03:18:11.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.941+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94440814891632] v27'118 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.941+0000 7f9084fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:11.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.941+0000 7f91d0765980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:11.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.941+0000 7f91d0765980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:11.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.941+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7398ee0 con 0x55e4b73873a0 2024-04-27T03:18:11.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.941+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'44 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.941+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7398f00 con 0x7f91c8035020 2024-04-27T03:18:11.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.945+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'21 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:11.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.945+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a2810 con 0x55e4b738c470 2024-04-27T03:18:11.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.945+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'21 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.945+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a2d80 con 0x55e4b7382eb0 2024-04-27T03:18:11.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.949+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'119 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.949+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3230 con 0x55e4b73873a0 2024-04-27T03:18:11.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.949+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'40 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.949+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3230 con 0x55e4b738c470 2024-04-27T03:18:11.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.949+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'33 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.949+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3630 con 0x55e4b73873a0 2024-04-27T03:18:11.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.953+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'22 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.953+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3630 con 0x55e4b7382eb0 2024-04-27T03:18:11.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.953+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'28 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.953+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3ba0 con 0x55e4b73873a0 2024-04-27T03:18:11.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.957+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'14 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.957+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3ba0 con 0x55e4b73873a0 2024-04-27T03:18:11.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.957+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'20 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.957+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3ba0 con 0x55e4b73873a0 2024-04-27T03:18:11.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.961+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'41 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.961+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3ba0 con 0x55e4b738c470 2024-04-27T03:18:11.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.961+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'16 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.961+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3ba0 con 0x55e4b73873a0 2024-04-27T03:18:11.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.965+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'20 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.965+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a3ba0 con 0x55e4b7382eb0 2024-04-27T03:18:11.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.965+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'39 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.965+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b7382eb0 2024-04-27T03:18:11.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.965+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'120 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.965+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:11.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.969+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'32 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.969+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:11.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.973+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'42 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.973+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b738c470 2024-04-27T03:18:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.973+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'27 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.973+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.977+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'22 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.977+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b7382eb0 2024-04-27T03:18:11.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.977+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'121 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:11.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.977+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:11.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.981+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'33 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.981+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.981+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'30 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.981+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b738c470 2024-04-27T03:18:11.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.981+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'22 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.981+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:11.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.985+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'32 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:11.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.985+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b738c470 2024-04-27T03:18:11.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.985+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'16 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.985+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b738c470 2024-04-27T03:18:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.989+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'34 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.989+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b738c470 2024-04-27T03:18:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.989+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'22 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:11.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.989+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:11.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.993+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'26 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:12.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.993+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b7382eb0 2024-04-27T03:18:12.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.993+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'29 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:12.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.993+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b73873a0 2024-04-27T03:18:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'34 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:12.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x55e4b7382eb0 2024-04-27T03:18:12.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'40 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:12.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a4050 con 0x7f91c8035020 2024-04-27T03:18:12.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'20 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:12.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f8fff7fe640 20 reqs_thread_entry: start 2024-04-27T03:18:12.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:12.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 20 Filter name: none 2024-04-27T03:18:12.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f8ffd7fa640 20 reqs_thread_entry: start 2024-04-27T03:18:12.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73a21d0 con 0x7f91c8035020 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(94 fud [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:11.997+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 8.1 8:810902b1:users.uid::fud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73c1c20 con 0x55e4b7382eb0 2024-04-27T03:18:12.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.001+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(95 fud.buckets [call out=183b] v0'0 uv1 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:12.010 INFO:teuthology.orchestra.run.smithi110.stderr:could not remove user: unable to remove user, must specify purge data to remove user with buckets 2024-04-27T03:18:12.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.001+0000 7f9084fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.001+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94440814891632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b737d230 con 0x55e4b7382eb0 2024-04-27T03:18:12.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.001+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(96 data_loggenerations_metadata [watch unwatch cookie 94440814891632] v27'122 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:12.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.001+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94440814873264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7395270 con 0x55e4b73873a0 2024-04-27T03:18:12.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.005+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(97 notify.0 [watch unwatch cookie 94440814873264] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:12.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.005+0000 7f91d0765980 20 remove_watcher() i=0 2024-04-27T03:18:12.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.005+0000 7f91d0765980 2 removed watcher, disabling cache 2024-04-27T03:18:12.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.005+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:98 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94440814878368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7395270 con 0x55e4b7382eb0 2024-04-27T03:18:12.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.005+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(98 notify.1 [watch unwatch cookie 94440814878368] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x55e4b7382eb0 2024-04-27T03:18:12.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.005+0000 7f91d0765980 20 remove_watcher() i=1 2024-04-27T03:18:12.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.005+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94440814882192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7395270 con 0x55e4b73873a0 2024-04-27T03:18:12.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(99 notify.2 [watch unwatch cookie 94440814882192] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:12.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91d0765980 20 remove_watcher() i=2 2024-04-27T03:18:12.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:100 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94440814884560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7392d90 con 0x7f91c8035020 2024-04-27T03:18:12.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(100 notify.3 [watch unwatch cookie 94440814884560] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:12.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91d0765980 20 remove_watcher() i=3 2024-04-27T03:18:12.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94440814877168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b7395270 con 0x55e4b73873a0 2024-04-27T03:18:12.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91cea81640 1 -- 172.21.15.110:0/392122084 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(101 notify.4 [watch unwatch cookie 94440814877168] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c80083f0 con 0x55e4b73873a0 2024-04-27T03:18:12.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91d0765980 20 remove_watcher() i=4 2024-04-27T03:18:12.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94440814889504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73936d0 con 0x55e4b738c470 2024-04-27T03:18:12.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(102 notify.5 [watch unwatch cookie 94440814889504] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:12.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.009+0000 7f91d0765980 20 remove_watcher() i=5 2024-04-27T03:18:12.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:103 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94440814885616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73919f0 con 0x55e4b738c470 2024-04-27T03:18:12.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91ce280640 1 -- 172.21.15.110:0/392122084 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(103 notify.6 [watch unwatch cookie 94440814885616] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91c00cd060 con 0x55e4b738c470 2024-04-27T03:18:12.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 20 remove_watcher() i=6 2024-04-27T03:18:12.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:104 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94440814895104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e4b73919f0 con 0x7f91c8035020 2024-04-27T03:18:12.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(104 notify.7 [watch unwatch cookie 94440814895104] v27'119 uv117 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f91b803d040 con 0x7f91c8035020 2024-04-27T03:18:12.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 20 remove_watcher() i=7 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73873a0 msgr2=0x55e4b7389790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73873a0 0x55e4b7389790 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7382eb0 msgr2=0x55e4b7385370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7382eb0 0x55e4b7385370 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f91c8035020 msgr2=0x7f91c802c620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f91c8035020 0x7f91c802c620 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e4b738c470 msgr2=0x55e4b738e860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e4b738c470 0x55e4b738e860 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9098034650 msgr2=0x7f9098036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9098034650 0x7f9098036b10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f91c00e2f80 tx=0x7f91c00e7000 comp rx=0 tx=0).stop 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91cda7f640 1 -- 172.21.15.110:0/392122084 reap_dead start 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 msgr2=0x55e4b73777f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73777f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f91c8049900 tx=0x7f91c8049890 comp rx=0 tx=0).stop 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 shutdown_connections 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/392122084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73703b0 0x55e4b73777f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 >> 172.21.15.110:0/392122084 conn(0x55e4b736c6f0 msgr2=0x55e4b7371be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 shutdown_connections 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/392122084 wait complete. 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90a8034650 msgr2=0x7f90a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90a8034650 0x7f90a8036b10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x55e4b734fdb0 tx=0x7f91c00aa670 comp rx=0 tx=0).stop 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73631e0 msgr2=0x55e4b7363600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73631e0 0x55e4b7363600 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f91c800aab0 tx=0x7f91c800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 shutdown_connections 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90a8034650 0x7f90a8036b10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/740101641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b73631e0 0x55e4b7363600 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 >> 172.21.15.110:0/740101641 conn(0x55e4b7357ff0 msgr2=0x55e4b7358b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 shutdown_connections 2024-04-27T03:18:12.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/740101641 wait complete. 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73505f0 msgr2=0x55e4b73529e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73505f0 0x55e4b73529e0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7259480 msgr2=0x55e4b734f250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7259480 0x55e4b734f250 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90b4034650 msgr2=0x7f90b4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90b4034650 0x7f90b4036b10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f91b8000f70 tx=0x7f91b8001420 comp rx=0 tx=0).stop 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 msgr2=0x55e4b7348370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7348370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91c001aec0 tx=0x7f91c0017e30 comp rx=0 tx=0).stop 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 shutdown_connections 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e4b73505f0 0x55e4b73529e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e4b7259480 0x55e4b734f250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f90b4034650 0x7f90b4036b10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 --2- 172.21.15.110:0/3054443012 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e4b71ca850 0x55e4b7348370 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 >> 172.21.15.110:0/3054443012 conn(0x55e4b7250a20 msgr2=0x55e4b733baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 shutdown_connections 2024-04-27T03:18:12.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.013+0000 7f91d0765980 1 -- 172.21.15.110:0/3054443012 wait complete. 2024-04-27T03:18:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 17 2024-04-27T03:18:12.032 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'unlink', '--uid', 'fud', '--bucket', 'myfoo'] 2024-04-27T03:18:12.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'unlink', '--uid', 'fud', '--bucket', 'myfoo'] 2024-04-27T03:18:12.033 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket unlink --uid fud --bucket myfoo 2024-04-27T03:18:12.114 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:12.114 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:12.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.105+0000 7f13dab3e980 1 Processor -- start 2024-04-27T03:18:12.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.105+0000 7f13dab3e980 1 -- start start 2024-04-27T03:18:12.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.105+0000 7f13dab3e980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5579debf14c0 0x5579debf18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.105+0000 7f13dab3e980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5579deaae0d0 con 0x5579debf1de0 2024-04-27T03:18:12.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.105+0000 7f13dab3e980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5579deaa3430 con 0x5579debf14c0 2024-04-27T03:18:12.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d8659640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5579debf14c0 0x5579debf18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d8659640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5579debf14c0 0x5579debf18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52050/0 (socket says 172.21.15.110:52050) 2024-04-27T03:18:12.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d8659640 1 -- 172.21.15.110:0/2131711971 learned_addr learned my addr 172.21.15.110:0/2131711971 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d7e58640 1 --1- 172.21.15.110:0/2131711971 >> v1:172.21.15.77:6789/0 conn(0x5579debf1de0 0x5579debf27a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34874/0 (socket says 172.21.15.110:34874) 2024-04-27T03:18:12.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d8659640 1 -- 172.21.15.110:0/2131711971 >> v1:172.21.15.77:6789/0 conn(0x5579debf1de0 legacy=0x5579debf27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d8659640 1 -- 172.21.15.110:0/2131711971 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579dea9c510 con 0x5579debf14c0 2024-04-27T03:18:12.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d8659640 1 --2- 172.21.15.110:0/2131711971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579debf18a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f13c800a5c0 tx=0x7f13c801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b629cbf20b7491ff server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d7657640 1 -- 172.21.15.110:0/2131711971 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c80a6070 con 0x5579debf14c0 2024-04-27T03:18:12.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d7657640 1 -- 172.21.15.110:0/2131711971 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c80af070 con 0x5579debf14c0 2024-04-27T03:18:12.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.109+0000 7f13d7657640 1 -- 172.21.15.110:0/2131711971 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c80aa5c0 con 0x5579debf14c0 2024-04-27T03:18:12.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.113+0000 7f13dab3e980 1 -- 172.21.15.110:0/2131711971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 msgr2=0x5579debf18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.113+0000 7f13dab3e980 1 --2- 172.21.15.110:0/2131711971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579debf18a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f13c800a5c0 tx=0x7f13c801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:12.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.113+0000 7f13dab3e980 1 -- 172.21.15.110:0/2131711971 shutdown_connections 2024-04-27T03:18:12.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.113+0000 7f13dab3e980 1 --2- 172.21.15.110:0/2131711971 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579debf18a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.113+0000 7f13dab3e980 1 -- 172.21.15.110:0/2131711971 >> 172.21.15.110:0/2131711971 conn(0x5579deb64ba0 msgr2=0x5579deb64fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.113+0000 7f13dab3e980 1 -- 172.21.15.110:0/2131711971 shutdown_connections 2024-04-27T03:18:12.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.113+0000 7f13dab3e980 1 -- 172.21.15.110:0/2131711971 wait complete. 2024-04-27T03:18:12.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 Processor -- start 2024-04-27T03:18:12.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 -- start start 2024-04-27T03:18:12.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579decdcd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5579deaf22d0 con 0x5579debf14c0 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13d8659640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579decdcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13d8659640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579decdcd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52058/0 (socket says 172.21.15.110:52058) 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13d8659640 1 -- 172.21.15.110:0/4120718346 learned_addr learned my addr 172.21.15.110:0/4120718346 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13d8659640 1 -- 172.21.15.110:0/4120718346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579dea78ae0 con 0x5579debf14c0 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13d8659640 1 --2- 172.21.15.110:0/4120718346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579decdcd20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f13c80199f0 tx=0x7f13c8003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bffdba19d595eba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f12c660c640 1 -- 172.21.15.110:0/4120718346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c80aaa00 con 0x5579debf14c0 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f12c660c640 1 -- 172.21.15.110:0/4120718346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c80aaba0 con 0x5579debf14c0 2024-04-27T03:18:12.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f12c660c640 1 -- 172.21.15.110:0/4120718346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c80bd3f0 con 0x5579debf14c0 2024-04-27T03:18:12.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 -- 172.21.15.110:0/4120718346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 msgr2=0x5579decdcd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 --2- 172.21.15.110:0/4120718346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579decdcd20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f13c80199f0 tx=0x7f13c8003030 comp rx=0 tx=0).stop 2024-04-27T03:18:12.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 -- 172.21.15.110:0/4120718346 shutdown_connections 2024-04-27T03:18:12.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 --2- 172.21.15.110:0/4120718346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579decdcd20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.125+0000 7f13dab3e980 1 -- 172.21.15.110:0/4120718346 >> 172.21.15.110:0/4120718346 conn(0x5579deb64ba0 msgr2=0x5579decd88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 -- 172.21.15.110:0/4120718346 shutdown_connections 2024-04-27T03:18:12.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 -- 172.21.15.110:0/4120718346 wait complete. 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 Processor -- start 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 -- start start 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579dece3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5579debfb510 con 0x5579debf14c0 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13d8659640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579dece3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13d8659640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579dece3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52068/0 (socket says 172.21.15.110:52068) 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13d8659640 1 -- 172.21.15.110:0/938558904 learned_addr learned my addr 172.21.15.110:0/938558904 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13d8659640 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579dece41e0 con 0x5579debf14c0 2024-04-27T03:18:12.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13d8659640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579dece3ca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f13c801b170 tx=0x7f13c80180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f12c4e09640 1 -- 172.21.15.110:0/938558904 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c80bd7c0 con 0x5579debf14c0 2024-04-27T03:18:12.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5579dece4470 con 0x5579debf14c0 2024-04-27T03:18:12.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f12c4e09640 1 -- 172.21.15.110:0/938558904 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c80bd960 con 0x5579debf14c0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f12c4e09640 1 -- 172.21.15.110:0/938558904 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13c80bdc40 con 0x5579debf14c0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579dece7050 con 0x5579debf14c0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f12c4e09640 1 -- 172.21.15.110:0/938558904 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f13c80ce030 con 0x5579debf14c0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f12c4e09640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12c0034650 0x7f12c0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f12c4e09640 1 -- 172.21.15.110:0/938558904 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f13c80cc020 con 0x5579debf14c0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13d7e58640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12c0034650 0x7f12c0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579debf1de0 0x5579deceab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579deceb150 con 0x5579debf1de0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.129+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579debf1de0 0x5579deceab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d7e58640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12c0034650 0x7f12c0036b10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000f70 tx=0x7f13cc001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579debf1de0 0x5579deceab30 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8e5a640 1 -- 172.21.15.110:0/938558904 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f13d000a4d0 con 0x5579debf1de0 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579decebf40 0x5579decee330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579decee970 con 0x5579decebf40 2024-04-27T03:18:12.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8659640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579decebf40 0x5579decee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8659640 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579decebf40 0x5579decee330 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8659640 1 -- 172.21.15.110:0/938558904 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f13c80c6d20 con 0x5579decebf40 2024-04-27T03:18:12.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579decf0220 con 0x5579debf1de0 2024-04-27T03:18:12.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8e5a640 1 -- 172.21.15.110:0/938558904 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f13d000a4d0 con 0x5579debf1de0 2024-04-27T03:18:12.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579decef450 con 0x5579decebf40 2024-04-27T03:18:12.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8659640 1 -- 172.21.15.110:0/938558904 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f13c80c6d20 con 0x5579decebf40 2024-04-27T03:18:12.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579decef8a0 con 0x5579debf1de0 2024-04-27T03:18:12.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13d8e5a640 1 -- 172.21.15.110:0/938558904 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f13d000a4d0 con 0x5579debf1de0 2024-04-27T03:18:12.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 Processor -- start 2024-04-27T03:18:12.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 -- start start 2024-04-27T03:18:12.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579decfeb60 0x5579decfef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.133+0000 7f13dab3e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5579deaf6310 con 0x5579decfeb60 2024-04-27T03:18:12.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579decfeb60 0x5579decfef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579decfeb60 0x5579decfef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52072/0 (socket says 172.21.15.110:52072) 2024-04-27T03:18:12.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 -- 172.21.15.110:0/3220976454 learned_addr learned my addr 172.21.15.110:0/3220976454 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 -- 172.21.15.110:0/3220976454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579decff4c0 con 0x5579decfeb60 2024-04-27T03:18:12.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579decfeb60 0x5579decfef80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f13d000dfc0 tx=0x7f13d00178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f12bdffb640 1 -- 172.21.15.110:0/3220976454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d00157d0 con 0x5579decfeb60 2024-04-27T03:18:12.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5579decfff60 con 0x5579decfeb60 2024-04-27T03:18:12.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f12bdffb640 1 -- 172.21.15.110:0/3220976454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d0015970 con 0x5579decfeb60 2024-04-27T03:18:12.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f12bdffb640 1 -- 172.21.15.110:0/3220976454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d0015c50 con 0x5579decfeb60 2024-04-27T03:18:12.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579ded00430 con 0x5579decfeb60 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f12bdffb640 1 -- 172.21.15.110:0/3220976454 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f13d0020440 con 0x5579decfeb60 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f12bdffb640 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12b0034650 0x7f12b0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f12bdffb640 1 -- 172.21.15.110:0/3220976454 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f13d000e040 con 0x5579decfeb60 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8659640 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12b0034650 0x7f12b0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13dab3e980 1 Processor -- start 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13dab3e980 1 -- start start 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8659640 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12b0034650 0x7f12b0036b10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5579deceef70 tx=0x7f13c80e36c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13dab3e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded0c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13dab3e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5579decdd3b0 con 0x5579ded0bd60 2024-04-27T03:18:12.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded0c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded0c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52082/0 (socket says 172.21.15.110:52082) 2024-04-27T03:18:12.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.137+0000 7f13d8e5a640 1 -- 172.21.15.110:0/1802925447 learned_addr learned my addr 172.21.15.110:0/1802925447 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13d8e5a640 1 -- 172.21.15.110:0/1802925447 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579ded0c750 con 0x5579ded0bd60 2024-04-27T03:18:12.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/1802925447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded0c180 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f13d004aaa0 tx=0x7f13d004aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbe514a689cfd25 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f12b77fe640 1 -- 172.21.15.110:0/1802925447 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d0041070 con 0x5579ded0bd60 2024-04-27T03:18:12.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f12b77fe640 1 -- 172.21.15.110:0/1802925447 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d0012570 con 0x5579ded0bd60 2024-04-27T03:18:12.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f12b77fe640 1 -- 172.21.15.110:0/1802925447 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d0020ca0 con 0x5579ded0bd60 2024-04-27T03:18:12.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- 172.21.15.110:0/1802925447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 msgr2=0x5579ded0c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 --2- 172.21.15.110:0/1802925447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded0c180 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f13d004aaa0 tx=0x7f13d004aca0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- 172.21.15.110:0/1802925447 shutdown_connections 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 --2- 172.21.15.110:0/1802925447 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded0c180 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- 172.21.15.110:0/1802925447 >> 172.21.15.110:0/1802925447 conn(0x5579ded080a0 msgr2=0x5579ded06100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- 172.21.15.110:0/1802925447 shutdown_connections 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- 172.21.15.110:0/1802925447 wait complete. 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 Processor -- start 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- start start 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded13160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5579ded0ce00 con 0x5579ded0bd60 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13d8e5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded13160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13d8e5a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded13160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52092/0 (socket says 172.21.15.110:52092) 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 learned_addr learned my addr 172.21.15.110:0/428528202 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579ded136a0 con 0x5579ded0bd60 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded13160 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f13d0046040 tx=0x7f13d004aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f12b5ffb640 1 -- 172.21.15.110:0/428528202 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d0038070 con 0x5579ded0bd60 2024-04-27T03:18:12.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5579ded13930 con 0x5579ded0bd60 2024-04-27T03:18:12.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f12b5ffb640 1 -- 172.21.15.110:0/428528202 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d00113a0 con 0x5579ded0bd60 2024-04-27T03:18:12.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f12b5ffb640 1 -- 172.21.15.110:0/428528202 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13d0041040 con 0x5579ded0bd60 2024-04-27T03:18:12.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.141+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579ded16610 con 0x5579ded0bd60 2024-04-27T03:18:12.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f12b5ffb640 1 -- 172.21.15.110:0/428528202 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f13d003b070 con 0x5579ded0bd60 2024-04-27T03:18:12.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f12b5ffb640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12a8034600 0x7f12a8036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f12b5ffb640 1 -- 172.21.15.110:0/428528202 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f13d0034c80 con 0x5579ded0bd60 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d8659640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12a8034600 0x7f12a8036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579ded1e860 0x5579ded20d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded21450 con 0x5579ded1e860 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d7e58640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579ded1e860 0x5579ded20d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d8659640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12a8034600 0x7f12a8036ac0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13c80c6fd0 tx=0x7f13c80e9000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d7e58640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579ded1e860 0x5579ded20d00 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 realm 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded1dd70 con 0x5579ded1e860 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded21f20 con 0x5579ded1e860 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579ded22d00 0x5579ded250f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded25780 con 0x5579ded22d00 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579ded22d00 0x5579ded250f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579ded22d00 0x5579ded250f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded25fd0 con 0x5579ded1e860 2024-04-27T03:18:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.145+0000 7f13dab3e980 20 searching for the correct realm 2024-04-27T03:18:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5579ded27d40 0x5579ded2a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded27d40 2024-04-27T03:18:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8659640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5579ded27d40 0x5579ded2a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8659640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5579ded27d40 0x5579ded2a130 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded22d00 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8e5a640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f13d00133b0 0x7f13d003c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x7f13d00133b0 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f13d00133b0 0x7f13d003c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f13d00133b0 0x7f13d003c790 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded27d40 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.149+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded22d00 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded22d00 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x7f13d00133b0 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x7f13d00133b0 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded22d00 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded27d40 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded22d00 2024-04-27T03:18:12.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded27d40 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.153+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded27d40 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x7f13d00133b0 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x5579ded1e860 2024-04-27T03:18:12.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2a740 con 0x7f13d00133b0 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f13cc03e040 con 0x5579ded27d40 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f13cc03e040 con 0x7f13d00133b0 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f13cc03e040 con 0x5579ded22d00 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f13cc03e040 con 0x5579ded22d00 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f13cc03e040 con 0x5579ded27d40 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.157+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f13cc03e040 con 0x5579ded22d00 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f13cc03e040 con 0x5579ded22d00 2024-04-27T03:18:12.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:12.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:12.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:12.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded26950 con 0x5579ded1e860 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded26590 con 0x5579ded22d00 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded272b0 con 0x5579ded1e860 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 zone default found 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 4 Realm: () 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 zonegroup default 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2b070 con 0x7f13d00133b0 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2ce70 con 0x5579ded27d40 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2d210 con 0x5579ded27d40 2024-04-27T03:18:12.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 started sync module instance, tier type = 2024-04-27T03:18:12.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:12.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.161+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2c310 con 0x5579ded22d00 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93981917759280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2bf70 con 0x5579ded22d00 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2e0d0 con 0x5579ded1e860 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93981917764640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2e040 con 0x5579ded1e860 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2efd0 con 0x5579ded22d00 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93981917759280] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.165+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93981917764640] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93981917766848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2eae0 con 0x5579ded22d00 2024-04-27T03:18:12.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2f870 con 0x7f13d00133b0 2024-04-27T03:18:12.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93981917770688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2eae0 con 0x7f13d00133b0 2024-04-27T03:18:12.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded30310 con 0x5579ded22d00 2024-04-27T03:18:12.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93981917770688] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93981917766848] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93981917763072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded302d0 con 0x5579ded22d00 2024-04-27T03:18:12.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.169+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded30700 con 0x5579ded27d40 2024-04-27T03:18:12.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93981917771744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded30f50 con 0x5579ded27d40 2024-04-27T03:18:12.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded31910 con 0x5579ded27d40 2024-04-27T03:18:12.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93981917763072] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93981917771744] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93981917779040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded31d00 con 0x5579ded27d40 2024-04-27T03:18:12.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded32590 con 0x7f13d00133b0 2024-04-27T03:18:12.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'120 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93981917781248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded32160 con 0x7f13d00133b0 2024-04-27T03:18:12.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 20 add_watcher() i=0 2024-04-27T03:18:12.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 20 add_watcher() i=1 2024-04-27T03:18:12.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 20 add_watcher() i=2 2024-04-27T03:18:12.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 20 add_watcher() i=3 2024-04-27T03:18:12.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 20 add_watcher() i=4 2024-04-27T03:18:12.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 20 add_watcher() i=5 2024-04-27T03:18:12.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93981917779040] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.173+0000 7f13dab3e980 20 add_watcher() i=6 2024-04-27T03:18:12.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93981917781248] v27'121 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 20 add_watcher() i=7 2024-04-27T03:18:12.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:12.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded2fe00 con 0x5579ded1e860 2024-04-27T03:18:12.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93981917773312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded32fe0 con 0x5579ded1e860 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93981917773312] v27'123 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f128efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 5 note: GC not initialized 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f128dfe3640 20 reqs_thread_entry: start 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 20 Filter name: none 2024-04-27T03:18:12.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f1286ffd640 20 reqs_thread_entry: start 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded592a0 con 0x7f13d00133b0 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.177+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 fud [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded5b380 con 0x7f13d00133b0 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv20 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded5cd40 con 0x7f13d00133b0 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(64 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv4 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache get: name=default.rgw.meta+users.uid+fud : hit (requested=0x13, cached=0x13) 2024-04-27T03:18:12.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 8.1 8:810902b1:users.uid::fud.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded5e2b0 con 0x5579ded1e860 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 fud.buckets [call] v27'2 uv2 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache get: name=default.rgw.meta+root+myfoo : type miss (requested=0x13, cached=0x11) 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded5e6b0 con 0x7f13d00133b0 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(66 myfoo [call out=48b,read 0~165 out=165b,getxattrs out=75b] v0'0 uv20 ondisk = 0) ==== 233+0+288 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x13 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 8.9 8:91b71841:root::myfoo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded642a0 con 0x7f13d00133b0 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(67 myfoo [delete,create,call,call,writefull 0~165] v27'21 uv21 ondisk = 0) ==== 317+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2024-04-27T03:18:12.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 7.14 7:2b04a3e9:::notify.6:head [notify cookie 93981917969840 in=350b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5579ded61770 con 0x5579ded27d40 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== watch-notify(notify (1) cookie 93981917779040 notify 115964117002 ret 0) ==== 380+0+0 (crc 0 0 0) 0x5579dece7050 con 0x5579ded27d40 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f12b77fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117002 cookie 93981917779040 notifier 4506 bl.length()=338 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f12b77fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x17 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f12b77fe640 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f12b77fe640 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f129c003eb0 con 0x5579ded27d40 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(68 notify.6 [notify cookie 93981917969840 out=8b] v0'0 uv120 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.181+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(69 notify.6 [notify-ack] v0'0 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.185+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== watch-notify(notify_complete (2) cookie 93981917969840 notify 115964117002 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5579dece4470 con 0x5579ded27d40 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.185+0000 7f128efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.185+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93981917773312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x5579ded1e860 2024-04-27T03:18:12.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.185+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 93981917773312] v27'124 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.185+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93981917759280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x5579ded22d00 2024-04-27T03:18:12.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.189+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 93981917759280] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.189+0000 7f13dab3e980 20 remove_watcher() i=0 2024-04-27T03:18:12.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.189+0000 7f13dab3e980 2 removed watcher, disabling cache 2024-04-27T03:18:12.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.189+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93981917764640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x5579ded1e860 2024-04-27T03:18:12.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.189+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 93981917764640] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x5579ded1e860 2024-04-27T03:18:12.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.189+0000 7f13dab3e980 20 remove_watcher() i=1 2024-04-27T03:18:12.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.189+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93981917766848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x5579ded22d00 2024-04-27T03:18:12.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 93981917766848] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 20 remove_watcher() i=2 2024-04-27T03:18:12.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93981917770688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x7f13d00133b0 2024-04-27T03:18:12.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 93981917770688] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 20 remove_watcher() i=3 2024-04-27T03:18:12.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93981917763072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x5579ded22d00 2024-04-27T03:18:12.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13d8e5a640 1 -- 172.21.15.110:0/428528202 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 93981917763072] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13d003a030 con 0x5579ded22d00 2024-04-27T03:18:12.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 20 remove_watcher() i=4 2024-04-27T03:18:12.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93981917771744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x5579ded27d40 2024-04-27T03:18:12.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 93981917771744] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 20 remove_watcher() i=5 2024-04-27T03:18:12.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.193+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93981917779040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x5579ded27d40 2024-04-27T03:18:12.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13d8659640 1 -- 172.21.15.110:0/428528202 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 93981917779040] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13c80fb040 con 0x5579ded27d40 2024-04-27T03:18:12.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 20 remove_watcher() i=6 2024-04-27T03:18:12.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93981917781248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5579ded56fd0 con 0x7f13d00133b0 2024-04-27T03:18:12.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 93981917781248] v27'122 uv120 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f13cc00f620 con 0x7f13d00133b0 2024-04-27T03:18:12.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 20 remove_watcher() i=7 2024-04-27T03:18:12.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579ded22d00 msgr2=0x5579ded250f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579ded22d00 0x5579ded250f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579ded1e860 msgr2=0x5579ded20d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579ded1e860 0x5579ded20d00 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f13d00133b0 msgr2=0x7f13d003c790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f13d00133b0 0x7f13d003c790 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5579ded27d40 msgr2=0x5579ded2a130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5579ded27d40 0x5579ded2a130 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12a8034600 msgr2=0x7f12a8036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12a8034600 0x7f12a8036ac0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f13c80c6fd0 tx=0x7f13c80e9000 comp rx=0 tx=0).stop 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 msgr2=0x5579ded13160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13d7e58640 1 -- 172.21.15.110:0/428528202 reap_dead start 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded13160 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f13d0046040 tx=0x7f13d004aa30 comp rx=0 tx=0).stop 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 shutdown_connections 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/428528202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579ded0bd60 0x5579ded13160 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 >> 172.21.15.110:0/428528202 conn(0x5579ded080a0 msgr2=0x5579ded0d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 shutdown_connections 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/428528202 wait complete. 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12b0034650 msgr2=0x7f12b0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12b0034650 0x7f12b0036b10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x5579deceef70 tx=0x7f13c80e36c0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579decfeb60 msgr2=0x5579decfef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579decfeb60 0x5579decfef80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f13d000dfc0 tx=0x7f13d00178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 shutdown_connections 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12b0034650 0x7f12b0036b10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/3220976454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579decfeb60 0x5579decfef80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 >> 172.21.15.110:0/3220976454 conn(0x5579decf3970 msgr2=0x5579decf4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 shutdown_connections 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/3220976454 wait complete. 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579decebf40 msgr2=0x5579decee330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579decebf40 0x5579decee330 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579debf1de0 msgr2=0x5579deceab30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579debf1de0 0x5579deceab30 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12c0034650 msgr2=0x7f12c0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12c0034650 0x7f12c0036b10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f13cc000f70 tx=0x7f13cc001380 comp rx=0 tx=0).stop 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 msgr2=0x5579dece3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579dece3ca0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f13c801b170 tx=0x7f13c80180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 shutdown_connections 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5579decebf40 0x5579decee330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5579debf1de0 0x5579deceab30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f12c0034650 0x7f12c0036b10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 --2- 172.21.15.110:0/938558904 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5579debf14c0 0x5579dece3ca0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 >> 172.21.15.110:0/938558904 conn(0x5579deb64ba0 msgr2=0x5579decd73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 shutdown_connections 2024-04-27T03:18:12.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.197+0000 7f13dab3e980 1 -- 172.21.15.110:0/938558904 wait complete. 2024-04-27T03:18:12.215 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--uid', 'foo', '--bucket', 'myfoo', '--bucket-id', '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1'] 2024-04-27T03:18:12.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--uid', 'foo', '--bucket', 'myfoo', '--bucket-id', '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1'] 2024-04-27T03:18:12.216 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --uid foo --bucket myfoo --bucket-id 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:12.297 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:12.298 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1aa70980 1 Processor -- start 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1aa70980 1 -- start start 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1aa70980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x562419c0a830 0x562419c98ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1aa70980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x562419b550d0 con 0x562419c993e0 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1aa70980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x562419b4a430 con 0x562419c0a830 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1858b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x562419c0a830 0x562419c98ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1858b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x562419c0a830 0x562419c98ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52106/0 (socket says 172.21.15.110:52106) 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff1858b640 1 -- 172.21.15.110:0/1042957105 learned_addr learned my addr 172.21.15.110:0/1042957105 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.289+0000 7eff17d8a640 1 --1- 172.21.15.110:0/1042957105 >> v1:172.21.15.77:6789/0 conn(0x562419c993e0 0x562419c9b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34886/0 (socket says 172.21.15.110:34886) 2024-04-27T03:18:12.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1858b640 1 -- 172.21.15.110:0/1042957105 >> v1:172.21.15.77:6789/0 conn(0x562419c993e0 legacy=0x562419c9b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1858b640 1 -- 172.21.15.110:0/1042957105 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x562419b43510 con 0x562419c0a830 2024-04-27T03:18:12.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1858b640 1 --2- 172.21.15.110:0/1042957105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419c98ea0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7eff0800a5c0 tx=0x7eff0801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149deabac4c3e1db server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff17589640 1 -- 172.21.15.110:0/1042957105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff080a6070 con 0x562419c0a830 2024-04-27T03:18:12.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff17589640 1 -- 172.21.15.110:0/1042957105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff080af070 con 0x562419c0a830 2024-04-27T03:18:12.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff17589640 1 -- 172.21.15.110:0/1042957105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff080aa5c0 con 0x562419c0a830 2024-04-27T03:18:12.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1aa70980 1 -- 172.21.15.110:0/1042957105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 msgr2=0x562419c98ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1042957105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419c98ea0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7eff0800a5c0 tx=0x7eff0801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:12.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1aa70980 1 -- 172.21.15.110:0/1042957105 shutdown_connections 2024-04-27T03:18:12.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1042957105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419c98ea0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1aa70980 1 -- 172.21.15.110:0/1042957105 >> 172.21.15.110:0/1042957105 conn(0x562419c909c0 msgr2=0x562419c92de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1aa70980 1 -- 172.21.15.110:0/1042957105 shutdown_connections 2024-04-27T03:18:12.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.293+0000 7eff1aa70980 1 -- 172.21.15.110:0/1042957105 wait complete. 2024-04-27T03:18:12.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 Processor -- start 2024-04-27T03:18:12.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- start start 2024-04-27T03:18:12.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d81470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562419b992d0 con 0x562419c0a830 2024-04-27T03:18:12.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d81470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d81470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52116/0 (socket says 172.21.15.110:52116) 2024-04-27T03:18:12.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 -- 172.21.15.110:0/520917967 learned_addr learned my addr 172.21.15.110:0/520917967 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 -- 172.21.15.110:0/520917967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562419b1fae0 con 0x562419c0a830 2024-04-27T03:18:12.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 --2- 172.21.15.110:0/520917967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d81470 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7eff080199f0 tx=0x7eff08003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2bb3748178e8e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7efe0660c640 1 -- 172.21.15.110:0/520917967 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff080aaa00 con 0x562419c0a830 2024-04-27T03:18:12.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7efe0660c640 1 -- 172.21.15.110:0/520917967 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff080aaba0 con 0x562419c0a830 2024-04-27T03:18:12.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7efe0660c640 1 -- 172.21.15.110:0/520917967 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff080bd3f0 con 0x562419c0a830 2024-04-27T03:18:12.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- 172.21.15.110:0/520917967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 msgr2=0x562419d81470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 --2- 172.21.15.110:0/520917967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d81470 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7eff080199f0 tx=0x7eff08003030 comp rx=0 tx=0).stop 2024-04-27T03:18:12.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- 172.21.15.110:0/520917967 shutdown_connections 2024-04-27T03:18:12.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 --2- 172.21.15.110:0/520917967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d81470 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- 172.21.15.110:0/520917967 >> 172.21.15.110:0/520917967 conn(0x562419c909c0 msgr2=0x562419d7d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- 172.21.15.110:0/520917967 shutdown_connections 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- 172.21.15.110:0/520917967 wait complete. 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 Processor -- start 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- start start 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d88480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1aa70980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562419c9da00 con 0x562419c0a830 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d88480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d88480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52130/0 (socket says 172.21.15.110:52130) 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.309+0000 7eff1858b640 1 -- 172.21.15.110:0/1004126573 learned_addr learned my addr 172.21.15.110:0/1004126573 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1858b640 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562419d889c0 con 0x562419c0a830 2024-04-27T03:18:12.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1858b640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d88480 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7eff0801b170 tx=0x7eff080180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7efe04e09640 1 -- 172.21.15.110:0/1004126573 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff080bd7c0 con 0x562419c0a830 2024-04-27T03:18:12.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7efe04e09640 1 -- 172.21.15.110:0/1004126573 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff080bd960 con 0x562419c0a830 2024-04-27T03:18:12.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7efe04e09640 1 -- 172.21.15.110:0/1004126573 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff080bdc40 con 0x562419c0a830 2024-04-27T03:18:12.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562419d88c50 con 0x562419c0a830 2024-04-27T03:18:12.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562419d8b880 con 0x562419c0a830 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7efe04e09640 1 -- 172.21.15.110:0/1004126573 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7eff080ce020 con 0x562419c0a830 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7efe04e09640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efe000346a0 0x7efe00036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7efe04e09640 1 -- 172.21.15.110:0/1004126573 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7eff080cc020 con 0x562419c0a830 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff17d8a640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efe000346a0 0x7efe00036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419c993e0 0x562419d8f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419d8f8a0 con 0x562419c993e0 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff18d8c640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419c993e0 0x562419d8f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff17d8a640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efe000346a0 0x7efe00036b60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7eff0c000f70 tx=0x7eff0c001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff18d8c640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419c993e0 0x562419d8f360 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff18d8c640 1 -- 172.21.15.110:0/1004126573 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7eff1000a4d0 con 0x562419c993e0 2024-04-27T03:18:12.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419d90670 0x562419d92a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.313+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419d930a0 con 0x562419d90670 2024-04-27T03:18:12.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1858b640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419d90670 0x562419d92a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1858b640 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419d90670 0x562419d92a60 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1858b640 1 -- 172.21.15.110:0/1004126573 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7eff080e5050 con 0x562419d90670 2024-04-27T03:18:12.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419d94950 con 0x562419c993e0 2024-04-27T03:18:12.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff18d8c640 1 -- 172.21.15.110:0/1004126573 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7eff1000a4d0 con 0x562419c993e0 2024-04-27T03:18:12.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419d93b80 con 0x562419d90670 2024-04-27T03:18:12.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1858b640 1 -- 172.21.15.110:0/1004126573 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7eff080e5050 con 0x562419d90670 2024-04-27T03:18:12.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419d93f70 con 0x562419c993e0 2024-04-27T03:18:12.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff18d8c640 1 -- 172.21.15.110:0/1004126573 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7eff1000a4d0 con 0x562419c993e0 2024-04-27T03:18:12.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1aa70980 1 Processor -- start 2024-04-27T03:18:12.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1aa70980 1 -- start start 2024-04-27T03:18:12.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1aa70980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419da3290 0x562419da36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff1aa70980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562419b9d310 con 0x562419da3290 2024-04-27T03:18:12.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff18d8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419da3290 0x562419da36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff18d8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419da3290 0x562419da36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52136/0 (socket says 172.21.15.110:52136) 2024-04-27T03:18:12.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.317+0000 7eff18d8c640 1 -- 172.21.15.110:0/3064507062 learned_addr learned my addr 172.21.15.110:0/3064507062 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff18d8c640 1 -- 172.21.15.110:0/3064507062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562419da3bf0 con 0x562419da3290 2024-04-27T03:18:12.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff18d8c640 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419da3290 0x562419da36b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7eff1000dfc0 tx=0x7eff100178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7efdfdffb640 1 -- 172.21.15.110:0/3064507062 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff100157d0 con 0x562419da3290 2024-04-27T03:18:12.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562419da4690 con 0x562419da3290 2024-04-27T03:18:12.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7efdfdffb640 1 -- 172.21.15.110:0/3064507062 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff10015970 con 0x562419da3290 2024-04-27T03:18:12.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562419da4b60 con 0x562419da3290 2024-04-27T03:18:12.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7efdfdffb640 1 -- 172.21.15.110:0/3064507062 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff10015c50 con 0x562419da3290 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7efdfdffb640 1 -- 172.21.15.110:0/3064507062 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7eff10020440 con 0x562419da3290 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7efdfdffb640 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efdf0034650 0x7efdf0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7efdfdffb640 1 -- 172.21.15.110:0/3064507062 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7eff1000e040 con 0x562419da3290 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1858b640 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efdf0034650 0x7efdf0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1aa70980 1 Processor -- start 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1aa70980 1 -- start start 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1858b640 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efdf0034650 0x7efdf0036b10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x562419d936a0 tx=0x7eff080e3780 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1aa70980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419c95360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff1aa70980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562419d81b00 con 0x562419c94f40 2024-04-27T03:18:12.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff18d8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419c95360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff18d8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419c95360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52142/0 (socket says 172.21.15.110:52142) 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff18d8c640 1 -- 172.21.15.110:0/3998141065 learned_addr learned my addr 172.21.15.110:0/3998141065 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff18d8c640 1 -- 172.21.15.110:0/3998141065 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562419c95930 con 0x562419c94f40 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.321+0000 7eff18d8c640 1 --2- 172.21.15.110:0/3998141065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419c95360 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7eff1004aaa0 tx=0x7eff10013870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7904658c3b75ebe server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7efdf77fe640 1 -- 172.21.15.110:0/3998141065 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff10041070 con 0x562419c94f40 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7efdf77fe640 1 -- 172.21.15.110:0/3998141065 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff10020b50 con 0x562419c94f40 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7efdf77fe640 1 -- 172.21.15.110:0/3998141065 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff10011cc0 con 0x562419c94f40 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- 172.21.15.110:0/3998141065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 msgr2=0x562419c95360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 --2- 172.21.15.110:0/3998141065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419c95360 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7eff1004aaa0 tx=0x7eff10013870 comp rx=0 tx=0).stop 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- 172.21.15.110:0/3998141065 shutdown_connections 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 --2- 172.21.15.110:0/3998141065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419c95360 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- 172.21.15.110:0/3998141065 >> 172.21.15.110:0/3998141065 conn(0x562419dac7a0 msgr2=0x562419daa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- 172.21.15.110:0/3998141065 shutdown_connections 2024-04-27T03:18:12.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- 172.21.15.110:0/3998141065 wait complete. 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 Processor -- start 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- start start 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419db9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562419c95fc0 con 0x562419c94f40 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff18d8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419db9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff18d8c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419db9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52152/0 (socket says 172.21.15.110:52152) 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 learned_addr learned my addr 172.21.15.110:0/620321580 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562419db9f50 con 0x562419c94f40 2024-04-27T03:18:12.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff18d8c640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419db9a10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7eff10046040 tx=0x7eff1004aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7efdf5ffb640 1 -- 172.21.15.110:0/620321580 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff10038070 con 0x562419c94f40 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562419dba1e0 con 0x562419c94f40 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7efdf5ffb640 1 -- 172.21.15.110:0/620321580 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff10011730 con 0x562419c94f40 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7efdf5ffb640 1 -- 172.21.15.110:0/620321580 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff10041040 con 0x562419c94f40 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562419dbcec0 con 0x562419c94f40 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.325+0000 7efdf5ffb640 1 -- 172.21.15.110:0/620321580 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7eff1003b070 con 0x562419c94f40 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7efdf5ffb640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efde80346a0 0x7efde8036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7efdf5ffb640 1 -- 172.21.15.110:0/620321580 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7eff1004b030 con 0x562419c94f40 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1858b640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efde80346a0 0x7efde8036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419dc5070 0x562419dc7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dc7c80 con 0x562419dc5070 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff17d8a640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419dc5070 0x562419dc7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1858b640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efde80346a0 0x7efde8036b60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7eff080e5320 tx=0x7eff080e9000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff17d8a640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419dc5070 0x562419dc7530 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7eff0c00f620 con 0x562419dc5070 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 realm 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dc44c0 con 0x562419dc5070 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7eff0c00f620 con 0x562419dc5070 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dc8780 con 0x562419dc5070 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7eff0c00f620 con 0x562419dc5070 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419dc9560 0x562419dcb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dcbfd0 con 0x562419dc9560 2024-04-27T03:18:12.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff18d8c640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419dc9560 0x562419dcb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff18d8c640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419dc9560 0x562419dcb950 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dcc820 con 0x562419dc5070 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7eff0c00f620 con 0x562419dc5070 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 20 searching for the correct realm 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562419dce590 0x562419dd0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dce590 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.329+0000 7eff1858b640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562419dce590 0x562419dd0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff1858b640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562419dce590 0x562419dd0980 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff18d8c640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7eff100133b0 0x7eff1001fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x7eff100133b0 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7eff100133b0 0x7eff1001fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7eff100133b0 0x7eff1001fa80 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c00f620 con 0x7eff100133b0 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c00f620 con 0x562419dc5070 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dce590 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.333+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x7eff100133b0 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x7eff100133b0 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x7eff100133b0 2024-04-27T03:18:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7eff0c02b040 con 0x7eff100133b0 2024-04-27T03:18:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dce590 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dce590 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dce590 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.337+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x7eff100133b0 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x7eff100133b0 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc5070 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c02b040 con 0x562419dc5070 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x7eff100133b0 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dce590 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x7eff100133b0 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dce590 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.341+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1000 con 0x562419dc9560 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dcd1d0 con 0x562419dc5070 2024-04-27T03:18:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dcda70 con 0x562419dc9560 2024-04-27T03:18:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dcce10 con 0x562419dc5070 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 zone default found 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 4 Realm: () 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 zonegroup default 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dd1910 con 0x7eff100133b0 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dd36e0 con 0x562419dce590 2024-04-27T03:18:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dd3ab0 con 0x562419dce590 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 started sync module instance, tier type = 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.345+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd2b80 con 0x562419dc9560 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94713052737440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd27e0 con 0x562419dc9560 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd49e0 con 0x562419dc5070 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94713052742960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd4e00 con 0x562419dc5070 2024-04-27T03:18:12.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd5850 con 0x562419dc9560 2024-04-27T03:18:12.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94713052737440] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94713052742960] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94713052746560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd4950 con 0x562419dc9560 2024-04-27T03:18:12.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.349+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd60f0 con 0x7eff100133b0 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94713052748864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd4950 con 0x7eff100133b0 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd6bc0 con 0x562419dc9560 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94713052748864] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94713052746560] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94713052741392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd6b80 con 0x562419dc9560 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd6fb0 con 0x562419dce590 2024-04-27T03:18:12.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94713052749920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd7800 con 0x562419dce590 2024-04-27T03:18:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd8200 con 0x562419dce590 2024-04-27T03:18:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94713052741392] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.353+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94713052749920] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94713052751536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd7350 con 0x562419dce590 2024-04-27T03:18:12.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd8ad0 con 0x7eff100133b0 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'123 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94713052759584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd8ef0 con 0x7eff100133b0 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=0 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=1 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=2 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=3 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=4 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=5 2024-04-27T03:18:12.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94713052751536] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=6 2024-04-27T03:18:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94713052759584] v27'124 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 20 add_watcher() i=7 2024-04-27T03:18:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419ddbb30 con 0x562419dc5070 2024-04-27T03:18:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.357+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94713052757264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dd8ef0 con 0x562419dc5070 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94713052757264] v27'125 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7efdce7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 5 note: GC not initialized 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7efdcd7e2640 20 reqs_thread_entry: start 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 Filter name: none 2024-04-27T03:18:12.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7efd3effd640 20 reqs_thread_entry: start 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419dff3c0 con 0x562419dc5070 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419e014a0 con 0x7eff100133b0 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv21 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:12.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419e02cf0 con 0x7eff100133b0 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(64 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv4 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 10 cache get: name=default.rgw.meta+users.uid+foo : hit (requested=0x13, cached=0x13) 2024-04-27T03:18:12.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 8.1 8:810902b1:users.uid::fud.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419e07550 con 0x562419dc5070 2024-04-27T03:18:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(65 fud.buckets [call] v27'3 uv2 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x562419dc5070 2024-04-27T03:18:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.361+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~379 in=379b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419e0d500 con 0x7eff100133b0 2024-04-27T03:18:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(66 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [delete,create,call,call,writefull 0~379,setxattr (123)] v27'5 uv5 ondisk = 0) ==== 416+0+0 (crc 0 0 0) 0x7eff0c03e040 con 0x7eff100133b0 2024-04-27T03:18:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 10 moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:12.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [notify cookie 94713052935088 in=764b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419e07980 con 0x562419dc9560 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94713052746560 notify 115964117012 ret 0) ==== 794+0+0 (crc 0 0 0) 0x562419dbcec0 con 0x562419dc9560 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7efdf77fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117012 cookie 94713052746560 notifier 4513 bl.length()=752 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7efdf77fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7efdf77fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7efdf77fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [notify cookie 94713052935088 out=8b] v0'0 uv123 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7efdf77fe640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7efddc003eb0 con 0x562419dc9560 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.2 [notify-ack] v0'0 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94713052935088 notify 115964117012 ret 0) ==== 42+0+48 (crc 0 0 0) 0x562419dba1e0 con 0x562419dc9560 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_sources= new_sources= 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential sources added=[] removed=[] 2024-04-27T03:18:12.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_dests= new_dests= 2024-04-27T03:18:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential dests added=[] removed=[] 2024-04-27T03:18:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419e03390 con 0x7eff100133b0 2024-04-27T03:18:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(69 foo.buckets [call] v27'22 uv22 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7eff0c04a040 con 0x7eff100133b0 2024-04-27T03:18:12.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.365+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 8.9 8:91b71841:root::myfoo:head [delete,create,call version.inc in=38b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419e03390 con 0x7eff100133b0 2024-04-27T03:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(70 myfoo [delete,create,call,writefull 0~165] v27'23 uv23 ondisk = 0) ==== 275+0+0 (crc 0 0 0) 0x7eff0c04a040 con 0x7eff100133b0 2024-04-27T03:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1aa70980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2024-04-27T03:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1aa70980 10 moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1aa70980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:18:12.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94713052935056 in=326b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562419e0c1f0 con 0x562419dce590 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== watch-notify(notify (1) cookie 94713052751536 notify 115964117003 ret 0) ==== 356+0+0 (crc 0 0 0) 0x562419d8b880 con 0x562419dce590 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7efdf6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117003 cookie 94713052751536 notifier 4513 bl.length()=314 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(71 notify.6 [notify cookie 94713052935056 out=8b] v0'0 uv123 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7efdf6ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7efdf6ffd640 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7efdf6ffd640 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7efd4c003410 con 0x562419dce590 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(72 notify.6 [notify-ack] v0'0 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== watch-notify(notify_complete (2) cookie 94713052935056 notify 115964117003 ret 0) ==== 42+0+48 (crc 0 0 0) 0x562419d88c50 con 0x562419dce590 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7efdce7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:73 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94713052757264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x562419dc5070 2024-04-27T03:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(73 data_loggenerations_metadata [watch unwatch cookie 94713052757264] v27'126 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7eff0c04a040 con 0x562419dc5070 2024-04-27T03:18:12.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.369+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94713052737440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x562419dc9560 2024-04-27T03:18:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.373+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(74 notify.0 [watch unwatch cookie 94713052737440] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.373+0000 7eff1aa70980 20 remove_watcher() i=0 2024-04-27T03:18:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.373+0000 7eff1aa70980 2 removed watcher, disabling cache 2024-04-27T03:18:12.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.373+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94713052742960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x562419dc5070 2024-04-27T03:18:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.373+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(75 notify.1 [watch unwatch cookie 94713052742960] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c04a040 con 0x562419dc5070 2024-04-27T03:18:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.373+0000 7eff1aa70980 20 remove_watcher() i=1 2024-04-27T03:18:12.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.373+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:76 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94713052746560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x562419dc9560 2024-04-27T03:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.377+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(76 notify.2 [watch unwatch cookie 94713052746560] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.377+0000 7eff1aa70980 20 remove_watcher() i=2 2024-04-27T03:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.377+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:77 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94713052748864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x7eff100133b0 2024-04-27T03:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.377+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(77 notify.3 [watch unwatch cookie 94713052748864] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c04a040 con 0x7eff100133b0 2024-04-27T03:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.377+0000 7eff1aa70980 20 remove_watcher() i=3 2024-04-27T03:18:12.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.377+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94713052741392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x562419dc9560 2024-04-27T03:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff18d8c640 1 -- 172.21.15.110:0/620321580 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(78 notify.4 [watch unwatch cookie 94713052741392] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff1003a030 con 0x562419dc9560 2024-04-27T03:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 20 remove_watcher() i=4 2024-04-27T03:18:12.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94713052749920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x562419dce590 2024-04-27T03:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(79 notify.5 [watch unwatch cookie 94713052749920] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 20 remove_watcher() i=5 2024-04-27T03:18:12.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:80 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94713052751536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x562419dce590 2024-04-27T03:18:12.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1858b640 1 -- 172.21.15.110:0/620321580 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(80 notify.6 [watch unwatch cookie 94713052751536] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff080e5050 con 0x562419dce590 2024-04-27T03:18:12.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 20 remove_watcher() i=6 2024-04-27T03:18:12.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:81 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94713052759584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562419dfd0f0 con 0x7eff100133b0 2024-04-27T03:18:12.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(81 notify.7 [watch unwatch cookie 94713052759584] v27'125 uv123 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7eff0c04a040 con 0x7eff100133b0 2024-04-27T03:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 20 remove_watcher() i=7 2024-04-27T03:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419dc9560 msgr2=0x562419dcb950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419dc9560 0x562419dcb950 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419dc5070 msgr2=0x562419dc7530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419dc5070 0x562419dc7530 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7eff100133b0 msgr2=0x7eff1001fa80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7eff100133b0 0x7eff1001fa80 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562419dce590 msgr2=0x562419dd0980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562419dce590 0x562419dd0980 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efde80346a0 msgr2=0x7efde8036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efde80346a0 0x7efde8036b60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7eff080e5320 tx=0x7eff080e9000 comp rx=0 tx=0).stop 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff17d8a640 1 -- 172.21.15.110:0/620321580 reap_dead start 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 msgr2=0x562419db9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419db9a10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7eff10046040 tx=0x7eff1004aa30 comp rx=0 tx=0).stop 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 shutdown_connections 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/620321580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c94f40 0x562419db9a10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 >> 172.21.15.110:0/620321580 conn(0x562419dac7a0 msgr2=0x562419da9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 shutdown_connections 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/620321580 wait complete. 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efdf0034650 msgr2=0x7efdf0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efdf0034650 0x7efdf0036b10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x562419d936a0 tx=0x7eff080e3780 comp rx=0 tx=0).stop 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419da3290 msgr2=0x562419da36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419da3290 0x562419da36b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7eff1000dfc0 tx=0x7eff100178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 shutdown_connections 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efdf0034650 0x7efdf0036b10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 --2- 172.21.15.110:0/3064507062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419da3290 0x562419da36b0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 >> 172.21.15.110:0/3064507062 conn(0x562419d980a0 msgr2=0x562419d98bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 shutdown_connections 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.381+0000 7eff1aa70980 1 -- 172.21.15.110:0/3064507062 wait complete. 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419d90670 msgr2=0x562419d92a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419d90670 0x562419d92a60 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419c993e0 msgr2=0x562419d8f360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419c993e0 0x562419d8f360 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efe000346a0 msgr2=0x7efe00036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efe000346a0 0x7efe00036b60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7eff0c000f70 tx=0x7eff0c001380 comp rx=0 tx=0).stop 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 msgr2=0x562419d88480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d88480 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7eff0801b170 tx=0x7eff080180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 shutdown_connections 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562419d90670 0x562419d92a60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562419c993e0 0x562419d8f360 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efe000346a0 0x7efe00036b60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 --2- 172.21.15.110:0/1004126573 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562419c0a830 0x562419d88480 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 >> 172.21.15.110:0/1004126573 conn(0x562419c909c0 msgr2=0x562419d7bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 shutdown_connections 2024-04-27T03:18:12.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.385+0000 7eff1aa70980 1 -- 172.21.15.110:0/1004126573 wait complete. 2024-04-27T03:18:12.401 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud'] 2024-04-27T03:18:12.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud'] 2024-04-27T03:18:12.401 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud 2024-04-27T03:18:12.472 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:12.472 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:12.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b6704980 1 Processor -- start 2024-04-27T03:18:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b6704980 1 -- start start 2024-04-27T03:18:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b6704980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56101103e9c0 0x561011040db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b6704980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x561010ef70d0 con 0x56101103a4b0 2024-04-27T03:18:12.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b6704980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x561010eec430 con 0x56101103e9c0 2024-04-27T03:18:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b3a1e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56101103e9c0 0x561011040db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b3a1e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56101103e9c0 0x561011040db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52158/0 (socket says 172.21.15.110:52158) 2024-04-27T03:18:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b3a1e640 1 -- 172.21.15.110:0/2411023329 learned_addr learned my addr 172.21.15.110:0/2411023329 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b421f640 1 --1- 172.21.15.110:0/2411023329 >> v1:172.21.15.77:6789/0 conn(0x56101103a4b0 0x56101103a890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34900/0 (socket says 172.21.15.110:34900) 2024-04-27T03:18:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b3a1e640 1 -- 172.21.15.110:0/2411023329 >> v1:172.21.15.77:6789/0 conn(0x56101103a4b0 legacy=0x56101103a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.465+0000 7f40b3a1e640 1 -- 172.21.15.110:0/2411023329 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x561010ee5510 con 0x56101103e9c0 2024-04-27T03:18:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b3a1e640 1 --2- 172.21.15.110:0/2411023329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103e9c0 0x561011040db0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f409c00a5c0 tx=0x7f409c01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8222e18977b9cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b321d640 1 -- 172.21.15.110:0/2411023329 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f409c0a6070 con 0x56101103e9c0 2024-04-27T03:18:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b321d640 1 -- 172.21.15.110:0/2411023329 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409c0af070 con 0x56101103e9c0 2024-04-27T03:18:12.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b321d640 1 -- 172.21.15.110:0/2411023329 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f409c0aa5c0 con 0x56101103e9c0 2024-04-27T03:18:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b6704980 1 -- 172.21.15.110:0/2411023329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103e9c0 msgr2=0x561011040db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b6704980 1 --2- 172.21.15.110:0/2411023329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103e9c0 0x561011040db0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f409c00a5c0 tx=0x7f409c01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b6704980 1 -- 172.21.15.110:0/2411023329 shutdown_connections 2024-04-27T03:18:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b6704980 1 --2- 172.21.15.110:0/2411023329 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103e9c0 0x561011040db0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b6704980 1 -- 172.21.15.110:0/2411023329 >> 172.21.15.110:0/2411023329 conn(0x561010fadb90 msgr2=0x561010fadfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b6704980 1 -- 172.21.15.110:0/2411023329 shutdown_connections 2024-04-27T03:18:12.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.469+0000 7f40b6704980 1 -- 172.21.15.110:0/2411023329 wait complete. 2024-04-27T03:18:12.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b6704980 1 Processor -- start 2024-04-27T03:18:12.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b6704980 1 -- start start 2024-04-27T03:18:12.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b6704980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x5610111255d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b6704980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561010f3b2d0 con 0x56101103a4b0 2024-04-27T03:18:12.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b421f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x5610111255d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b421f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x5610111255d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52172/0 (socket says 172.21.15.110:52172) 2024-04-27T03:18:12.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b421f640 1 -- 172.21.15.110:0/2232415225 learned_addr learned my addr 172.21.15.110:0/2232415225 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.481+0000 7f40b421f640 1 -- 172.21.15.110:0/2232415225 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561010ec1ae0 con 0x56101103a4b0 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b421f640 1 --2- 172.21.15.110:0/2232415225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x5610111255d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f40a4001e50 tx=0x7f40a4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef8dbe48c7996df server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f3fa260c640 1 -- 172.21.15.110:0/2232415225 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40a400e5d0 con 0x56101103a4b0 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f3fa260c640 1 -- 172.21.15.110:0/2232415225 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a400e770 con 0x56101103a4b0 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f3fa260c640 1 -- 172.21.15.110:0/2232415225 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40a400eab0 con 0x56101103a4b0 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- 172.21.15.110:0/2232415225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 msgr2=0x5610111255d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 --2- 172.21.15.110:0/2232415225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x5610111255d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f40a4001e50 tx=0x7f40a4001e80 comp rx=0 tx=0).stop 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- 172.21.15.110:0/2232415225 shutdown_connections 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 --2- 172.21.15.110:0/2232415225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x5610111255d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- 172.21.15.110:0/2232415225 >> 172.21.15.110:0/2232415225 conn(0x561010fadb90 msgr2=0x561011121160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- 172.21.15.110:0/2232415225 shutdown_connections 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- 172.21.15.110:0/2232415225 wait complete. 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 Processor -- start 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- start start 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x56101112c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561011043de0 con 0x56101103a4b0 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b421f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x56101112c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b421f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x56101112c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52178/0 (socket says 172.21.15.110:52178) 2024-04-27T03:18:12.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b421f640 1 -- 172.21.15.110:0/3978092762 learned_addr learned my addr 172.21.15.110:0/3978092762 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b421f640 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56101112cb50 con 0x56101103a4b0 2024-04-27T03:18:12.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b421f640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x56101112c610 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x561011126460 tx=0x7f40a4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f3fa0e09640 1 -- 172.21.15.110:0/3978092762 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40a4001260 con 0x56101103a4b0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56101112cde0 con 0x56101103a4b0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f3fa0e09640 1 -- 172.21.15.110:0/3978092762 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a400fa40 con 0x56101103a4b0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56101112fa60 con 0x56101103a4b0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.485+0000 7f3fa0e09640 1 -- 172.21.15.110:0/3978092762 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40a402c060 con 0x56101103a4b0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f3fa0e09640 1 -- 172.21.15.110:0/3978092762 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f40a4017070 con 0x56101103a4b0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f3fa0e09640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f9c034650 0x7f3f9c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f3fa0e09640 1 -- 172.21.15.110:0/3978092762 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f40a4015020 con 0x56101103a4b0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b3a1e640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f9c034650 0x7f3f9c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56101103e9c0 0x561011133540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011133b60 con 0x56101103e9c0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b4a20640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56101103e9c0 0x561011133540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b3a1e640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f9c034650 0x7f3f9c036b10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f409c0ae000 tx=0x7f409c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b4a20640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56101103e9c0 0x561011133540 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b4a20640 1 -- 172.21.15.110:0/3978092762 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f40ac00a4d0 con 0x56101103e9c0 2024-04-27T03:18:12.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561011134950 0x561011136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011137380 con 0x561011134950 2024-04-27T03:18:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b421f640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561011134950 0x561011136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b421f640 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561011134950 0x561011136d40 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b421f640 1 -- 172.21.15.110:0/3978092762 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f40a402e070 con 0x561011134950 2024-04-27T03:18:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011138c30 con 0x56101103e9c0 2024-04-27T03:18:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b4a20640 1 -- 172.21.15.110:0/3978092762 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f40ac00a4d0 con 0x56101103e9c0 2024-04-27T03:18:12.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.489+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011137e60 con 0x561011134950 2024-04-27T03:18:12.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b421f640 1 -- 172.21.15.110:0/3978092762 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f40a402e070 con 0x561011134950 2024-04-27T03:18:12.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011138250 con 0x56101103e9c0 2024-04-27T03:18:12.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b4a20640 1 -- 172.21.15.110:0/3978092762 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f40ac00a4d0 con 0x56101103e9c0 2024-04-27T03:18:12.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b6704980 1 Processor -- start 2024-04-27T03:18:12.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b6704980 1 -- start start 2024-04-27T03:18:12.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b6704980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561011147530 0x561011147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b6704980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561010f3f310 con 0x561011147530 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b4a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561011147530 0x561011147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b4a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561011147530 0x561011147950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52194/0 (socket says 172.21.15.110:52194) 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b4a20640 1 -- 172.21.15.110:0/1854357250 learned_addr learned my addr 172.21.15.110:0/1854357250 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b4a20640 1 -- 172.21.15.110:0/1854357250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561011147e90 con 0x561011147530 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b4a20640 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561011147530 0x561011147950 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00dfc0 tx=0x7f40ac0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f3f99ffb640 1 -- 172.21.15.110:0/1854357250 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac0157d0 con 0x561011147530 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561011148930 con 0x561011147530 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f3f99ffb640 1 -- 172.21.15.110:0/1854357250 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ac015970 con 0x561011147530 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561011148e00 con 0x561011147530 2024-04-27T03:18:12.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.493+0000 7f3f99ffb640 1 -- 172.21.15.110:0/1854357250 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac015c50 con 0x561011147530 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f3f99ffb640 1 -- 172.21.15.110:0/1854357250 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f40ac020440 con 0x561011147530 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f3f99ffb640 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f8c034650 0x7f3f8c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f3f99ffb640 1 -- 172.21.15.110:0/1854357250 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f40ac00e040 con 0x561011147530 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b421f640 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f8c034650 0x7f3f8c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 Processor -- start 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- start start 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b421f640 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f8c034650 0x7f3f8c036b10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x561011137980 tx=0x7f40a4048e00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x561011154b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561011125d20 con 0x5610111546e0 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x561011154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x561011154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52208/0 (socket says 172.21.15.110:52208) 2024-04-27T03:18:12.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 -- 172.21.15.110:0/1056938366 learned_addr learned my addr 172.21.15.110:0/1056938366 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 -- 172.21.15.110:0/1056938366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610111550d0 con 0x5610111546e0 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 --2- 172.21.15.110:0/1056938366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x561011154b00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f40ac04aaa0 tx=0x7f40ac020940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d15fdf54950795d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f3f937fe640 1 -- 172.21.15.110:0/1056938366 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac048070 con 0x5610111546e0 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f3f937fe640 1 -- 172.21.15.110:0/1056938366 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ac0113a0 con 0x5610111546e0 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f3f937fe640 1 -- 172.21.15.110:0/1056938366 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac0133b0 con 0x5610111546e0 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- 172.21.15.110:0/1056938366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 msgr2=0x561011154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 --2- 172.21.15.110:0/1056938366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x561011154b00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f40ac04aaa0 tx=0x7f40ac020940 comp rx=0 tx=0).stop 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- 172.21.15.110:0/1056938366 shutdown_connections 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 --2- 172.21.15.110:0/1056938366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x561011154b00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- 172.21.15.110:0/1056938366 >> 172.21.15.110:0/1056938366 conn(0x561011150a70 msgr2=0x56101114ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- 172.21.15.110:0/1056938366 shutdown_connections 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- 172.21.15.110:0/1056938366 wait complete. 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 Processor -- start 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- start start 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x56101115bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b6704980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561011155840 con 0x5610111546e0 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x56101115bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x56101115bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52218/0 (socket says 172.21.15.110:52218) 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 learned_addr learned my addr 172.21.15.110:0/55123193 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.497+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56101115c150 con 0x5610111546e0 2024-04-27T03:18:12.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b4a20640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x56101115bc10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0118e0 tx=0x7f40ac04aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f3f91ffb640 1 -- 172.21.15.110:0/55123193 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac037020 con 0x5610111546e0 2024-04-27T03:18:12.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56101115c3e0 con 0x5610111546e0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f3f91ffb640 1 -- 172.21.15.110:0/55123193 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ac01fde0 con 0x5610111546e0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f3f91ffb640 1 -- 172.21.15.110:0/55123193 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40ac04c200 con 0x5610111546e0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56101115f0c0 con 0x5610111546e0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f3f91ffb640 1 -- 172.21.15.110:0/55123193 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f40ac03a020 con 0x5610111546e0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f3f91ffb640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f84034650 0x7f3f84036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f3f91ffb640 1 -- 172.21.15.110:0/55123193 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f40ac02f070 con 0x5610111546e0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b421f640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f84034650 0x7f3f84036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561011167230 0x5610111696c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011169e10 con 0x561011167230 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b3a1e640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561011167230 0x5610111696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b421f640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f84034650 0x7f3f84036b10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40a400a3f0 tx=0x7f40a4038000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b3a1e640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561011167230 0x5610111696c0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 realm 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101116a240 con 0x561011167230 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101116ab10 con 0x561011167230 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56101116b530 0x56101116d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101116e0d0 con 0x56101116b530 2024-04-27T03:18:12.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.501+0000 7f40b4a20640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56101116b530 0x56101116d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b4a20640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56101116b530 0x56101116d960 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b6704980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101116e920 con 0x561011167230 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b6704980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b6704980 20 searching for the correct realm 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561011170690 0x561011172a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561011173100 con 0x561011170690 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b421f640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561011170690 0x561011172a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b421f640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561011170690 0x561011172a80 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561011173100 con 0x56101116b530 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b4a20640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f40ac01f3e0 0x7f40ac03c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561011173100 con 0x7f40ac01f3e0 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b3a1e640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f40ac01f3e0 0x7f40ac03c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b3a1e640 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f40ac01f3e0 0x7f40ac03c790 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.505+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561011173100 con 0x561011167230 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561011173100 con 0x561011167230 2024-04-27T03:18:12.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561011173100 con 0x561011170690 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x7f40ac01f3e0 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x7f40ac01f3e0 2024-04-27T03:18:12.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.509+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011170690 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011170690 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011170690 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x7f40ac01f3e0 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011167230 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x7f40ac01f3e0 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011170690 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x7f40ac01f3e0 2024-04-27T03:18:12.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.513+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x561011170690 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f40a4054040 con 0x56101116b530 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101116f2a0 con 0x561011167230 2024-04-27T03:18:12.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101116eee0 con 0x56101116b530 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101116fc00 con 0x561011167230 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 zone default found 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 4 Realm: () 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:12.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 zonegroup default 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011173a10 con 0x7f40ac01f3e0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.517+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011175c90 con 0x561011170690 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561011176030 con 0x561011170690 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 20 started sync module instance, tier type = 2024-04-27T03:18:12.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111750d0 con 0x56101116b530 2024-04-27T03:18:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94627006207168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111758e0 con 0x56101116b530 2024-04-27T03:18:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011176d70 con 0x561011167230 2024-04-27T03:18:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94627006212224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011177610 con 0x561011167230 2024-04-27T03:18:12.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.521+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011178050 con 0x56101116b530 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94627006207168] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94627006212224] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94627006216992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011177160 con 0x56101116b530 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111788f0 con 0x7f40ac01f3e0 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94627006219328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011177160 con 0x7f40ac01f3e0 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111793b0 con 0x56101116b530 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.525+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94627006219328] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94627006216992] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94627006211024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011179370 con 0x56101116b530 2024-04-27T03:18:12.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111797a0 con 0x561011170690 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94627006220384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011179ff0 con 0x561011170690 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56101117a9b0 con 0x561011170690 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94627006211024] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94627006220384] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94627006221952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56101117a9b0 con 0x561011170690 2024-04-27T03:18:12.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56101117b290 con 0x7f40ac01f3e0 2024-04-27T03:18:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94627006221952] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'126 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94627006229920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56101117a9b0 con 0x7f40ac01f3e0 2024-04-27T03:18:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.529+0000 7f40b6704980 20 add_watcher() i=0 2024-04-27T03:18:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 add_watcher() i=1 2024-04-27T03:18:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 add_watcher() i=2 2024-04-27T03:18:12.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 add_watcher() i=3 2024-04-27T03:18:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 add_watcher() i=4 2024-04-27T03:18:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 add_watcher() i=5 2024-04-27T03:18:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 add_watcher() i=6 2024-04-27T03:18:12.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94627006229920] v27'127 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 add_watcher() i=7 2024-04-27T03:18:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56101117e740 con 0x561011167230 2024-04-27T03:18:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94627006228176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56101117bc60 con 0x561011167230 2024-04-27T03:18:12.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94627006228176] v27'127 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f3f6a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:12.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:12.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.533+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56101117e500 con 0x56101116b530 2024-04-27T03:18:12.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.537+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'45 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.537+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56101117e560 con 0x7f40ac01f3e0 2024-04-27T03:18:12.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.537+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'22 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.537+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011187890 con 0x561011170690 2024-04-27T03:18:12.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.541+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'22 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.541+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011187cb0 con 0x561011167230 2024-04-27T03:18:12.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.541+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'128 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.541+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188220 con 0x56101116b530 2024-04-27T03:18:12.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.545+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'43 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.545+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188220 con 0x561011170690 2024-04-27T03:18:12.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.545+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'35 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.545+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188220 con 0x56101116b530 2024-04-27T03:18:12.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.545+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'23 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.545+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188220 con 0x561011167230 2024-04-27T03:18:12.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.549+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'30 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.549+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.549+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'15 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.549+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.553+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'21 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.553+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.553+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'44 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.553+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011170690 2024-04-27T03:18:12.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.557+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'17 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.557+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188840 con 0x56101116b530 2024-04-27T03:18:12.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.557+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'21 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.557+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188840 con 0x561011167230 2024-04-27T03:18:12.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.557+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'41 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.557+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111888d0 con 0x561011167230 2024-04-27T03:18:12.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.561+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'129 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.561+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111888d0 con 0x56101116b530 2024-04-27T03:18:12.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.561+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'35 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.561+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111888d0 con 0x56101116b530 2024-04-27T03:18:12.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.565+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'45 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.565+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188840 con 0x561011170690 2024-04-27T03:18:12.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.569+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'28 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.569+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011188840 con 0x56101116b530 2024-04-27T03:18:12.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.569+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'23 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.569+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011167230 2024-04-27T03:18:12.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.573+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'130 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.573+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.573+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'36 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.573+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.577+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'31 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.577+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011170690 2024-04-27T03:18:12.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.577+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'23 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.577+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.581+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'33 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.581+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011170690 2024-04-27T03:18:12.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.581+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'17 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.581+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011170690 2024-04-27T03:18:12.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.581+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'36 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.581+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011170690 2024-04-27T03:18:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.585+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'23 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.585+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.585+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'27 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.585+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011167230 2024-04-27T03:18:12.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.589+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'31 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.589+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x56101116b530 2024-04-27T03:18:12.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.589+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'37 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.589+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x561011167230 2024-04-27T03:18:12.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'42 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111887b0 con 0x7f40ac01f3e0 2024-04-27T03:18:12.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'21 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f3f697e2640 20 reqs_thread_entry: start 2024-04-27T03:18:12.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:12.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 20 Filter name: none 2024-04-27T03:18:12.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f3edaffd640 20 reqs_thread_entry: start 2024-04-27T03:18:12.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:18:12.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5610111873d0 con 0x7f40ac01f3e0 2024-04-27T03:18:12.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(94 fud [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:18:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2024-04-27T03:18:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:18:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.593+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 8.1 8:810902b1:users.uid::fud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5610111a67a0 con 0x561011167230 2024-04-27T03:18:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(95 fud.buckets [call out=15b] v0'0 uv2 ondisk = 0) ==== 155+0+15 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b6704980 10 removing key index: p5YnriCv1nAtykxBrupQ 2024-04-27T03:18:12.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:96 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111a4780 con 0x561011170690 2024-04-27T03:18:12.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(96 p5YnriCv1nAtykxBrupQ [delete] v27'6 uv6 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b6704980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:18:12.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [notify cookie 94627006403296 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5610111a6d80 con 0x56101116b530 2024-04-27T03:18:12.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== watch-notify(notify (1) cookie 94627006216992 notify 115964117013 ret 0) ==== 212+0+0 (crc 0 0 0) 0x56101117dd20 con 0x56101116b530 2024-04-27T03:18:12.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f3f937fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117013 cookie 94627006216992 notifier 4520 bl.length()=170 2024-04-27T03:18:12.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(97 notify.2 [notify cookie 94627006403296 out=8b] v0'0 uv126 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f3f937fe640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f3f78003d70 con 0x56101116b530 2024-04-27T03:18:12.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(98 notify.2 [notify-ack] v0'0 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== watch-notify(notify_complete (2) cookie 94627006403296 notify 115964117013 ret 0) ==== 42+0+48 (crc 0 0 0) 0x56101115c3e0 con 0x56101116b530 2024-04-27T03:18:12.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b6704980 10 removing email index: 2024-04-27T03:18:12.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b6704980 10 removing user buckets index 2024-04-27T03:18:12.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.597+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:99 8.1 8:810902b1:users.uid::fud.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111a6ce0 con 0x561011167230 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(99 fud.buckets [delete] v27'4 uv4 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b6704980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:fud.buckets, ofs0, ns] 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94627006409952 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5610111a67d0 con 0x561011170690 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== watch-notify(notify (1) cookie 94627006221952 notify 115964117004 ret 0) ==== 202+0+0 (crc 0 0 0) 0x561011188dd0 con 0x561011170690 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f3f937fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117004 cookie 94627006221952 notifier 4520 bl.length()=160 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(100 notify.6 [notify cookie 94627006409952 out=8b] v0'0 uv126 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f3f937fe640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f3f78003d70 con 0x561011170690 2024-04-27T03:18:12.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== watch-notify(notify_complete (2) cookie 94627006409952 notify 115964117004 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f40a405d040 con 0x561011170690 2024-04-27T03:18:12.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b6704980 10 removing user index: fud 2024-04-27T03:18:12.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(101 notify.6 [notify-ack] v0'0 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 8.e 8:75f48285:users.uid::fud:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111a6ce0 con 0x7f40ac01f3e0 2024-04-27T03:18:12.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(102 fud [call,call,delete] v27'2 uv2 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b6704980 10 removing default.rgw.meta+users.uid+fud from cache 2024-04-27T03:18:12.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b6704980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2024-04-27T03:18:12.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:103 7.3 7:c609908c:::notify.5:head [notify cookie 94627006409952 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5610111a5740 con 0x561011170690 2024-04-27T03:18:12.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== watch-notify(notify (1) cookie 94627006220384 notify 115964117005 ret 0) ==== 194+0+0 (crc 0 0 0) 0x7f40a405d040 con 0x561011170690 2024-04-27T03:18:12.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f3f92ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117005 cookie 94627006220384 notifier 4520 bl.length()=152 2024-04-27T03:18:12.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(103 notify.5 [notify cookie 94627006409952 out=8b] v0'0 uv126 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f3f92ffd640 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:104 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f3ee80032d0 con 0x561011170690 2024-04-27T03:18:12.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.601+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(104 notify.5 [notify-ack] v0'0 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.605+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== watch-notify(notify_complete (2) cookie 94627006409952 notify 115964117005 ret 0) ==== 42+0+48 (crc 0 0 0) 0x56101112fa60 con 0x561011170690 2024-04-27T03:18:12.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.605+0000 7f3f6a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.605+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94627006228176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111a6ce0 con 0x561011167230 2024-04-27T03:18:12.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.605+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(105 data_loggenerations_metadata [watch unwatch cookie 94627006228176] v27'131 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.605+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94627006207168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111a6ce0 con 0x56101116b530 2024-04-27T03:18:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.609+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(106 notify.0 [watch unwatch cookie 94627006207168] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.609+0000 7f40b6704980 20 remove_watcher() i=0 2024-04-27T03:18:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.609+0000 7f40b6704980 2 removed watcher, disabling cache 2024-04-27T03:18:12.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.609+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:107 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94627006212224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111754c0 con 0x561011167230 2024-04-27T03:18:12.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.609+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(107 notify.1 [watch unwatch cookie 94627006212224] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x561011167230 2024-04-27T03:18:12.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.609+0000 7f40b6704980 20 remove_watcher() i=1 2024-04-27T03:18:12.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.609+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94627006216992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111754c0 con 0x56101116b530 2024-04-27T03:18:12.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(108 notify.2 [watch unwatch cookie 94627006216992] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b6704980 20 remove_watcher() i=2 2024-04-27T03:18:12.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:109 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94627006219328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111754c0 con 0x7f40ac01f3e0 2024-04-27T03:18:12.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(109 notify.3 [watch unwatch cookie 94627006219328] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b6704980 20 remove_watcher() i=3 2024-04-27T03:18:12.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:110 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94627006211024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011178440 con 0x56101116b530 2024-04-27T03:18:12.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b4a20640 1 -- 172.21.15.110:0/55123193 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(110 notify.4 [watch unwatch cookie 94627006211024] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40ac026020 con 0x56101116b530 2024-04-27T03:18:12.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b6704980 20 remove_watcher() i=4 2024-04-27T03:18:12.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.613+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:111 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94627006220384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111754c0 con 0x561011170690 2024-04-27T03:18:12.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(111 notify.5 [watch unwatch cookie 94627006220384] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 20 remove_watcher() i=5 2024-04-27T03:18:12.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:112 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94627006221952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5610111754c0 con 0x561011170690 2024-04-27T03:18:12.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b421f640 1 -- 172.21.15.110:0/55123193 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(112 notify.6 [watch unwatch cookie 94627006221952] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f40a402e070 con 0x561011170690 2024-04-27T03:18:12.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 20 remove_watcher() i=6 2024-04-27T03:18:12.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:113 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94627006229920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561011178e80 con 0x7f40ac01f3e0 2024-04-27T03:18:12.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(113 notify.7 [watch unwatch cookie 94627006229920] v27'128 uv126 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f409c0aaa20 con 0x7f40ac01f3e0 2024-04-27T03:18:12.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 20 remove_watcher() i=7 2024-04-27T03:18:12.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56101116b530 msgr2=0x56101116d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56101116b530 0x56101116d960 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561011167230 msgr2=0x5610111696c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561011167230 0x5610111696c0 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f40ac01f3e0 msgr2=0x7f40ac03c790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f40ac01f3e0 0x7f40ac03c790 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561011170690 msgr2=0x561011172a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561011170690 0x561011172a80 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f84034650 msgr2=0x7f3f84036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f84034650 0x7f3f84036b10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40a400a3f0 tx=0x7f40a4038000 comp rx=0 tx=0).stop 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 msgr2=0x56101115bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b3a1e640 1 -- 172.21.15.110:0/55123193 reap_dead start 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x56101115bc10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0118e0 tx=0x7f40ac04aa30 comp rx=0 tx=0).stop 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 shutdown_connections 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/55123193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5610111546e0 0x56101115bc10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 >> 172.21.15.110:0/55123193 conn(0x561011150a70 msgr2=0x561011155f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 shutdown_connections 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/55123193 wait complete. 2024-04-27T03:18:12.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f8c034650 msgr2=0x7f3f8c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f8c034650 0x7f3f8c036b10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x561011137980 tx=0x7f40a4048e00 comp rx=0 tx=0).stop 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561011147530 msgr2=0x561011147950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561011147530 0x561011147950 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00dfc0 tx=0x7f40ac0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 shutdown_connections 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f8c034650 0x7f3f8c036b10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 --2- 172.21.15.110:0/1854357250 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561011147530 0x561011147950 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 >> 172.21.15.110:0/1854357250 conn(0x56101113c340 msgr2=0x56101113ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 shutdown_connections 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.617+0000 7f40b6704980 1 -- 172.21.15.110:0/1854357250 wait complete. 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561011134950 msgr2=0x561011136d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561011134950 0x561011136d40 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56101103e9c0 msgr2=0x561011133540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56101103e9c0 0x561011133540 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f9c034650 msgr2=0x7f3f9c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f9c034650 0x7f3f9c036b10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f409c0ae000 tx=0x7f409c0a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 msgr2=0x56101112c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x56101112c610 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x561011126460 tx=0x7f40a4000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 shutdown_connections 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561011134950 0x561011136d40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56101103e9c0 0x561011133540 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f3f9c034650 0x7f3f9c036b10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 --2- 172.21.15.110:0/3978092762 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56101103a4b0 0x56101112c610 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 >> 172.21.15.110:0/3978092762 conn(0x561010fadb90 msgr2=0x56101111fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 shutdown_connections 2024-04-27T03:18:12.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.621+0000 7f40b6704980 1 -- 172.21.15.110:0/3978092762 wait complete. 2024-04-27T03:18:12.637 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--tenant', 'testx', '--uid', 'tenanteduser', '--display-name', 'tenanted-user', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2024-04-27T03:18:12.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--tenant', 'testx', '--uid', 'tenanteduser', '--display-name', 'tenanted-user', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '1'] 2024-04-27T03:18:12.637 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --tenant testx --uid tenanteduser --display-name tenanted-user --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh --max-buckets 1 2024-04-27T03:18:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:12.723 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28df605980 1 Processor -- start 2024-04-27T03:18:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28df605980 1 -- start start 2024-04-27T03:18:12.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28df605980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ddd6297650 0x55ddd6299a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28df605980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55ddd61530d0 con 0x55ddd62930f0 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28df605980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55ddd6148430 con 0x55ddd6297650 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28dc91f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ddd6297650 0x55ddd6299a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28dc91f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ddd6297650 0x55ddd6299a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52228/0 (socket says 172.21.15.110:52228) 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28dc91f640 1 -- 172.21.15.110:0/1482505815 learned_addr learned my addr 172.21.15.110:0/1482505815 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28dd120640 1 --1- 172.21.15.110:0/1482505815 >> v1:172.21.15.77:6789/0 conn(0x55ddd62930f0 0x55ddd62934d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34902/0 (socket says 172.21.15.110:34902) 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28dc91f640 1 -- 172.21.15.110:0/1482505815 >> v1:172.21.15.77:6789/0 conn(0x55ddd62930f0 legacy=0x55ddd62934d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.717+0000 7f28dc91f640 1 -- 172.21.15.110:0/1482505815 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd6141510 con 0x55ddd6297650 2024-04-27T03:18:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28dc91f640 1 --2- 172.21.15.110:0/1482505815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd6299a40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f28d000a4e0 tx=0x7f28d001b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d427a77d9eea82 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28d7fff640 1 -- 172.21.15.110:0/1482505815 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d00aa540 con 0x55ddd6297650 2024-04-27T03:18:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28d7fff640 1 -- 172.21.15.110:0/1482505815 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d00aa6e0 con 0x55ddd6297650 2024-04-27T03:18:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28d7fff640 1 -- 172.21.15.110:0/1482505815 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d00aaa40 con 0x55ddd6297650 2024-04-27T03:18:12.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28df605980 1 -- 172.21.15.110:0/1482505815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 msgr2=0x55ddd6299a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28df605980 1 --2- 172.21.15.110:0/1482505815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd6299a40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f28d000a4e0 tx=0x7f28d001b090 comp rx=0 tx=0).stop 2024-04-27T03:18:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28df605980 1 -- 172.21.15.110:0/1482505815 shutdown_connections 2024-04-27T03:18:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28df605980 1 --2- 172.21.15.110:0/1482505815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd6299a40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28df605980 1 -- 172.21.15.110:0/1482505815 >> 172.21.15.110:0/1482505815 conn(0x55ddd628e9c0 msgr2=0x55ddd6290de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28df605980 1 -- 172.21.15.110:0/1482505815 shutdown_connections 2024-04-27T03:18:12.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.721+0000 7f28df605980 1 -- 172.21.15.110:0/1482505815 wait complete. 2024-04-27T03:18:12.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28df605980 1 Processor -- start 2024-04-27T03:18:12.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28df605980 1 -- start start 2024-04-27T03:18:12.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28df605980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd62930f0 0x55ddd6207f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28df605980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ddd61972d0 con 0x55ddd62930f0 2024-04-27T03:18:12.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28dd120640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd62930f0 0x55ddd6207f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28dd120640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd62930f0 0x55ddd6207f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52240/0 (socket says 172.21.15.110:52240) 2024-04-27T03:18:12.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28dd120640 1 -- 172.21.15.110:0/3050738499 learned_addr learned my addr 172.21.15.110:0/3050738499 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28dd120640 1 -- 172.21.15.110:0/3050738499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd611dae0 con 0x55ddd62930f0 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f28dd120640 1 --2- 172.21.15.110:0/3050738499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd62930f0 0x55ddd6207f00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f28c8001e50 tx=0x7f28c8001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4f5f2018a03212 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f27cb606640 1 -- 172.21.15.110:0/3050738499 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c800e5d0 con 0x55ddd62930f0 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f27cb606640 1 -- 172.21.15.110:0/3050738499 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c800e770 con 0x55ddd62930f0 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.733+0000 7f27cb606640 1 -- 172.21.15.110:0/3050738499 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c800eab0 con 0x55ddd62930f0 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- 172.21.15.110:0/3050738499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd62930f0 msgr2=0x55ddd6207f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 --2- 172.21.15.110:0/3050738499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd62930f0 0x55ddd6207f00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f28c8001e50 tx=0x7f28c8001e80 comp rx=0 tx=0).stop 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- 172.21.15.110:0/3050738499 shutdown_connections 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 --2- 172.21.15.110:0/3050738499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd62930f0 0x55ddd6207f00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- 172.21.15.110:0/3050738499 >> 172.21.15.110:0/3050738499 conn(0x55ddd628e9c0 msgr2=0x55ddd6379ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- 172.21.15.110:0/3050738499 shutdown_connections 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- 172.21.15.110:0/3050738499 wait complete. 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 Processor -- start 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- start start 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd63866e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ddd629ca70 con 0x55ddd6297650 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28dd120640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd63866e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28dd120640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd63866e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52248/0 (socket says 172.21.15.110:52248) 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28dd120640 1 -- 172.21.15.110:0/1962013720 learned_addr learned my addr 172.21.15.110:0/1962013720 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28dd120640 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd6386c20 con 0x55ddd6297650 2024-04-27T03:18:12.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28dd120640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd63866e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f28c8004000 tx=0x7f28c8000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f27c9e03640 1 -- 172.21.15.110:0/1962013720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c8001260 con 0x55ddd6297650 2024-04-27T03:18:12.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ddd6386eb0 con 0x55ddd6297650 2024-04-27T03:18:12.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f27c9e03640 1 -- 172.21.15.110:0/1962013720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c800f9a0 con 0x55ddd6297650 2024-04-27T03:18:12.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f27c9e03640 1 -- 172.21.15.110:0/1962013720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28c800fce0 con 0x55ddd6297650 2024-04-27T03:18:12.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddd6389b30 con 0x55ddd6297650 2024-04-27T03:18:12.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.737+0000 7f27c9e03640 1 -- 172.21.15.110:0/1962013720 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f28c8017070 con 0x55ddd6297650 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f27c9e03640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c4034650 0x7f27c4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f27c9e03640 1 -- 172.21.15.110:0/1962013720 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f28c8015020 con 0x55ddd6297650 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dc91f640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c4034650 0x7f27c4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd638b600 0x55ddd638d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd638e010 con 0x55ddd638b600 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd921640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd638b600 0x55ddd638d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dc91f640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c4034650 0x7f27c4036b10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f28d00ab000 tx=0x7f28d00a5000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd921640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd638b600 0x55ddd638d9f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd921640 1 -- 172.21.15.110:0/1962013720 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d80083f0 con 0x55ddd638b600 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd638ee00 0x55ddd63911f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd6391830 con 0x55ddd638ee00 2024-04-27T03:18:12.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd120640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd638ee00 0x55ddd63911f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd120640 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd638ee00 0x55ddd63911f0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd120640 1 -- 172.21.15.110:0/1962013720 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd638ee00 2024-04-27T03:18:12.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63930e0 con 0x55ddd638b600 2024-04-27T03:18:12.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd921640 1 -- 172.21.15.110:0/1962013720 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f28d80083f0 con 0x55ddd638b600 2024-04-27T03:18:12.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd6392310 con 0x55ddd638ee00 2024-04-27T03:18:12.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28dd120640 1 -- 172.21.15.110:0/1962013720 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd638ee00 2024-04-27T03:18:12.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.741+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd6392700 con 0x55ddd638b600 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28dd921640 1 -- 172.21.15.110:0/1962013720 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f28d80083f0 con 0x55ddd638b600 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28df605980 1 Processor -- start 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28df605980 1 -- start start 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28df605980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd63a1a20 0x55ddd63a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28df605980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ddd619b310 con 0x55ddd63a1a20 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28dd921640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd63a1a20 0x55ddd63a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28dd921640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd63a1a20 0x55ddd63a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52256/0 (socket says 172.21.15.110:52256) 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28dd921640 1 -- 172.21.15.110:0/2686991267 learned_addr learned my addr 172.21.15.110:0/2686991267 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28dd921640 1 -- 172.21.15.110:0/2686991267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd63a2380 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28dd921640 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd63a1a20 0x55ddd63a1e40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f28d800aab0 tx=0x7f28d800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f27c2ffd640 1 -- 172.21.15.110:0/2686991267 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8007d30 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ddd63a2e20 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f27c2ffd640 1 -- 172.21.15.110:0/2686991267 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d801d070 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f27c2ffd640 1 -- 172.21.15.110:0/2686991267 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8018440 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddd63a32f0 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f27c2ffd640 1 -- 172.21.15.110:0/2686991267 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f28d8013070 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f27c2ffd640 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27b4034650 0x7f27b4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f27c2ffd640 1 -- 172.21.15.110:0/2686991267 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f28d804b5f0 con 0x55ddd63a1a20 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.745+0000 7f28dd120640 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27b4034650 0x7f27b4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 Processor -- start 2024-04-27T03:18:12.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- start start 2024-04-27T03:18:12.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd120640 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27b4034650 0x7f27b4036b10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55ddd6391e30 tx=0x7f28c8028f80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd637fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ddd6208650 con 0x55ddd637f7f0 2024-04-27T03:18:12.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd637fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd637fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52272/0 (socket says 172.21.15.110:52272) 2024-04-27T03:18:12.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 -- 172.21.15.110:0/879984606 learned_addr learned my addr 172.21.15.110:0/879984606 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 -- 172.21.15.110:0/879984606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd637e040 con 0x55ddd637f7f0 2024-04-27T03:18:12.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 --2- 172.21.15.110:0/879984606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd637fc10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f28d8049940 tx=0x7f28d803a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1d7d0d23857170 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f27c0ff9640 1 -- 172.21.15.110:0/879984606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d803f020 con 0x55ddd637f7f0 2024-04-27T03:18:12.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f27c0ff9640 1 -- 172.21.15.110:0/879984606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d803b310 con 0x55ddd637f7f0 2024-04-27T03:18:12.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f27c0ff9640 1 -- 172.21.15.110:0/879984606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d8022460 con 0x55ddd637f7f0 2024-04-27T03:18:12.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- 172.21.15.110:0/879984606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 msgr2=0x55ddd637fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 --2- 172.21.15.110:0/879984606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd637fc10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f28d8049940 tx=0x7f28d803a110 comp rx=0 tx=0).stop 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- 172.21.15.110:0/879984606 shutdown_connections 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 --2- 172.21.15.110:0/879984606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd637fc10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- 172.21.15.110:0/879984606 >> 172.21.15.110:0/879984606 conn(0x55ddd63aaf30 msgr2=0x55ddd63a8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- 172.21.15.110:0/879984606 shutdown_connections 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- 172.21.15.110:0/879984606 wait complete. 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 Processor -- start 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- start start 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd63b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28df605980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ddd637e510 con 0x55ddd637f7f0 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd63b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd63b82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52274/0 (socket says 172.21.15.110:52274) 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 learned_addr learned my addr 172.21.15.110:0/498406747 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd63b8800 con 0x55ddd637f7f0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f28dd921640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd63b82c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f28d80168b0 tx=0x7f28d80498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.749+0000 7f27b2ffd640 1 -- 172.21.15.110:0/498406747 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d803f040 con 0x55ddd637f7f0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f27b2ffd640 1 -- 172.21.15.110:0/498406747 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d8020c80 con 0x55ddd637f7f0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ddd63b8a90 con 0x55ddd637f7f0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f27b2ffd640 1 -- 172.21.15.110:0/498406747 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d804b0d0 con 0x55ddd637f7f0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddd63bb770 con 0x55ddd637f7f0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f27b2ffd640 1 -- 172.21.15.110:0/498406747 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f28d8016040 con 0x55ddd637f7f0 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f27b2ffd640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27a8034650 0x7f27a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f27b2ffd640 1 -- 172.21.15.110:0/498406747 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f28d804a710 con 0x55ddd637f7f0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dd120640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27a8034650 0x7f27a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd63c3970 0x55ddd63c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63c6560 con 0x55ddd63c3970 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dc91f640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd63c3970 0x55ddd63c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dd120640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27a8034650 0x7f27a8036b10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f28c8044010 tx=0x7f28c8033040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dc91f640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd63c3970 0x55ddd63c5e10 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 realm 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63c6930 con 0x55ddd63c3970 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63c7200 con 0x55ddd63c3970 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd63c7c20 0x55ddd63ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63ca7c0 con 0x55ddd63c7c20 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dd921640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd63c7c20 0x55ddd63ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.753+0000 7f28dd921640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd63c7c20 0x55ddd63ca050 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28df605980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cb010 con 0x55ddd63c3970 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28df605980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28df605980 20 searching for the correct realm 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ddd63ccd80 0x55ddd63cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd120640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ddd63ccd80 0x55ddd63cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd120640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ddd63ccd80 0x55ddd63cf170 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd921640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28d8034c20 0x7f28d804aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x7f28d8034c20 2024-04-27T03:18:12.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28d8034c20 0x7f28d804aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28d8034c20 0x7f28d804aa30 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.757+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x7f28d8034c20 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x7f28d8034c20 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.761+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x7f28d8034c20 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c3970 2024-04-27T03:18:12.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x7f28d8034c20 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x7f28d8034c20 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.765+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cf7a0 con 0x55ddd63c7c20 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:12.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cb960 con 0x55ddd63c3970 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cc200 con 0x55ddd63c7c20 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63cb5a0 con 0x55ddd63c3970 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 zone default found 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 4 Realm: () 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 zonegroup default 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d00d0 con 0x7f28d8034c20 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d2300 con 0x55ddd63ccd80 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d26a0 con 0x55ddd63ccd80 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 started sync module instance, tier type = 2024-04-27T03:18:12.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:12.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.769+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d1790 con 0x55ddd63c7c20 2024-04-27T03:18:12.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94411270462384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d13f0 con 0x55ddd63c7c20 2024-04-27T03:18:12.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d3430 con 0x55ddd63c3970 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94411270467392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d3360 con 0x55ddd63c3970 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d4330 con 0x55ddd63c7c20 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94411270462384] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.773+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94411270467392] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94411270469664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d3e40 con 0x55ddd63c7c20 2024-04-27T03:18:12.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d4c00 con 0x7f28d8034c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94411270473552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d3e40 con 0x7f28d8034c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d5640 con 0x55ddd63c7c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94411270469664] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94411270473552] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94411270474608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d5190 con 0x55ddd63c7c20 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.777+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d5ee0 con 0x55ddd63ccd80 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94411270478384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d5190 con 0x55ddd63ccd80 2024-04-27T03:18:12.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d6d50 con 0x55ddd63ccd80 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94411270474608] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94411270478384] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94411270482080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d70f0 con 0x55ddd63ccd80 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d7940 con 0x7f28d8034c20 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94411270482080] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'129 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94411270484208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d79c0 con 0x7f28d8034c20 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 20 add_watcher() i=0 2024-04-27T03:18:12.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 20 add_watcher() i=1 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 20 add_watcher() i=2 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 20 add_watcher() i=3 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 20 add_watcher() i=4 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 20 add_watcher() i=5 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28df605980 20 add_watcher() i=6 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.781+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94411270484208] v27'130 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 add_watcher() i=7 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:12.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63dacd0 con 0x55ddd63c3970 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94411270485632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63d7f50 con 0x55ddd63c3970 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94411270485632] v27'132 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f27937e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 5 note: GC not initialized 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f27927e4640 20 reqs_thread_entry: start 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 Filter name: none 2024-04-27T03:18:12.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f2703fff640 20 reqs_thread_entry: start 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : miss 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:62 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fe800 con 0x55ddd63ccd80 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(62 testx$tenanteduser [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 246+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x0 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 adding default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fcd90 con 0x55ddd63ccd80 2024-04-27T03:18:12.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(63 p5YnriCv1nAtykxBrupQ [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 206+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.785+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [delete,create,call version.set in=58b,writefull 0~419 in=419b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd6401600 con 0x55ddd63ccd80 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(64 testx$tenanteduser [delete,create,call,writefull 0~419] v27'7 uv7 ondisk = 0) ==== 288+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x17 2024-04-27T03:18:12.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 10 moving default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser, ofs0, ns] 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 7.e 7:7759931f:::notify.3:head [notify cookie 94411270655920 in=622b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd64020d0 con 0x7f28d8034c20 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== watch-notify(notify (1) cookie 94411270473552 notify 115964117000 ret 0) ==== 652+0+0 (crc 0 0 0) 0x55ddd6141510 con 0x7f28d8034c20 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27c0ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117000 cookie 94411270473552 notifier 4527 bl.length()=610 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27c0ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x17 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27c0ff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(65 notify.3 [notify cookie 94411270655920 out=8b] v0'0 uv129 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27c0ff9640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f27a0003eb0 con 0x7f28d8034c20 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(66 notify.3 [notify-ack] v0'0 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== watch-notify(notify_complete (2) cookie 94411270655920 notify 115964117000 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f28d00c1a00 con 0x7f28d8034c20 2024-04-27T03:18:12.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd6400fb0 con 0x55ddd63ccd80 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(67 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~22] v27'8 uv8 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify cookie 94411270655920 in=204b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ddd64020d0 con 0x55ddd63c7c20 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94411270469664 notify 115964117014 ret 0) ==== 234+0+0 (crc 0 0 0) 0x55ddd63bb770 con 0x55ddd63c7c20 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27b3fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117014 cookie 94411270469664 notifier 4527 bl.length()=192 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27b3fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27b3fff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f27b3fff640 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f2710003410 con 0x55ddd63c7c20 2024-04-27T03:18:12.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [notify cookie 94411270655920 out=8b] v0'0 uv129 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.789+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.2 [notify-ack] v0'0 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.793+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94411270655920 notify 115964117014 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55ddd63b8a90 con 0x55ddd63c7c20 2024-04-27T03:18:12.809 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"testx$tenanteduser","display_name":"tenanted-user","email":"","suspended":0,"max_buckets":1,"subusers":[],"keys":[{"user":"testx$tenanteduser","access_key":"p5YnriCv1nAtykxBrupQ","secret_key":"Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh","active":true,"create_date":"2024-04-27T03:18:12.789455Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:12.789444Z","tags":[],"group_ids":[]} 2024-04-27T03:18:12.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.793+0000 7f27937e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.793+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94411270485632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x55ddd63c3970 2024-04-27T03:18:12.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.793+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94411270485632] v27'133 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.793+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94411270462384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x55ddd63c7c20 2024-04-27T03:18:12.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.797+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94411270462384] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.797+0000 7f28df605980 20 remove_watcher() i=0 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.797+0000 7f28df605980 2 removed watcher, disabling cache 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.797+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94411270467392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x55ddd63c3970 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.797+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94411270467392] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x55ddd63c3970 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.797+0000 7f28df605980 20 remove_watcher() i=1 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.797+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94411270469664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x55ddd63c7c20 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94411270469664] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28df605980 20 remove_watcher() i=2 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94411270473552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x7f28d8034c20 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94411270473552] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28df605980 20 remove_watcher() i=3 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94411270474608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x55ddd63c7c20 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28dd921640 1 -- 172.21.15.110:0/498406747 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94411270474608] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d8031020 con 0x55ddd63c7c20 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28df605980 20 remove_watcher() i=4 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.801+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94411270478384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x55ddd63ccd80 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94411270478384] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 20 remove_watcher() i=5 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94411270482080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x55ddd63ccd80 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28dd120640 1 -- 172.21.15.110:0/498406747 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94411270482080] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28c802e070 con 0x55ddd63ccd80 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 20 remove_watcher() i=6 2024-04-27T03:18:12.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94411270484208] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ddd63fc300 con 0x7f28d8034c20 2024-04-27T03:18:12.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94411270484208] v27'131 uv129 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d00aaad0 con 0x7f28d8034c20 2024-04-27T03:18:12.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 20 remove_watcher() i=7 2024-04-27T03:18:12.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd63c7c20 msgr2=0x55ddd63ca050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd63c7c20 0x55ddd63ca050 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd63c3970 msgr2=0x55ddd63c5e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd63c3970 0x55ddd63c5e10 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28d8034c20 msgr2=0x7f28d804aa30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28d8034c20 0x7f28d804aa30 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ddd63ccd80 msgr2=0x55ddd63cf170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ddd63ccd80 0x55ddd63cf170 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27a8034650 msgr2=0x7f27a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27a8034650 0x7f27a8036b10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f28c8044010 tx=0x7f28c8033040 comp rx=0 tx=0).stop 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 msgr2=0x55ddd63b82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28dc91f640 1 -- 172.21.15.110:0/498406747 reap_dead start 2024-04-27T03:18:12.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd63b82c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f28d80168b0 tx=0x7f28d80498d0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 shutdown_connections 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/498406747 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd637f7f0 0x55ddd63b82c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 >> 172.21.15.110:0/498406747 conn(0x55ddd63aaf30 msgr2=0x55ddd63a8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 shutdown_connections 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/498406747 wait complete. 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27b4034650 msgr2=0x7f27b4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27b4034650 0x7f27b4036b10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55ddd6391e30 tx=0x7f28c8028f80 comp rx=0 tx=0).stop 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd63a1a20 msgr2=0x55ddd63a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd63a1a20 0x55ddd63a1e40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f28d800aab0 tx=0x7f28d800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 shutdown_connections 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27b4034650 0x7f27b4036b10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/2686991267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd63a1a20 0x55ddd63a1e40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 >> 172.21.15.110:0/2686991267 conn(0x55ddd6396830 msgr2=0x55ddd6397340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 shutdown_connections 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/2686991267 wait complete. 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd638ee00 msgr2=0x55ddd63911f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd638ee00 0x55ddd63911f0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd638b600 msgr2=0x55ddd638d9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd638b600 0x55ddd638d9f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c4034650 msgr2=0x7f27c4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c4034650 0x7f27c4036b10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f28d00ab000 tx=0x7f28d00a5000 comp rx=0 tx=0).stop 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 msgr2=0x55ddd63866e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd63866e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f28c8004000 tx=0x7f28c8000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 shutdown_connections 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ddd638ee00 0x55ddd63911f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ddd638b600 0x55ddd638d9f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c4034650 0x7f27c4036b10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 --2- 172.21.15.110:0/1962013720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ddd6297650 0x55ddd63866e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 >> 172.21.15.110:0/1962013720 conn(0x55ddd628e9c0 msgr2=0x55ddd6378a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 shutdown_connections 2024-04-27T03:18:12.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.805+0000 7f28df605980 1 -- 172.21.15.110:0/1962013720 wait complete. 2024-04-27T03:18:12.824 DEBUG:tasks.util.rgw: json result: {'user_id': 'testx$tenanteduser', 'display_name': 'tenanted-user', 'email': '', 'suspended': 0, 'max_buckets': 1, 'subusers': [], 'keys': [{'user': 'testx$tenanteduser', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True, 'create_date': '2024-04-27T03:18:12.789455Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:12.789444Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:12.824 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--bucket', '/myfoo', '--tenant', 'testx', '--uid', 'tenanteduser'] 2024-04-27T03:18:12.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--bucket', '/myfoo', '--tenant', 'testx', '--uid', 'tenanteduser'] 2024-04-27T03:18:12.824 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --bucket /myfoo --tenant testx --uid tenanteduser 2024-04-27T03:18:12.856 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:12.856 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:12.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f68223c6980 1 Processor -- start 2024-04-27T03:18:12.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f68223c6980 1 -- start start 2024-04-27T03:18:12.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f68223c6980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55db7abac230 0x55db7abac610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f68223c6980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55db7aa6b0d0 con 0x55db7abacb50 2024-04-27T03:18:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f68223c6980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55db7aa60430 con 0x55db7abac230 2024-04-27T03:18:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f681fee1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55db7abac230 0x55db7abac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f681f6e0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55db7abacb50 0x55db7abad510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34914/0 (socket says 172.21.15.110:34914) 2024-04-27T03:18:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f681f6e0640 1 -- 172.21.15.110:0/4275780964 learned_addr learned my addr 172.21.15.110:0/4275780964 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:12.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f681fee1640 1 -- 172.21.15.110:0/4275780964 >> v1:172.21.15.77:6789/0 conn(0x55db7abacb50 legacy=0x55db7abad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.849+0000 7f681fee1640 1 -- 172.21.15.110:0/4275780964 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7aa59510 con 0x55db7abac230 2024-04-27T03:18:12.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f681fee1640 1 --2- 172.21.15.110:0/4275780964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7abac610 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f681000a480 tx=0x7f681001b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e816083e5c9c2b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f681eedf640 1 -- 172.21.15.110:0/4275780964 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68100b0070 con 0x55db7abac230 2024-04-27T03:18:12.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f681eedf640 1 -- 172.21.15.110:0/4275780964 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68100ab540 con 0x55db7abac230 2024-04-27T03:18:12.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f681eedf640 1 -- 172.21.15.110:0/4275780964 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68100ab840 con 0x55db7abac230 2024-04-27T03:18:12.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f68223c6980 1 -- 172.21.15.110:0/4275780964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 msgr2=0x55db7abac610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f68223c6980 1 --2- 172.21.15.110:0/4275780964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7abac610 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f681000a480 tx=0x7f681001b280 comp rx=0 tx=0).stop 2024-04-27T03:18:12.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f68223c6980 1 -- 172.21.15.110:0/4275780964 shutdown_connections 2024-04-27T03:18:12.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f68223c6980 1 --2- 172.21.15.110:0/4275780964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7abac610 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f68223c6980 1 -- 172.21.15.110:0/4275780964 >> 172.21.15.110:0/4275780964 conn(0x55db7aba7b00 msgr2=0x55db7aba9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f68223c6980 1 -- 172.21.15.110:0/4275780964 shutdown_connections 2024-04-27T03:18:12.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.853+0000 7f68223c6980 1 -- 172.21.15.110:0/4275780964 wait complete. 2024-04-27T03:18:12.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.865+0000 7f68223c6980 1 Processor -- start 2024-04-27T03:18:12.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.865+0000 7f68223c6980 1 -- start start 2024-04-27T03:18:12.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.865+0000 7f68223c6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac97b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.865+0000 7f68223c6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db7aaaf2d0 con 0x55db7abac230 2024-04-27T03:18:12.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.865+0000 7f681fee1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac97b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.865+0000 7f681fee1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac97b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52286/0 (socket says 172.21.15.110:52286) 2024-04-27T03:18:12.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.865+0000 7f681fee1640 1 -- 172.21.15.110:0/3493851159 learned_addr learned my addr 172.21.15.110:0/3493851159 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f681fee1640 1 -- 172.21.15.110:0/3493851159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7aa35ae0 con 0x55db7abac230 2024-04-27T03:18:12.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f681fee1640 1 --2- 172.21.15.110:0/3493851159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac97b10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6810017e90 tx=0x7f6810003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c507c67beff2f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f670de0b640 1 -- 172.21.15.110:0/3493851159 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68100b0070 con 0x55db7abac230 2024-04-27T03:18:12.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f670de0b640 1 -- 172.21.15.110:0/3493851159 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68100b1040 con 0x55db7abac230 2024-04-27T03:18:12.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f670de0b640 1 -- 172.21.15.110:0/3493851159 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68100abc30 con 0x55db7abac230 2024-04-27T03:18:12.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- 172.21.15.110:0/3493851159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 msgr2=0x55db7ac97b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 --2- 172.21.15.110:0/3493851159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac97b10 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6810017e90 tx=0x7f6810003030 comp rx=0 tx=0).stop 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- 172.21.15.110:0/3493851159 shutdown_connections 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 --2- 172.21.15.110:0/3493851159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac97b10 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- 172.21.15.110:0/3493851159 >> 172.21.15.110:0/3493851159 conn(0x55db7aba7b00 msgr2=0x55db7ac936a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- 172.21.15.110:0/3493851159 shutdown_connections 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- 172.21.15.110:0/3493851159 wait complete. 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 Processor -- start 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- start start 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac9ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db7abb6280 con 0x55db7abac230 2024-04-27T03:18:12.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f681fee1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac9ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f681fee1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac9ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52292/0 (socket says 172.21.15.110:52292) 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f681fee1640 1 -- 172.21.15.110:0/828986076 learned_addr learned my addr 172.21.15.110:0/828986076 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f681fee1640 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7ac9efd0 con 0x55db7abac230 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f681fee1640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac9ea90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6810017e90 tx=0x7f6810018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f6707fff640 1 -- 172.21.15.110:0/828986076 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68100c6070 con 0x55db7abac230 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55db7ac9f260 con 0x55db7abac230 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f6707fff640 1 -- 172.21.15.110:0/828986076 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68100b78c0 con 0x55db7abac230 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f6707fff640 1 -- 172.21.15.110:0/828986076 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68100d0060 con 0x55db7abac230 2024-04-27T03:18:12.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.869+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db7aca1e90 con 0x55db7abac230 2024-04-27T03:18:12.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f6707fff640 1 -- 172.21.15.110:0/828986076 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f68100c9070 con 0x55db7abac230 2024-04-27T03:18:12.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f6707fff640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f67080346a0 0x7f6708036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f6707fff640 1 -- 172.21.15.110:0/828986076 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f68100bc070 con 0x55db7abac230 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f681f6e0640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f67080346a0 0x7f6708036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7abacb50 0x55db7aca5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7aca5f40 con 0x55db7abacb50 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68206e2640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7abacb50 0x55db7aca5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f681f6e0640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f67080346a0 0x7f6708036b60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6814000f70 tx=0x7f68140012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68206e2640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7abacb50 0x55db7aca5970 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68206e2640 1 -- 172.21.15.110:0/828986076 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f681800a4d0 con 0x55db7abacb50 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7aca6d10 0x55db7aca9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7aca9740 con 0x55db7aca6d10 2024-04-27T03:18:12.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f681fee1640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7aca6d10 0x55db7aca9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f681fee1640 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7aca6d10 0x55db7aca9100 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f681fee1640 1 -- 172.21.15.110:0/828986076 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f68100c8030 con 0x55db7aca6d10 2024-04-27T03:18:12.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.873+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7acaaff0 con 0x55db7abacb50 2024-04-27T03:18:12.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68206e2640 1 -- 172.21.15.110:0/828986076 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f681800a4d0 con 0x55db7abacb50 2024-04-27T03:18:12.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7acaa220 con 0x55db7aca6d10 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f681fee1640 1 -- 172.21.15.110:0/828986076 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f68100c8030 con 0x55db7aca6d10 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7acaa640 con 0x55db7abacb50 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68206e2640 1 -- 172.21.15.110:0/828986076 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f681800a4d0 con 0x55db7abacb50 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 Processor -- start 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 -- start start 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acb9900 0x55db7acb9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db7aab3310 con 0x55db7acb9900 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68206e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acb9900 0x55db7acb9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68206e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acb9900 0x55db7acb9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52306/0 (socket says 172.21.15.110:52306) 2024-04-27T03:18:12.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68206e2640 1 -- 172.21.15.110:0/788279256 learned_addr learned my addr 172.21.15.110:0/788279256 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68206e2640 1 -- 172.21.15.110:0/788279256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7acba260 con 0x55db7acb9900 2024-04-27T03:18:12.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68206e2640 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acb9900 0x55db7acb9d20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f681800dfc0 tx=0x7f68180178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f67057fa640 1 -- 172.21.15.110:0/788279256 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68180157d0 con 0x55db7acb9900 2024-04-27T03:18:12.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f67057fa640 1 -- 172.21.15.110:0/788279256 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818015970 con 0x55db7acb9900 2024-04-27T03:18:12.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55db7acbad00 con 0x55db7acb9900 2024-04-27T03:18:12.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f67057fa640 1 -- 172.21.15.110:0/788279256 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6818015c70 con 0x55db7acb9900 2024-04-27T03:18:12.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.877+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db7acbb1d0 con 0x55db7acb9900 2024-04-27T03:18:12.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f67057fa640 1 -- 172.21.15.110:0/788279256 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6818029440 con 0x55db7acb9900 2024-04-27T03:18:12.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f67057fa640 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66f8034650 0x7f66f8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f67057fa640 1 -- 172.21.15.110:0/788279256 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f681800e040 con 0x55db7acb9900 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f681fee1640 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66f8034650 0x7f66f8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 Processor -- start 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- start start 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f681fee1640 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66f8034650 0x7f66f8036b10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55db7aca64d0 tx=0x7f68100d4ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7acc6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db7ac981a0 con 0x55db7acc6ab0 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7acc6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7acc6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52310/0 (socket says 172.21.15.110:52310) 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 -- 172.21.15.110:0/1499653192 learned_addr learned my addr 172.21.15.110:0/1499653192 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 -- 172.21.15.110:0/1499653192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7acc74a0 con 0x55db7acc6ab0 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 --2- 172.21.15.110:0/1499653192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7acc6ed0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6818015f40 tx=0x7f681804aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e0d64d22ab141d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f66feffd640 1 -- 172.21.15.110:0/1499653192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68180283a0 con 0x55db7acc6ab0 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f66feffd640 1 -- 172.21.15.110:0/1499653192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818028540 con 0x55db7acc6ab0 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f66feffd640 1 -- 172.21.15.110:0/1499653192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f681804c110 con 0x55db7acc6ab0 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- 172.21.15.110:0/1499653192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 msgr2=0x55db7acc6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 --2- 172.21.15.110:0/1499653192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7acc6ed0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f6818015f40 tx=0x7f681804aa00 comp rx=0 tx=0).stop 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- 172.21.15.110:0/1499653192 shutdown_connections 2024-04-27T03:18:12.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 --2- 172.21.15.110:0/1499653192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7acc6ed0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- 172.21.15.110:0/1499653192 >> 172.21.15.110:0/1499653192 conn(0x55db7acc2e40 msgr2=0x55db7acc0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- 172.21.15.110:0/1499653192 shutdown_connections 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- 172.21.15.110:0/1499653192 wait complete. 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 Processor -- start 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- start start 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7accdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68223c6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db7acc7c10 con 0x55db7acc6ab0 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7accdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7accdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52320/0 (socket says 172.21.15.110:52320) 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.881+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 learned_addr learned my addr 172.21.15.110:0/118442442 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:12.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7acce520 con 0x55db7acc6ab0 2024-04-27T03:18:12.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68206e2640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7accdfe0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6818015f40 tx=0x7f681804a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f66fd7fa640 1 -- 172.21.15.110:0/118442442 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6818036070 con 0x55db7acc6ab0 2024-04-27T03:18:12.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55db7acce7b0 con 0x55db7acc6ab0 2024-04-27T03:18:12.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f66fd7fa640 1 -- 172.21.15.110:0/118442442 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818049460 con 0x55db7acc6ab0 2024-04-27T03:18:12.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f66fd7fa640 1 -- 172.21.15.110:0/118442442 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6818041040 con 0x55db7acc6ab0 2024-04-27T03:18:12.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db7acd1490 con 0x55db7acc6ab0 2024-04-27T03:18:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f66fd7fa640 1 -- 172.21.15.110:0/118442442 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6818039070 con 0x55db7acc6ab0 2024-04-27T03:18:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f66fd7fa640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66e8034650 0x7f66e8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f66fd7fa640 1 -- 172.21.15.110:0/118442442 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f6818045280 con 0x55db7acc6ab0 2024-04-27T03:18:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f681fee1640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66e8034650 0x7f66e8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7acd9600 0x55db7acdba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7acdc1e0 con 0x55db7acd9600 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f681f6e0640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7acd9600 0x55db7acdba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f681fee1640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66e8034650 0x7f66e8036b10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f68100d4fb0 tx=0x7f68100e9000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f681f6e0640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7acd9600 0x55db7acdba90 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f681400e5a0 con 0x55db7acd9600 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 realm 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7acd8b10 con 0x55db7acd9600 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f681400e5a0 con 0x55db7acd9600 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:12.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7acdcce0 con 0x55db7acd9600 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f681400e5a0 con 0x55db7acd9600 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7acddac0 0x55db7acdfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.885+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace0560 con 0x55db7acddac0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68206e2640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7acddac0 0x55db7acdfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68206e2640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7acddac0 0x55db7acdfeb0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68223c6980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace0db0 con 0x55db7acd9600 2024-04-27T03:18:12.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f681400e5a0 con 0x55db7acd9600 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68223c6980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68223c6980 20 searching for the correct realm 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db7ace2b20 0x55db7ace4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7ace2b20 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f681fee1640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db7ace2b20 0x55db7ace4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f681fee1640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db7ace2b20 0x55db7ace4f10 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acddac0 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68206e2640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6818014030 0x7f681804c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:12.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x7f6818014030 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f681f6e0640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6818014030 0x7f681804c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.889+0000 7f681f6e0640 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6818014030 0x7f681804c690 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f681400e5a0 con 0x7f6818014030 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acd9600 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f681400e5a0 con 0x55db7acd9600 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acd9600 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7ace2b20 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acd9600 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acddac0 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acddac0 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acd9600 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x7f6818014030 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x7f6818014030 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acd9600 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x7f6818014030 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f6814029040 con 0x7f6818014030 2024-04-27T03:18:12.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.893+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace5590 con 0x55db7acddac0 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7ace2b20 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acd9600 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acddac0 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7ace2b20 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acd9600 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acd9600 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acd9600 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7ace2b20 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x7f6818014030 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x7f6818014030 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acd9600 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x7f6818014030 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x7f6818014030 2024-04-27T03:18:12.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7ace2b20 2024-04-27T03:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x7f6818014030 2024-04-27T03:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6814029040 con 0x7f6818014030 2024-04-27T03:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.897+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acddac0 2024-04-27T03:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acddac0 2024-04-27T03:18:12.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7ace2b20 2024-04-27T03:18:12.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acddac0 2024-04-27T03:18:12.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f6818063040 con 0x55db7acddac0 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace1760 con 0x55db7acd9600 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f6814029040 con 0x55db7acd9600 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace13a0 con 0x55db7acddac0 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace20c0 con 0x55db7acd9600 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:12.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 zone default found 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 4 Realm: () 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 zonegroup default 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace1c50 con 0x7f6818014030 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.901+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace7900 con 0x55db7ace2b20 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace7ca0 con 0x55db7ace2b20 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 20 started sync module instance, tier type = 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace6da0 con 0x55db7acddac0 2024-04-27T03:18:12.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94401146548928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace6da0 con 0x55db7acddac0 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace8750 con 0x55db7acd9600 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94401146552896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace8710 con 0x55db7acd9600 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.905+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace9660 con 0x55db7acddac0 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94401146548928] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94401146552896] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94401146555200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace9660 con 0x55db7acddac0 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace9ac0 con 0x7f6818014030 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94401146546416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ace9a80 con 0x7f6818014030 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7acea980 con 0x55db7acddac0 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94401146546416] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.909+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94401146555200] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94401146556816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7acea4d0 con 0x55db7acddac0 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7aceb220 con 0x55db7ace2b20 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94401146563952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7acea4d0 con 0x55db7ace2b20 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7acebf80 con 0x55db7ace2b20 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94401146556816] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94401146563952] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94401146567280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7acec370 con 0x55db7ace2b20 2024-04-27T03:18:12.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7aceb600 con 0x7f6818014030 2024-04-27T03:18:12.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.913+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94401146567280] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'132 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94401146560096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7aceb600 con 0x7f6818014030 2024-04-27T03:18:12.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=0 2024-04-27T03:18:12.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=1 2024-04-27T03:18:12.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=2 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=3 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=4 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=5 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=6 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94401146560096] v27'133 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 add_watcher() i=7 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7acefd90 con 0x55db7acd9600 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94401146570928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7aced220 con 0x55db7acd9600 2024-04-27T03:18:12.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94401146570928] v27'134 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f66d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.917+0000 7f68223c6980 5 note: GC not initialized 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f66d57e2640 20 reqs_thread_entry: start 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 Filter name: none 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f66ce7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : miss 2024-04-27T03:18:12.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:62 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad136f0 con 0x55db7ace2b20 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(62 testx$tenanteduser [call out=48b,read 0~419 out=419b,getxattrs out=75b] v0'0 uv7 ondisk = 0) ==== 246+0+542 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 rados_obj.operate() r=0 bl.length=419 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x13 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 adding default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad13140 con 0x7f6818014030 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv23 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:12.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 rados->read ofs=0 len=0 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad16ec0 con 0x7f6818014030 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(64 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv5 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:12.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : hit (requested=0x13, cached=0x13) 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1b670 con 0x7f6818014030 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(65 foo.buckets [call] v27'24 uv24 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.921+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 8.3 8:c2eafa05:root::.bucket.meta.testx%3amyfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [create,call version.inc in=38b,writefull 0~412 in=412b,setxattr user.rgw.acl (203) in=215b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ef80 con 0x55db7acd9600 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(66 .bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [create,call,writefull 0~412,setxattr (203)] v27'2 uv2 ondisk = 0) ==== 338+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 10 cache put: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x7 2024-04-27T03:18:12.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 10 adding default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 10 updating xattr: name=user.rgw.acl bl.length()=203 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [notify cookie 94401146768816 in=859b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1e9f0 con 0x55db7acd9600 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== watch-notify(notify (1) cookie 94401146552896 notify 115964116992 ret 0) ==== 889+0+0 (crc 0 0 0) 0x55db7ad1f320 con 0x55db7acd9600 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f66feffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116992 cookie 94401146552896 notifier 4534 bl.length()=847 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f66feffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x7 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f66feffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f66feffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=203 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f66feffd640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f66e4003eb0 con 0x55db7acd9600 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(67 notify.1 [notify cookie 94401146768816 out=8b] v0'0 uv132 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(68 notify.1 [notify-ack] v0'0 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== watch-notify(notify_complete (2) cookie 94401146768816 notify 115964116992 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55db7ad1b4c0 con 0x55db7acd9600 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 20 handle_bi_update(): bucket=testx:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_sources= new_sources= 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 20 handle_bi_update(): bucket=testx:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential sources added=[] removed=[] 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 20 handle_bi_update(): bucket=testx:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_dests= new_dests= 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 20 handle_bi_update(): bucket=testx:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential dests added=[] removed=[] 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ead0 con 0x55db7acd9600 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(69 testx$tenanteduser.buckets [call] v27'1 uv1 ondisk = 0) ==== 170+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.925+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [delete,create,call version.inc in=38b,writefull 0~193 in=193b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ead0 con 0x55db7acd9600 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(70 testx/myfoo [delete,create,call,writefull 0~193] v27'3 uv3 ondisk = 0) ==== 281+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 10 cache put: name=default.rgw.meta+root+testx/myfoo info.flags=0x7 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 10 adding default.rgw.meta+root+testx/myfoo to cache LRU end 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:root:testx/myfoo, ofs0, ns] 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [notify cookie 94401146775072 in=360b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ea40 con 0x55db7acddac0 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94401146556816 notify 115964117015 ret 0) ==== 390+0+0 (crc 0 0 0) 0x55db7acd1490 con 0x55db7acddac0 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f66fe7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117015 cookie 94401146556816 notifier 4534 bl.length()=348 2024-04-27T03:18:12.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f66fe7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+testx/myfoo info.flags=0x7 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f66fe7fc640 10 rgw watcher librados: moving default.rgw.meta+root+testx/myfoo to cache LRU end 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f66fe7fc640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f6648003410 con 0x55db7acddac0 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(71 notify.4 [notify cookie 94401146775072 out=8b] v0'0 uv132 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(72 notify.4 [notify-ack] v0'0 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94401146775072 notify 115964117015 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55db7acce7b0 con 0x55db7acddac0 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 8.9 8:91b71841:root::myfoo:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ead0 con 0x7f6818014030 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(73 myfoo [call,delete] v27'25 uv25 ondisk = 0) ==== 191+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 10 removing default.rgw.meta+root+myfoo from cache 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:74 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94401146775072 in=161b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ea40 con 0x55db7ace2b20 2024-04-27T03:18:12.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== watch-notify(notify (1) cookie 94401146567280 notify 115964117006 ret 0) ==== 191+0+0 (crc 0 0 0) 0x7f68100f8040 con 0x55db7ace2b20 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f66feffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117006 cookie 94401146567280 notifier 4534 bl.length()=149 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f66feffd640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f66e4003eb0 con 0x55db7ace2b20 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.929+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(74 notify.6 [notify cookie 94401146775072 out=8b] v0'0 uv132 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(75 notify.6 [notify-ack] v0'0 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== watch-notify(notify_complete (2) cookie 94401146775072 notify 115964117006 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f68100db040 con 0x55db7ace2b20 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ead0 con 0x7f6818014030 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(76 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call,delete] v27'6 uv6 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68223c6980 10 removing default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 from cache 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68223c6980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 7.15 7:a93a5511:::notify.2:head [notify cookie 94401146746176 in=218b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad1ead0 con 0x55db7acddac0 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94401146555200 notify 115964117016 ret 0) ==== 248+0+0 (crc 0 0 0) 0x55db7acce7b0 con 0x55db7acddac0 2024-04-27T03:18:12.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f66fe7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117016 cookie 94401146555200 notifier 4534 bl.length()=206 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f66fe7fc640 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f6648003410 con 0x55db7acddac0 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(77 notify.2 [notify cookie 94401146746176 out=8b] v0'0 uv132 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(78 notify.2 [notify-ack] v0'0 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== watch-notify(notify_complete (2) cookie 94401146746176 notify 115964117016 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55db7acce520 con 0x55db7acddac0 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f66d67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.933+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94401146570928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x55db7acd9600 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.937+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(79 data_loggenerations_metadata [watch unwatch cookie 94401146570928] v27'135 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.937+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94401146548928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x55db7acddac0 2024-04-27T03:18:12.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.937+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(80 notify.0 [watch unwatch cookie 94401146548928] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.937+0000 7f68223c6980 20 remove_watcher() i=0 2024-04-27T03:18:12.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.937+0000 7f68223c6980 2 removed watcher, disabling cache 2024-04-27T03:18:12.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.937+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94401146552896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x55db7acd9600 2024-04-27T03:18:12.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(81 notify.1 [watch unwatch cookie 94401146552896] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x55db7acd9600 2024-04-27T03:18:12.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f68223c6980 20 remove_watcher() i=1 2024-04-27T03:18:12.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94401146555200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x55db7acddac0 2024-04-27T03:18:12.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(82 notify.2 [watch unwatch cookie 94401146555200] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f68223c6980 20 remove_watcher() i=2 2024-04-27T03:18:12.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:83 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94401146546416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x7f6818014030 2024-04-27T03:18:12.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(83 notify.3 [watch unwatch cookie 94401146546416] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f68223c6980 20 remove_watcher() i=3 2024-04-27T03:18:12.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.941+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:84 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94401146556816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x55db7acddac0 2024-04-27T03:18:12.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f68206e2640 1 -- 172.21.15.110:0/118442442 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(84 notify.4 [watch unwatch cookie 94401146556816] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6818038030 con 0x55db7acddac0 2024-04-27T03:18:12.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f68223c6980 20 remove_watcher() i=4 2024-04-27T03:18:12.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:85 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94401146563952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x55db7ace2b20 2024-04-27T03:18:12.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(85 notify.5 [watch unwatch cookie 94401146563952] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f68223c6980 20 remove_watcher() i=5 2024-04-27T03:18:12.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94401146567280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x55db7ace2b20 2024-04-27T03:18:12.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f681fee1640 1 -- 172.21.15.110:0/118442442 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(86 notify.6 [watch unwatch cookie 94401146567280] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f68100ef040 con 0x55db7ace2b20 2024-04-27T03:18:12.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f68223c6980 20 remove_watcher() i=6 2024-04-27T03:18:12.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.945+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:87 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94401146560096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55db7ad11280 con 0x7f6818014030 2024-04-27T03:18:12.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(87 notify.7 [watch unwatch cookie 94401146560096] v27'134 uv132 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f681403c1c0 con 0x7f6818014030 2024-04-27T03:18:12.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 20 remove_watcher() i=7 2024-04-27T03:18:12.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7acddac0 msgr2=0x55db7acdfeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7acddac0 0x55db7acdfeb0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7acd9600 msgr2=0x55db7acdba90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7acd9600 0x55db7acdba90 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6818014030 msgr2=0x7f681804c690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f6818014030 0x7f681804c690 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db7ace2b20 msgr2=0x55db7ace4f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db7ace2b20 0x55db7ace4f10 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66e8034650 msgr2=0x7f66e8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66e8034650 0x7f66e8036b10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f68100d4fb0 tx=0x7f68100e9000 comp rx=0 tx=0).stop 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 msgr2=0x55db7accdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/118442442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acc6ab0 0x55db7accdfe0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6818015f40 tx=0x7f681804a850 comp rx=0 tx=0).stop 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 reap_dead start 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f681f6e0640 1 -- 172.21.15.110:0/118442442 reap_dead start 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 shutdown_connections 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 >> 172.21.15.110:0/118442442 conn(0x55db7acc2e40 msgr2=0x55db7acc8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 shutdown_connections 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/118442442 wait complete. 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66f8034650 msgr2=0x7f66f8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66f8034650 0x7f66f8036b10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55db7aca64d0 tx=0x7f68100d4ec0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acb9900 msgr2=0x55db7acb9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acb9900 0x55db7acb9d20 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f681800dfc0 tx=0x7f68180178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 shutdown_connections 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f66f8034650 0x7f66f8036b10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/788279256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7acb9900 0x55db7acb9d20 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 >> 172.21.15.110:0/788279256 conn(0x55db7acae710 msgr2=0x55db7acaf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 shutdown_connections 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/788279256 wait complete. 2024-04-27T03:18:12.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7aca6d10 msgr2=0x55db7aca9100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7aca6d10 0x55db7aca9100 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7abacb50 msgr2=0x55db7aca5970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7abacb50 0x55db7aca5970 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f67080346a0 msgr2=0x7f6708036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f67080346a0 0x7f6708036b60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6814000f70 tx=0x7f68140012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 msgr2=0x55db7ac9ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac9ea90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f6810017e90 tx=0x7f6810018020 comp rx=0 tx=0).stop 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 shutdown_connections 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db7aca6d10 0x55db7aca9100 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db7abacb50 0x55db7aca5970 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f67080346a0 0x7f6708036b60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 --2- 172.21.15.110:0/828986076 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db7abac230 0x55db7ac9ea90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:12.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 >> 172.21.15.110:0/828986076 conn(0x55db7aba7b00 msgr2=0x55db7ac921b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:12.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 shutdown_connections 2024-04-27T03:18:12.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:12.949+0000 7f68223c6980 1 -- 172.21.15.110:0/828986076 wait complete. 2024-04-27T03:18:12.966 INFO:tasks.util.rgw:rgwadmin: client.0 : ['metadata', 'get', 'bucket:testx/myfoo'] 2024-04-27T03:18:12.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'metadata', 'get', 'bucket:testx/myfoo'] 2024-04-27T03:18:12.966 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph metadata get bucket:testx/myfoo 2024-04-27T03:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:13.039 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e0bbb3980 1 Processor -- start 2024-04-27T03:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e0bbb3980 1 -- start start 2024-04-27T03:18:13.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e0bbb3980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5575aee0e2d0 0x5575aee0e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e0bbb3980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5575aeccd0d0 con 0x5575aee0ebf0 2024-04-27T03:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e0bbb3980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5575aecc2430 con 0x5575aee0e2d0 2024-04-27T03:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e096ce640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5575aee0e2d0 0x5575aee0e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e08ecd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5575aee0ebf0 0x5575aee0f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34920/0 (socket says 172.21.15.110:34920) 2024-04-27T03:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e08ecd640 1 -- 172.21.15.110:0/1507862322 learned_addr learned my addr 172.21.15.110:0/1507862322 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e096ce640 1 -- 172.21.15.110:0/1507862322 >> v1:172.21.15.77:6789/0 conn(0x5575aee0ebf0 legacy=0x5575aee0f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.033+0000 7f9e096ce640 1 -- 172.21.15.110:0/1507862322 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575aecbb510 con 0x5575aee0e2d0 2024-04-27T03:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e096ce640 1 --2- 172.21.15.110:0/1507862322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aee0e6b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00a480 tx=0x7f9dfc01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da18f6a9517379b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e03fff640 1 -- 172.21.15.110:0/1507862322 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc0b0070 con 0x5575aee0e2d0 2024-04-27T03:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e03fff640 1 -- 172.21.15.110:0/1507862322 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0ab540 con 0x5575aee0e2d0 2024-04-27T03:18:13.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e03fff640 1 -- 172.21.15.110:0/1507862322 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc0ab840 con 0x5575aee0e2d0 2024-04-27T03:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1507862322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 msgr2=0x5575aee0e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1507862322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aee0e6b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00a480 tx=0x7f9dfc01b280 comp rx=0 tx=0).stop 2024-04-27T03:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1507862322 shutdown_connections 2024-04-27T03:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1507862322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aee0e6b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1507862322 >> 172.21.15.110:0/1507862322 conn(0x5575aee09ba0 msgr2=0x5575aee0bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1507862322 shutdown_connections 2024-04-27T03:18:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.037+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1507862322 wait complete. 2024-04-27T03:18:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.049+0000 7f9e0bbb3980 1 Processor -- start 2024-04-27T03:18:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.049+0000 7f9e0bbb3980 1 -- start start 2024-04-27T03:18:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.049+0000 7f9e0bbb3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aeef9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.049+0000 7f9e0bbb3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575aed112d0 con 0x5575aee0e2d0 2024-04-27T03:18:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aeef9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aeef9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52336/0 (socket says 172.21.15.110:52336) 2024-04-27T03:18:13.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 -- 172.21.15.110:0/2328962594 learned_addr learned my addr 172.21.15.110:0/2328962594 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 -- 172.21.15.110:0/2328962594 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575aec97ae0 con 0x5575aee0e2d0 2024-04-27T03:18:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 --2- 172.21.15.110:0/2328962594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aeef9b90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0ae040 tx=0x7f9dfc003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62986b0dfed8c96d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9cf7606640 1 -- 172.21.15.110:0/2328962594 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc0b0040 con 0x5575aee0e2d0 2024-04-27T03:18:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9cf7606640 1 -- 172.21.15.110:0/2328962594 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0ab9d0 con 0x5575aee0e2d0 2024-04-27T03:18:13.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9cf7606640 1 -- 172.21.15.110:0/2328962594 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc0abcf0 con 0x5575aee0e2d0 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2328962594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 msgr2=0x5575aeef9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/2328962594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aeef9b90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc0ae040 tx=0x7f9dfc003030 comp rx=0 tx=0).stop 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2328962594 shutdown_connections 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/2328962594 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aeef9b90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2328962594 >> 172.21.15.110:0/2328962594 conn(0x5575aee09ba0 msgr2=0x5575aeef5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2328962594 shutdown_connections 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2328962594 wait complete. 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 Processor -- start 2024-04-27T03:18:13.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 -- start start 2024-04-27T03:18:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aef00b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e0bbb3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575aee182d0 con 0x5575aee0e2d0 2024-04-27T03:18:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aef00b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aef00b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52344/0 (socket says 172.21.15.110:52344) 2024-04-27T03:18:13.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 -- 172.21.15.110:0/1997832178 learned_addr learned my addr 172.21.15.110:0/1997832178 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575aef01050 con 0x5575aee0e2d0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.053+0000 7f9e096ce640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aef00b10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc018eb0 tx=0x7f9dfc018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9cf5e03640 1 -- 172.21.15.110:0/1997832178 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc0b0070 con 0x5575aee0e2d0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9cf5e03640 1 -- 172.21.15.110:0/1997832178 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0b68f0 con 0x5575aee0e2d0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5575aef012e0 con 0x5575aee0e2d0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9cf5e03640 1 -- 172.21.15.110:0/1997832178 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9dfc0b6bd0 con 0x5575aee0e2d0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575aef03ec0 con 0x5575aee0e2d0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9cf5e03640 1 -- 172.21.15.110:0/1997832178 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9dfc0b7420 con 0x5575aee0e2d0 2024-04-27T03:18:13.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9cf5e03640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cf00346a0 0x7f9cf0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9cf5e03640 1 -- 172.21.15.110:0/1997832178 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9dfc0bc070 con 0x5575aee0e2d0 2024-04-27T03:18:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e08ecd640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cf00346a0 0x7f9cf0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aee0ebf0 0x5575aef079a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef07fc0 con 0x5575aee0ebf0 2024-04-27T03:18:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aee0ebf0 0x5575aef079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e08ecd640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cf00346a0 0x7f9cf0036b60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9df4000f70 tx=0x7f9df40012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aee0ebf0 0x5575aef079a0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e09ecf640 1 -- 172.21.15.110:0/1997832178 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9e0400a4d0 con 0x5575aee0ebf0 2024-04-27T03:18:13.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef08de0 0x5575aef0b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef0b810 con 0x5575aef08de0 2024-04-27T03:18:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e096ce640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef08de0 0x5575aef0b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.057+0000 7f9e096ce640 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef08de0 0x5575aef0b1d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e096ce640 1 -- 172.21.15.110:0/1997832178 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef08de0 2024-04-27T03:18:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef0d0c0 con 0x5575aee0ebf0 2024-04-27T03:18:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e09ecf640 1 -- 172.21.15.110:0/1997832178 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f9e0400a4d0 con 0x5575aee0ebf0 2024-04-27T03:18:13.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef0c2f0 con 0x5575aef08de0 2024-04-27T03:18:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e096ce640 1 -- 172.21.15.110:0/1997832178 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef08de0 2024-04-27T03:18:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef0c6e0 con 0x5575aee0ebf0 2024-04-27T03:18:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e09ecf640 1 -- 172.21.15.110:0/1997832178 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f9e0400a4d0 con 0x5575aee0ebf0 2024-04-27T03:18:13.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 Processor -- start 2024-04-27T03:18:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 -- start start 2024-04-27T03:18:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aef1b9e0 0x5575aef1be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575aed15310 con 0x5575aef1b9e0 2024-04-27T03:18:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e09ecf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aef1b9e0 0x5575aef1be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e09ecf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aef1b9e0 0x5575aef1be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52348/0 (socket says 172.21.15.110:52348) 2024-04-27T03:18:13.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e09ecf640 1 -- 172.21.15.110:0/2229269127 learned_addr learned my addr 172.21.15.110:0/2229269127 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e09ecf640 1 -- 172.21.15.110:0/2229269127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575aef1c340 con 0x5575aef1b9e0 2024-04-27T03:18:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aef1b9e0 0x5575aef1be00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9e040056d0 tx=0x7f9e04017820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9ceeffd640 1 -- 172.21.15.110:0/2229269127 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e04012840 con 0x5575aef1b9e0 2024-04-27T03:18:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9ceeffd640 1 -- 172.21.15.110:0/2229269127 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e040129e0 con 0x5575aef1b9e0 2024-04-27T03:18:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5575aef1cde0 con 0x5575aef1b9e0 2024-04-27T03:18:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9ceeffd640 1 -- 172.21.15.110:0/2229269127 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e04012ce0 con 0x5575aef1b9e0 2024-04-27T03:18:13.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.061+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575aef1d2b0 con 0x5575aef1b9e0 2024-04-27T03:18:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9ceeffd640 1 -- 172.21.15.110:0/2229269127 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9e04020440 con 0x5575aef1b9e0 2024-04-27T03:18:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9ceeffd640 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9ce00346a0 0x7f9ce0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9ceeffd640 1 -- 172.21.15.110:0/2229269127 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9e0400e040 con 0x5575aef1b9e0 2024-04-27T03:18:13.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e096ce640 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9ce00346a0 0x7f9ce0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e0bbb3980 1 Processor -- start 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e0bbb3980 1 -- start start 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e096ce640 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9ce00346a0 0x7f9ce0036b60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x5575aef0be10 tx=0x7f9dfc0e2bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e0bbb3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aed84090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e0bbb3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575aeefa220 con 0x5575aed83c70 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e09ecf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aed84090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e09ecf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aed84090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52362/0 (socket says 172.21.15.110:52362) 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e09ecf640 1 -- 172.21.15.110:0/3601412093 learned_addr learned my addr 172.21.15.110:0/3601412093 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e09ecf640 1 -- 172.21.15.110:0/3601412093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575aed824c0 con 0x5575aed83c70 2024-04-27T03:18:13.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/3601412093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aed84090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e0404a600 tx=0x7f9e04014610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d203630c70ad6f06 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9cecff9640 1 -- 172.21.15.110:0/3601412093 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e0401f3a0 con 0x5575aed83c70 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9cecff9640 1 -- 172.21.15.110:0/3601412093 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0401f540 con 0x5575aed83c70 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.065+0000 7f9cecff9640 1 -- 172.21.15.110:0/3601412093 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e04020860 con 0x5575aed83c70 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/3601412093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 msgr2=0x5575aed84090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/3601412093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aed84090 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9e0404a600 tx=0x7f9e04014610 comp rx=0 tx=0).stop 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/3601412093 shutdown_connections 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/3601412093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aed84090 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/3601412093 >> 172.21.15.110:0/3601412093 conn(0x5575aef24ed0 msgr2=0x5575aef22f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/3601412093 shutdown_connections 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/3601412093 wait complete. 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 Processor -- start 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- start start 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aef32240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5575aed82990 con 0x5575aed83c70 2024-04-27T03:18:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e09ecf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aef32240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e09ecf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aef32240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52378/0 (socket says 172.21.15.110:52378) 2024-04-27T03:18:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 learned_addr learned my addr 172.21.15.110:0/884970967 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575aef32780 con 0x5575aed83c70 2024-04-27T03:18:13.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aef32240 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9e04020ec0 tx=0x7f9e0404a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9cdeffd640 1 -- 172.21.15.110:0/884970967 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e04013c60 con 0x5575aed83c70 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5575aef32a10 con 0x5575aed83c70 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9cdeffd640 1 -- 172.21.15.110:0/884970967 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e04013e00 con 0x5575aed83c70 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9cdeffd640 1 -- 172.21.15.110:0/884970967 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e040134e0 con 0x5575aed83c70 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575aef356f0 con 0x5575aed83c70 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9cdeffd640 1 -- 172.21.15.110:0/884970967 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9e0401f820 con 0x5575aed83c70 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9cdeffd640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cd4034650 0x7f9cd4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9cdeffd640 1 -- 172.21.15.110:0/884970967 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f9e04033c80 con 0x5575aed83c70 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e096ce640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cd4034650 0x7f9cd4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aef3d8c0 0x5575aef3fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef404a0 con 0x5575aef3d8c0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.069+0000 7f9e08ecd640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aef3d8c0 0x5575aef3fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e096ce640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cd4034650 0x7f9cd4036b10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc01b0b0 tx=0x7f9dfc0de000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e08ecd640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aef3d8c0 0x5575aef3fd50 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9df400e5a0 con 0x5575aef3d8c0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 realm 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef3cdd0 con 0x5575aef3d8c0 2024-04-27T03:18:13.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9df400e5a0 con 0x5575aef3d8c0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef40fa0 con 0x5575aef3d8c0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9df400e5a0 con 0x5575aef3d8c0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef41d80 0x5575aef44170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef447f0 con 0x5575aef41d80 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef41d80 0x5575aef44170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef41d80 0x5575aef44170 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef45090 con 0x5575aef3d8c0 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f9df400e5a0 con 0x5575aef3d8c0 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 20 searching for the correct realm 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5575aef46e00 0x5575aef491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef46e00 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e096ce640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5575aef46e00 0x5575aef491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.073+0000 7f9e096ce640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5575aef46e00 0x5575aef491f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e09ecf640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9e04014030 0x7f9e04031770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x7f9e04014030 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9e04014030 0x7f9e04031770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9e04014030 0x7f9e04031770 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df400e5a0 con 0x7f9e04014030 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df400e5a0 con 0x5575aef3d8c0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef46e00 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.077+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x7f9e04014030 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x7f9e04014030 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x7f9e04014030 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f9df4029040 con 0x7f9e04014030 2024-04-27T03:18:13.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef46e00 2024-04-27T03:18:13.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef46e00 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef46e00 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.081+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x7f9e04014030 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x7f9e04014030 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef3d8c0 2024-04-27T03:18:13.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x7f9e04014030 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x7f9e04014030 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef46e00 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x7f9e04014030 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9df4029040 con 0x7f9e04014030 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef46e00 2024-04-27T03:18:13.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.085+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5575aef497d0 con 0x5575aef41d80 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:13.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef45a40 con 0x5575aef3d8c0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f9df4029040 con 0x5575aef3d8c0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef45ec0 con 0x5575aef41d80 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef46320 con 0x5575aef3d8c0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 zone default found 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 4 Realm: () 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 zonegroup default 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef49c50 con 0x7f9e04014030 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x7f9e04014030 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef45680 con 0x5575aef46e00 2024-04-27T03:18:13.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4bbd0 con 0x5575aef46e00 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 started sync module instance, tier type = 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.089+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4af70 con 0x5575aef41d80 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93963934806880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4abd0 con 0x5575aef41d80 2024-04-27T03:18:13.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4cb00 con 0x5575aef3d8c0 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93963934811728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4ca70 con 0x5575aef3d8c0 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4da60 con 0x5575aef41d80 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93963934806880] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93963934811728] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93963934813936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4da60 con 0x5575aef41d80 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.093+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4de90 con 0x7f9e04014030 2024-04-27T03:18:13.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x7f9e04014030 2024-04-27T03:18:13.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93963934815552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4de50 con 0x7f9e04014030 2024-04-27T03:18:13.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4ed50 con 0x5575aef41d80 2024-04-27T03:18:13.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93963934813936] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93963934815552] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x7f9e04014030 2024-04-27T03:18:13.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93963934818944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4e8a0 con 0x5575aef41d80 2024-04-27T03:18:13.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4f5f0 con 0x5575aef46e00 2024-04-27T03:18:13.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93963934822720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4e8a0 con 0x5575aef46e00 2024-04-27T03:18:13.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.097+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef50460 con 0x5575aef46e00 2024-04-27T03:18:13.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93963934818944] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93963934826416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef50880 con 0x5575aef46e00 2024-04-27T03:18:13.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef51240 con 0x7f9e04014030 2024-04-27T03:18:13.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 93963934822720] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93963934826416] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'135 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x7f9e04014030 2024-04-27T03:18:13.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93963934827472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef507f0 con 0x7f9e04014030 2024-04-27T03:18:13.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=0 2024-04-27T03:18:13.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=1 2024-04-27T03:18:13.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=2 2024-04-27T03:18:13.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=3 2024-04-27T03:18:13.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=4 2024-04-27T03:18:13.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=5 2024-04-27T03:18:13.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=6 2024-04-27T03:18:13.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93963934827472] v27'136 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x7f9e04014030 2024-04-27T03:18:13.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 20 add_watcher() i=7 2024-04-27T03:18:13.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:13.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef54270 con 0x5575aef3d8c0 2024-04-27T03:18:13.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.101+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93963934829968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef51660 con 0x5575aef3d8c0 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93963934829968] v27'136 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9cbf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 5 note: GC not initialized 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9cbe7e4640 20 reqs_thread_entry: start 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:13.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 20 Filter name: none 2024-04-27T03:18:13.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9c2ffff640 20 reqs_thread_entry: start 2024-04-27T03:18:13.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5575aef77410 con 0x5575aef3d8c0 2024-04-27T03:18:13.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 testx/myfoo [call out=48b,stat out=16b,read 0~193 out=193b,getxattrs out=75b] v0'0 uv3 ondisk = 0) ==== 281+0+332 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 20 rados_obj.operate() r=0 bl.length=193 2024-04-27T03:18:13.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9cbf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.105+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93963934829968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef566a0 con 0x5575aef3d8c0 2024-04-27T03:18:13.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.109+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 93963934829968] v27'137 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.109+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93963934806880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef50d90 con 0x5575aef41d80 2024-04-27T03:18:13.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.109+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 93963934806880] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.109+0000 7f9e0bbb3980 20 remove_watcher() i=0 2024-04-27T03:18:13.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.109+0000 7f9e0bbb3980 2 removed watcher, disabling cache 2024-04-27T03:18:13.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.109+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93963934811728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef50d90 con 0x5575aef3d8c0 2024-04-27T03:18:13.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 93963934811728] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x5575aef3d8c0 2024-04-27T03:18:13.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e0bbb3980 20 remove_watcher() i=1 2024-04-27T03:18:13.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93963934813936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4c650 con 0x5575aef41d80 2024-04-27T03:18:13.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 93963934813936] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e0bbb3980 20 remove_watcher() i=2 2024-04-27T03:18:13.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93963934815552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef50d90 con 0x7f9e04014030 2024-04-27T03:18:13.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 93963934815552] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x7f9e04014030 2024-04-27T03:18:13.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e0bbb3980 20 remove_watcher() i=3 2024-04-27T03:18:13.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.113+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93963934818944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4cef0 con 0x5575aef41d80 2024-04-27T03:18:13.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e09ecf640 1 -- 172.21.15.110:0/884970967 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 93963934818944] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9e04038020 con 0x5575aef41d80 2024-04-27T03:18:13.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e0bbb3980 20 remove_watcher() i=4 2024-04-27T03:18:13.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93963934822720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4c650 con 0x5575aef46e00 2024-04-27T03:18:13.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 93963934822720] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e0bbb3980 20 remove_watcher() i=5 2024-04-27T03:18:13.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93963934826416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef50d90 con 0x5575aef46e00 2024-04-27T03:18:13.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e096ce640 1 -- 172.21.15.110:0/884970967 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 93963934826416] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9dfc0b3040 con 0x5575aef46e00 2024-04-27T03:18:13.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e0bbb3980 20 remove_watcher() i=6 2024-04-27T03:18:13.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.117+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93963934827472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5575aef4cef0 con 0x7f9e04014030 2024-04-27T03:18:13.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 93963934827472] v27'137 uv135 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9df403c1c0 con 0x7f9e04014030 2024-04-27T03:18:13.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 20 remove_watcher() i=7 2024-04-27T03:18:13.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef41d80 msgr2=0x5575aef44170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef41d80 0x5575aef44170 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aef3d8c0 msgr2=0x5575aef3fd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aef3d8c0 0x5575aef3fd50 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9e04014030 msgr2=0x7f9e04031770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9e04014030 0x7f9e04031770 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5575aef46e00 msgr2=0x5575aef491f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5575aef46e00 0x5575aef491f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cd4034650 msgr2=0x7f9cd4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cd4034650 0x7f9cd4036b10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc01b0b0 tx=0x7f9dfc0de000 comp rx=0 tx=0).stop 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e08ecd640 1 -- 172.21.15.110:0/884970967 reap_dead start 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 msgr2=0x5575aef32240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aef32240 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9e04020ec0 tx=0x7f9e0404a590 comp rx=0 tx=0).stop 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 shutdown_connections 2024-04-27T03:18:13.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/884970967 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aed83c70 0x5575aef32240 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 >> 172.21.15.110:0/884970967 conn(0x5575aef24ed0 msgr2=0x5575aef22530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 shutdown_connections 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/884970967 wait complete. 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9ce00346a0 msgr2=0x7f9ce0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9ce00346a0 0x7f9ce0036b60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x5575aef0be10 tx=0x7f9dfc0e2bf0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aef1b9e0 msgr2=0x5575aef1be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aef1b9e0 0x5575aef1be00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f9e040056d0 tx=0x7f9e04017820 comp rx=0 tx=0).stop 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 shutdown_connections 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9ce00346a0 0x7f9ce0036b60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/2229269127 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aef1b9e0 0x5575aef1be00 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 >> 172.21.15.110:0/2229269127 conn(0x5575aef107f0 msgr2=0x5575aef11300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 shutdown_connections 2024-04-27T03:18:13.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/2229269127 wait complete. 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef08de0 msgr2=0x5575aef0b1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef08de0 0x5575aef0b1d0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aee0ebf0 msgr2=0x5575aef079a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aee0ebf0 0x5575aef079a0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cf00346a0 msgr2=0x7f9cf0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cf00346a0 0x7f9cf0036b60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9df4000f70 tx=0x7f9df40012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 msgr2=0x5575aef00b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aef00b10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc018eb0 tx=0x7f9dfc018020 comp rx=0 tx=0).stop 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 shutdown_connections 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5575aef08de0 0x5575aef0b1d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5575aee0ebf0 0x5575aef079a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9cf00346a0 0x7f9cf0036b60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 --2- 172.21.15.110:0/1997832178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5575aee0e2d0 0x5575aef00b10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 >> 172.21.15.110:0/1997832178 conn(0x5575aee09ba0 msgr2=0x5575aeef4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 shutdown_connections 2024-04-27T03:18:13.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.121+0000 7f9e0bbb3980 1 -- 172.21.15.110:0/1997832178 wait complete. 2024-04-27T03:18:13.138 INFO:teuthology.orchestra.run.smithi110.stdout:{"key":"bucket:testx/myfoo","ver":{"tag":"C/W8HbASFW26LaBfcTrcuSqS","ver":2},"mtime":"2024-04-27T03:18:12.928422Z","data":{"bucket":{"name":"myfoo","marker":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","bucket_id":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","tenant":"testx","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""}},"owner":"testx$tenanteduser","creation_time":"2024-04-27T03:18:09.694744Z","linked":true,"has_bucket_info":false}} 2024-04-27T03:18:13.138 DEBUG:tasks.util.rgw: json result: {'key': 'bucket:testx/myfoo', 'ver': {'tag': 'C/W8HbASFW26LaBfcTrcuSqS', 'ver': 2}, 'mtime': '2024-04-27T03:18:12.928422Z', 'data': {'bucket': {'name': 'myfoo', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'bucket_id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'tenant': 'testx', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}}, 'owner': 'testx$tenanteduser', 'creation_time': '2024-04-27T03:18:09.694744Z', 'linked': True, 'has_bucket_info': False}} 2024-04-27T03:18:13.138 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'link', '--bucket', 'testx/myfoo', '--uid', 'foo'] 2024-04-27T03:18:13.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'link', '--bucket', 'testx/myfoo', '--uid', 'foo'] 2024-04-27T03:18:13.139 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket link --bucket testx/myfoo --uid foo 2024-04-27T03:18:13.208 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:13.208 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:13.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fadd4a980 1 Processor -- start 2024-04-27T03:18:13.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fadd4a980 1 -- start start 2024-04-27T03:18:13.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fadd4a980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559790a4e970 0x559790a4ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fadd4a980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55979090d0d0 con 0x559790a4f290 2024-04-27T03:18:13.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fadd4a980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x559790902430 con 0x559790a4e970 2024-04-27T03:18:13.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fab865640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559790a4e970 0x559790a4ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fab064640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x559790a4f290 0x559790a55c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34936/0 (socket says 172.21.15.110:34936) 2024-04-27T03:18:13.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fab064640 1 -- 172.21.15.110:0/3585647695 learned_addr learned my addr 172.21.15.110:0/3585647695 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:13.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fab865640 1 -- 172.21.15.110:0/3585647695 >> v1:172.21.15.77:6789/0 conn(0x559790a4f290 legacy=0x559790a55c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.201+0000 7f4fab865640 1 -- 172.21.15.110:0/3585647695 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597908fb510 con 0x559790a4e970 2024-04-27T03:18:13.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fab865640 1 --2- 172.21.15.110:0/3585647695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790a4ed50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00a480 tx=0x7f4f9c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcae9bdae43df3a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4faa863640 1 -- 172.21.15.110:0/3585647695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f9c0b0070 con 0x559790a4e970 2024-04-27T03:18:13.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4faa863640 1 -- 172.21.15.110:0/3585647695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9c0ab540 con 0x559790a4e970 2024-04-27T03:18:13.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4faa863640 1 -- 172.21.15.110:0/3585647695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f9c0ab840 con 0x559790a4e970 2024-04-27T03:18:13.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3585647695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 msgr2=0x559790a4ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3585647695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790a4ed50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00a480 tx=0x7f4f9c01b280 comp rx=0 tx=0).stop 2024-04-27T03:18:13.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3585647695 shutdown_connections 2024-04-27T03:18:13.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3585647695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790a4ed50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3585647695 >> 172.21.15.110:0/3585647695 conn(0x559790a481d0 msgr2=0x559790a4a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3585647695 shutdown_connections 2024-04-27T03:18:13.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.205+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3585647695 wait complete. 2024-04-27T03:18:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.217+0000 7f4fadd4a980 1 Processor -- start 2024-04-27T03:18:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.217+0000 7f4fadd4a980 1 -- start start 2024-04-27T03:18:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.217+0000 7f4fadd4a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b3b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.217+0000 7f4fadd4a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5597909512d0 con 0x559790a4e970 2024-04-27T03:18:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.217+0000 7f4fab865640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b3b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b3b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52400/0 (socket says 172.21.15.110:52400) 2024-04-27T03:18:13.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 -- 172.21.15.110:0/3417800261 learned_addr learned my addr 172.21.15.110:0/3417800261 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 -- 172.21.15.110:0/3417800261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597908d7ae0 con 0x559790a4e970 2024-04-27T03:18:13.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 --2- 172.21.15.110:0/3417800261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b3b6c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0ae040 tx=0x7f4f9c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bdae8cb928aed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4e99e0b640 1 -- 172.21.15.110:0/3417800261 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f9c0b0040 con 0x559790a4e970 2024-04-27T03:18:13.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4e99e0b640 1 -- 172.21.15.110:0/3417800261 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9c00a300 con 0x559790a4e970 2024-04-27T03:18:13.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4e99e0b640 1 -- 172.21.15.110:0/3417800261 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f9c0ab820 con 0x559790a4e970 2024-04-27T03:18:13.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3417800261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 msgr2=0x559790b3b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3417800261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b3b6c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0ae040 tx=0x7f4f9c003030 comp rx=0 tx=0).stop 2024-04-27T03:18:13.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3417800261 shutdown_connections 2024-04-27T03:18:13.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3417800261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b3b6c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3417800261 >> 172.21.15.110:0/3417800261 conn(0x559790a481d0 msgr2=0x559790b37250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3417800261 shutdown_connections 2024-04-27T03:18:13.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3417800261 wait complete. 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 Processor -- start 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- start start 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559790a57e40 con 0x559790a4e970 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b425e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52404/0 (socket says 172.21.15.110:52404) 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 -- 172.21.15.110:0/213707091 learned_addr learned my addr 172.21.15.110:0/213707091 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559790b42b20 con 0x559790a4e970 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fab865640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b425e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c019930 tx=0x7f4f9c018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4e93fff640 1 -- 172.21.15.110:0/213707091 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f9c0b0040 con 0x559790a4e970 2024-04-27T03:18:13.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559790b42db0 con 0x559790a4e970 2024-04-27T03:18:13.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4e93fff640 1 -- 172.21.15.110:0/213707091 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9c0c0870 con 0x559790a4e970 2024-04-27T03:18:13.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559790b459e0 con 0x559790a4e970 2024-04-27T03:18:13.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.221+0000 7f4e93fff640 1 -- 172.21.15.110:0/213707091 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f9c0c0b50 con 0x559790a4e970 2024-04-27T03:18:13.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4e93fff640 1 -- 172.21.15.110:0/213707091 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4f9c0c0cf0 con 0x559790a4e970 2024-04-27T03:18:13.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4e93fff640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e940346a0 0x7f4e94036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4e93fff640 1 -- 172.21.15.110:0/213707091 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f4f9c0bb070 con 0x559790a4e970 2024-04-27T03:18:13.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fab064640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e940346a0 0x7f4e94036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790a4f290 0x559790b494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b49a50 con 0x559790a4f290 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fac066640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790a4f290 0x559790b494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fab064640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e940346a0 0x7f4e94036b60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0001da0 tx=0x7f4fa0001300 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fac066640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790a4f290 0x559790b494c0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fac066640 1 -- 172.21.15.110:0/213707091 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4fa400a4d0 con 0x559790a4f290 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b4a820 0x559790b4cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b4d270 con 0x559790b4a820 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fab865640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b4a820 0x559790b4cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fab865640 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b4a820 0x559790b4cc10 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fab865640 1 -- 172.21.15.110:0/213707091 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b4a820 2024-04-27T03:18:13.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.225+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b4eb20 con 0x559790a4f290 2024-04-27T03:18:13.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fac066640 1 -- 172.21.15.110:0/213707091 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f4fa400a4d0 con 0x559790a4f290 2024-04-27T03:18:13.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b4dd50 con 0x559790b4a820 2024-04-27T03:18:13.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fab865640 1 -- 172.21.15.110:0/213707091 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b4a820 2024-04-27T03:18:13.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b4e140 con 0x559790a4f290 2024-04-27T03:18:13.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fac066640 1 -- 172.21.15.110:0/213707091 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f4fa400a4d0 con 0x559790a4f290 2024-04-27T03:18:13.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 Processor -- start 2024-04-27T03:18:13.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 -- start start 2024-04-27T03:18:13.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b5d440 0x559790b5d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559790955310 con 0x559790b5d440 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fac066640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b5d440 0x559790b5d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fac066640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b5d440 0x559790b5d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52420/0 (socket says 172.21.15.110:52420) 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fac066640 1 -- 172.21.15.110:0/3805220296 learned_addr learned my addr 172.21.15.110:0/3805220296 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fac066640 1 -- 172.21.15.110:0/3805220296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559790b5dda0 con 0x559790b5d440 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fac066640 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b5d440 0x559790b5d860 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4017680 tx=0x7f4fa4017a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4e917fa640 1 -- 172.21.15.110:0/3805220296 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa40157d0 con 0x559790b5d440 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559790b5e840 con 0x559790b5d440 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4e917fa640 1 -- 172.21.15.110:0/3805220296 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa4015970 con 0x559790b5d440 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559790b5ed10 con 0x559790b5d440 2024-04-27T03:18:13.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.229+0000 7f4e917fa640 1 -- 172.21.15.110:0/3805220296 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa4015c50 con 0x559790b5d440 2024-04-27T03:18:13.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4e917fa640 1 -- 172.21.15.110:0/3805220296 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4fa4012440 con 0x559790b5d440 2024-04-27T03:18:13.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4e917fa640 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e840346a0 0x7f4e84036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4e917fa640 1 -- 172.21.15.110:0/3805220296 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f4fa400e040 con 0x559790b5d440 2024-04-27T03:18:13.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fab865640 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e840346a0 0x7f4e84036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 Processor -- start 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- start start 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fab865640 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e840346a0 0x7f4e84036b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x559790b4d870 tx=0x7f4f9c0b6f30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b6aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559790b3bd50 con 0x559790b6a5f0 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fac066640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b6aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fac066640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b6aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52432/0 (socket says 172.21.15.110:52432) 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fac066640 1 -- 172.21.15.110:0/3634561425 learned_addr learned my addr 172.21.15.110:0/3634561425 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fac066640 1 -- 172.21.15.110:0/3634561425 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559790b6afe0 con 0x559790b6a5f0 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fac066640 1 --2- 172.21.15.110:0/3634561425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b6aa10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4015f60 tx=0x7f4fa403c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e3b97965bf9269 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4e8affd640 1 -- 172.21.15.110:0/3634561425 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa40491d0 con 0x559790b6a5f0 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4e8affd640 1 -- 172.21.15.110:0/3634561425 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa401f3f0 con 0x559790b6a5f0 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4e8affd640 1 -- 172.21.15.110:0/3634561425 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa4012b00 con 0x559790b6a5f0 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3634561425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 msgr2=0x559790b6aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3634561425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b6aa10 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4015f60 tx=0x7f4fa403c140 comp rx=0 tx=0).stop 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3634561425 shutdown_connections 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3634561425 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b6aa10 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3634561425 >> 172.21.15.110:0/3634561425 conn(0x559790b66980 msgr2=0x559790b649e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3634561425 shutdown_connections 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3634561425 wait complete. 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 Processor -- start 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- start start 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b719a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fadd4a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559790b6b6a0 con 0x559790b6a5f0 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.233+0000 7f4fac066640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fac066640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b719a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52438/0 (socket says 172.21.15.110:52438) 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 learned_addr learned my addr 172.21.15.110:0/2808797230 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559790b71ee0 con 0x559790b6a5f0 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fac066640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b719a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4046040 tx=0x7f4fa404a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4e897fa640 1 -- 172.21.15.110:0/2808797230 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa4011a40 con 0x559790b6a5f0 2024-04-27T03:18:13.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559790b72170 con 0x559790b6a5f0 2024-04-27T03:18:13.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4e897fa640 1 -- 172.21.15.110:0/2808797230 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa4011be0 con 0x559790b6a5f0 2024-04-27T03:18:13.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4e897fa640 1 -- 172.21.15.110:0/2808797230 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4fa40113a0 con 0x559790b6a5f0 2024-04-27T03:18:13.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559790b74e50 con 0x559790b6a5f0 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4e897fa640 1 -- 172.21.15.110:0/2808797230 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4fa4037020 con 0x559790b6a5f0 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4e897fa640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e74034650 0x7f4e74036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4e897fa640 1 -- 172.21.15.110:0/2808797230 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f4fa4035020 con 0x559790b6a5f0 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fab865640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e74034650 0x7f4e74036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790b7d0e0 0x559790b7f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b7fcd0 con 0x559790b7d0e0 2024-04-27T03:18:13.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fab064640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790b7d0e0 0x559790b7f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fab865640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e74034650 0x7f4e74036b10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0e2110 tx=0x7f4f9c0d3000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fab064640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790b7d0e0 0x559790b7f580 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4fa000e640 con 0x559790b7d0e0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 realm 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b7c5b0 con 0x559790b7d0e0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4fa000e640 con 0x559790b7d0e0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:13.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b807d0 con 0x559790b7d0e0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4fa000e640 con 0x559790b7d0e0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b815b0 0x559790b839a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.237+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b84020 con 0x559790b815b0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fac066640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b815b0 0x559790b839a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fac066640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b815b0 0x559790b839a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fadd4a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b84870 con 0x559790b7d0e0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f4fa000e640 con 0x559790b7d0e0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fadd4a980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fadd4a980 20 searching for the correct realm 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559790b865e0 0x559790b889d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b865e0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fab865640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559790b865e0 0x559790b889d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fab865640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559790b865e0 0x559790b889d0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fac066640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4fa403d020 0x7f4fa401a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x7f4fa403d020 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fab064640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4fa403d020 0x7f4fa401a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.241+0000 7f4fab064640 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4fa403d020 0x7f4fa401a070 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa000e640 con 0x7f4fa403d020 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa000e640 con 0x559790b7d0e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b865e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x7f4fa403d020 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x7f4fa403d020 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x7f4fa403d020 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f4fa0029040 con 0x7f4fa403d020 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.245+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b865e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b865e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b865e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x7f4fa403d020 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x7f4fa403d020 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b7d0e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x7f4fa403d020 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x7f4fa403d020 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b865e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x7f4fa403d020 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa0029040 con 0x7f4fa403d020 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.249+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b865e0 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x559790b89050 con 0x559790b815b0 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:13.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b851c0 con 0x559790b7d0e0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4fa0029040 con 0x559790b7d0e0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b84dd0 con 0x559790b815b0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b85ab0 con 0x559790b7d0e0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 zone default found 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 4 Realm: () 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 zonegroup default 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b894d0 con 0x7f4fa403d020 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b8b2f0 con 0x559790b865e0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b8b690 con 0x559790b865e0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 started sync module instance, tier type = 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.253+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8a790 con 0x559790b815b0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94109456444592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8a790 con 0x559790b815b0 2024-04-27T03:18:13.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8c210 con 0x559790b7d0e0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94109456448736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8c1d0 con 0x559790b7d0e0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8d140 con 0x559790b815b0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94109456444592] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.257+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94109456448736] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94109456451072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8d140 con 0x559790b815b0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8d590 con 0x7f4fa403d020 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94109456442080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8d550 con 0x7f4fa403d020 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8e470 con 0x559790b815b0 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94109456442080] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94109456451072] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94109456452688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8e470 con 0x559790b815b0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.261+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8e890 con 0x559790b865e0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94109456456064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8f0e0 con 0x559790b865e0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8faa0 con 0x559790b865e0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94109456452688] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94109456456064] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94109456463344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8f0e0 con 0x559790b865e0 2024-04-27T03:18:13.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b8fa90 con 0x7f4fa403d020 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'138 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94109456465552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b90840 con 0x7f4fa403d020 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=0 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=1 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=2 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=3 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=4 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=5 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94109456463344] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=6 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94109456465552] v27'139 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 20 add_watcher() i=7 2024-04-27T03:18:13.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:13.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.265+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790b90840 con 0x559790b7d0e0 2024-04-27T03:18:13.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94109456466752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790b90760 con 0x559790b7d0e0 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94109456466752] v27'138 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4e61fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 5 note: GC not initialized 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4e60fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 Filter name: none 2024-04-27T03:18:13.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4dda7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790bb7020 con 0x559790b7d0e0 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 cache get: name=default.rgw.meta+root+testx/myfoo : miss 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790bb6b70 con 0x559790b7d0e0 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 testx/myfoo [call out=48b,read 0~193 out=193b] v0'0 uv3 ondisk = 0) ==== 197+0+241 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 rados_obj.operate() r=0 bl.length=193 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 cache put: name=default.rgw.meta+root+testx/myfoo info.flags=0x11 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 adding default.rgw.meta+root+testx/myfoo to cache LRU end 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 10 cache get: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.269+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.3 8:c2eafa05:root::.bucket.meta.testx%3amyfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790bbad20 con 0x559790b7d0e0 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(64 .bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~412 out=412b,getxattrs out=298b] v0'0 uv2 ondisk = 0) ==== 338+0+774 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 20 rados_obj.operate() r=0 bl.length=412 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 cache put: name=default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 adding default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 updating xattr: name=user.rgw.acl bl.length()=203 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:13.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 cache get: name=default.rgw.meta+users.uid+foo : hit (requested=0x13, cached=0x13) 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bbf3a0 con 0x559790b7d0e0 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(65 testx$tenanteduser.buckets [call] v27'2 uv2 ondisk = 0) ==== 170+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [create,call version.inc in=38b,writefull 0~379 in=379b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bc2cb0 con 0x7f4fa403d020 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(66 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [create,call,writefull 0~379,setxattr (123)] v27'7 uv7 ondisk = 0) ==== 332+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x7 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [notify cookie 94109456665168 in=740b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790bc2790 con 0x559790b815b0 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94109456451072 notify 115964117017 ret 0) ==== 770+0+0 (crc 0 0 0) 0x559790b74e50 con 0x559790b815b0 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4e8affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117017 cookie 94109456451072 notifier 4548 bl.length()=728 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4e8affd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x7 2024-04-27T03:18:13.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4e8affd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4e8affd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4e8affd640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f4e70003eb0 con 0x559790b815b0 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [notify cookie 94109456665168 out=8b] v0'0 uv138 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.273+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.2 [notify-ack] v0'0 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94109456665168 notify 115964117017 ret 0) ==== 42+0+48 (crc 0 0 0) 0x559790b72170 con 0x559790b815b0 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_sources= new_sources= 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential sources added=[] removed=[] 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_dests= new_dests= 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential dests added=[] removed=[] 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bc2850 con 0x7f4fa403d020 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 foo.buckets [call] v27'26 uv26 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 8.9 8:91b71841:root::myfoo:head [delete,create,call version.inc in=38b,writefull 0~165 in=165b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bc2850 con 0x7f4fa403d020 2024-04-27T03:18:13.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(70 myfoo [delete,create,call,writefull 0~165] v27'27 uv27 ondisk = 0) ==== 275+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94109456671648 in=326b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790bc27c0 con 0x559790b865e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== watch-notify(notify (1) cookie 94109456463344 notify 115964117007 ret 0) ==== 356+0+0 (crc 0 0 0) 0x559790b459e0 con 0x559790b865e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4e8a7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117007 cookie 94109456463344 notifier 4548 bl.length()=314 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(71 notify.6 [notify cookie 94109456671648 out=8b] v0'0 uv138 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4e8a7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo info.flags=0x7 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4e8a7fc640 10 rgw watcher librados: moving default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4e8a7fc640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f4de0003410 con 0x559790b865e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(72 notify.6 [notify-ack] v0'0 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== watch-notify(notify_complete (2) cookie 94109456671648 notify 115964117007 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f4f9c0cd030 con 0x559790b865e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.277+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:73 8.5 8:a7a6c9d8:root::testx%2fmyfoo:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bc2850 con 0x559790b7d0e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(73 testx/myfoo [call,delete] v27'4 uv4 ondisk = 0) ==== 197+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fadd4a980 10 removing default.rgw.meta+root+testx/myfoo from cache 2024-04-27T03:18:13.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fadd4a980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testx/myfoo, ofs0, ns] 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 7.d 7:b4812045:::notify.4:head [notify cookie 94109456671648 in=167b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790bc27c0 con 0x559790b815b0 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94109456452688 notify 115964117018 ret 0) ==== 197+0+0 (crc 0 0 0) 0x559790b72170 con 0x559790b815b0 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4e8a7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117018 cookie 94109456452688 notifier 4548 bl.length()=155 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4e8a7fc640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f4de0003410 con 0x559790b815b0 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(74 notify.4 [notify cookie 94109456671648 out=8b] v0'0 uv138 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(75 notify.4 [notify-ack] v0'0 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== watch-notify(notify_complete (2) cookie 94109456671648 notify 115964117018 ret 0) ==== 42+0+48 (crc 0 0 0) 0x559790b71ee0 con 0x559790b815b0 2024-04-27T03:18:13.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.281+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 8.3 8:c2eafa05:root::.bucket.meta.testx%3amyfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bc2850 con 0x559790b7d0e0 2024-04-27T03:18:13.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(76 .bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call,delete] v27'3 uv3 ondisk = 0) ==== 254+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fadd4a980 10 removing default.rgw.meta+root+.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 from cache 2024-04-27T03:18:13.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fadd4a980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testx:myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:77 7.0 7:05bf5b68:::notify.1:head [notify cookie 94109456639440 in=224b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x559790bc2850 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== watch-notify(notify (1) cookie 94109456448736 notify 115964116993 ret 0) ==== 254+0+0 (crc 0 0 0) 0x559790bc2bf0 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4e8affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94109456448736 notifier 4548 bl.length()=212 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(77 notify.1 [notify cookie 94109456639440 out=8b] v0'0 uv138 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4e8affd640 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:78 7.0 7:05bf5b68:::notify.1:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f4e70003eb0 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(78 notify.1 [notify-ack] v0'0 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== watch-notify(notify_complete (2) cookie 94109456639440 notify 115964116993 ret 0) ==== 42+0+48 (crc 0 0 0) 0x559790bc1a60 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4e61fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94109456466752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(79 data_loggenerations_metadata [watch unwatch cookie 94109456466752] v27'139 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.285+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94109456444592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x559790b815b0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.289+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(80 notify.0 [watch unwatch cookie 94109456444592] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.289+0000 7f4fadd4a980 20 remove_watcher() i=0 2024-04-27T03:18:13.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.289+0000 7f4fadd4a980 2 removed watcher, disabling cache 2024-04-27T03:18:13.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.289+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94109456448736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x559790b7d0e0 2024-04-27T03:18:13.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.289+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(81 notify.1 [watch unwatch cookie 94109456448736] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x559790b7d0e0 2024-04-27T03:18:13.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.289+0000 7f4fadd4a980 20 remove_watcher() i=1 2024-04-27T03:18:13.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.289+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94109456451072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x559790b815b0 2024-04-27T03:18:13.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(82 notify.2 [watch unwatch cookie 94109456451072] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fadd4a980 20 remove_watcher() i=2 2024-04-27T03:18:13.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:83 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94109456442080] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x7f4fa403d020 2024-04-27T03:18:13.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(83 notify.3 [watch unwatch cookie 94109456442080] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fadd4a980 20 remove_watcher() i=3 2024-04-27T03:18:13.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:84 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94109456452688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x559790b815b0 2024-04-27T03:18:13.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fac066640 1 -- 172.21.15.110:0/2808797230 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(84 notify.4 [watch unwatch cookie 94109456452688] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa402e070 con 0x559790b815b0 2024-04-27T03:18:13.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fadd4a980 20 remove_watcher() i=4 2024-04-27T03:18:13.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.293+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:85 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94109456456064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x559790b865e0 2024-04-27T03:18:13.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(85 notify.5 [watch unwatch cookie 94109456456064] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 20 remove_watcher() i=5 2024-04-27T03:18:13.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94109456463344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x559790b865e0 2024-04-27T03:18:13.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fab865640 1 -- 172.21.15.110:0/2808797230 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 notify.6 [watch unwatch cookie 94109456463344] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f9c0b3040 con 0x559790b865e0 2024-04-27T03:18:13.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 20 remove_watcher() i=6 2024-04-27T03:18:13.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:87 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94109456465552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x559790bb4d50 con 0x7f4fa403d020 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(87 notify.7 [watch unwatch cookie 94109456465552] v27'140 uv138 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4fa003c1c0 con 0x7f4fa403d020 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 20 remove_watcher() i=7 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b815b0 msgr2=0x559790b839a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b815b0 0x559790b839a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790b7d0e0 msgr2=0x559790b7f580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790b7d0e0 0x559790b7f580 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4fa403d020 msgr2=0x7f4fa401a070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4fa403d020 0x7f4fa401a070 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559790b865e0 msgr2=0x559790b889d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559790b865e0 0x559790b889d0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e74034650 msgr2=0x7f4e74036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e74034650 0x7f4e74036b10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0e2110 tx=0x7f4f9c0d3000 comp rx=0 tx=0).stop 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 msgr2=0x559790b719a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fab064640 1 -- 172.21.15.110:0/2808797230 reap_dead start 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b719a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4046040 tx=0x7f4fa404a650 comp rx=0 tx=0).stop 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 shutdown_connections 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/2808797230 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b6a5f0 0x559790b719a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 >> 172.21.15.110:0/2808797230 conn(0x559790b66980 msgr2=0x559790b63fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 shutdown_connections 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/2808797230 wait complete. 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e840346a0 msgr2=0x7f4e84036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e840346a0 0x7f4e84036b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x559790b4d870 tx=0x7f4f9c0b6f30 comp rx=0 tx=0).stop 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b5d440 msgr2=0x559790b5d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b5d440 0x559790b5d860 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4017680 tx=0x7f4fa4017a80 comp rx=0 tx=0).stop 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 shutdown_connections 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e840346a0 0x7f4e84036b60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/3805220296 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790b5d440 0x559790b5d860 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 >> 172.21.15.110:0/3805220296 conn(0x559790b52250 msgr2=0x559790b52d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 shutdown_connections 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/3805220296 wait complete. 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b4a820 msgr2=0x559790b4cc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b4a820 0x559790b4cc10 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790a4f290 msgr2=0x559790b494c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790a4f290 0x559790b494c0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e940346a0 msgr2=0x7f4e94036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e940346a0 0x7f4e94036b60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0001da0 tx=0x7f4fa0001300 comp rx=0 tx=0).stop 2024-04-27T03:18:13.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 msgr2=0x559790b425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b425e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c019930 tx=0x7f4f9c018020 comp rx=0 tx=0).stop 2024-04-27T03:18:13.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 shutdown_connections 2024-04-27T03:18:13.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559790b4a820 0x559790b4cc10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559790a4f290 0x559790b494c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e940346a0 0x7f4e94036b60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.297+0000 7f4fadd4a980 1 --2- 172.21.15.110:0/213707091 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559790a4e970 0x559790b425e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.301+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 >> 172.21.15.110:0/213707091 conn(0x559790a481d0 msgr2=0x559790b35d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.301+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 shutdown_connections 2024-04-27T03:18:13.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.301+0000 7f4fadd4a980 1 -- 172.21.15.110:0/213707091 wait complete. 2024-04-27T03:18:13.317 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--tenant', 'testx', '--uid', 'tenanteduser'] 2024-04-27T03:18:13.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--tenant', 'testx', '--uid', 'tenanteduser'] 2024-04-27T03:18:13.317 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --tenant testx --uid tenanteduser 2024-04-27T03:18:13.392 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:13.392 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:13.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cdb4f980 1 Processor -- start 2024-04-27T03:18:13.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cdb4f980 1 -- start start 2024-04-27T03:18:13.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cdb4f980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5617922c4010 0x561792352f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cdb4f980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x56179220f0d0 con 0x561792353470 2024-04-27T03:18:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cdb4f980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x561792204430 con 0x5617922c4010 2024-04-27T03:18:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cb66a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5617922c4010 0x561792352f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cb66a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5617922c4010 0x561792352f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52444/0 (socket says 172.21.15.110:52444) 2024-04-27T03:18:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cae69640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x561792353470 0x561792355860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:34950/0 (socket says 172.21.15.110:34950) 2024-04-27T03:18:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cb66a640 1 -- 172.21.15.110:0/666483848 learned_addr learned my addr 172.21.15.110:0/666483848 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cb66a640 1 -- 172.21.15.110:0/666483848 >> v1:172.21.15.77:6789/0 conn(0x561792353470 legacy=0x561792355860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.385+0000 7fe0cb66a640 1 -- 172.21.15.110:0/666483848 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617921fd510 con 0x5617922c4010 2024-04-27T03:18:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/666483848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x561792352f30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00a5c0 tx=0x7fe0bc01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2f6014addef1d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0ca668640 1 -- 172.21.15.110:0/666483848 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc0a6070 con 0x5617922c4010 2024-04-27T03:18:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0ca668640 1 -- 172.21.15.110:0/666483848 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0bc0af070 con 0x5617922c4010 2024-04-27T03:18:13.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0ca668640 1 -- 172.21.15.110:0/666483848 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc0aa5c0 con 0x5617922c4010 2024-04-27T03:18:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/666483848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 msgr2=0x561792352f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/666483848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x561792352f30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00a5c0 tx=0x7fe0bc01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/666483848 shutdown_connections 2024-04-27T03:18:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/666483848 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x561792352f30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/666483848 >> 172.21.15.110:0/666483848 conn(0x56179234aa50 msgr2=0x56179234ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/666483848 shutdown_connections 2024-04-27T03:18:13.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.389+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/666483848 wait complete. 2024-04-27T03:18:13.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.401+0000 7fe0cdb4f980 1 Processor -- start 2024-04-27T03:18:13.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.401+0000 7fe0cdb4f980 1 -- start start 2024-04-27T03:18:13.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.401+0000 7fe0cdb4f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x56179243b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.401+0000 7fe0cdb4f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5617922532d0 con 0x5617922c4010 2024-04-27T03:18:13.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x56179243b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x56179243b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52448/0 (socket says 172.21.15.110:52448) 2024-04-27T03:18:13.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 -- 172.21.15.110:0/1896134366 learned_addr learned my addr 172.21.15.110:0/1896134366 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 -- 172.21.15.110:0/1896134366 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617921d9ae0 con 0x5617922c4010 2024-04-27T03:18:13.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/1896134366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x56179243b440 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00a5c0 tx=0x7fe0bc003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d5b804ecf58ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fdfb960a640 1 -- 172.21.15.110:0/1896134366 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc0aa9b0 con 0x5617922c4010 2024-04-27T03:18:13.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fdfb960a640 1 -- 172.21.15.110:0/1896134366 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0bc0aab50 con 0x5617922c4010 2024-04-27T03:18:13.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fdfb960a640 1 -- 172.21.15.110:0/1896134366 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc0aae50 con 0x5617922c4010 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1896134366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 msgr2=0x56179243b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1896134366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x56179243b440 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00a5c0 tx=0x7fe0bc003030 comp rx=0 tx=0).stop 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1896134366 shutdown_connections 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1896134366 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x56179243b440 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1896134366 >> 172.21.15.110:0/1896134366 conn(0x56179234aa50 msgr2=0x561792436fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1896134366 shutdown_connections 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1896134366 wait complete. 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 Processor -- start 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 -- start start 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x5617924423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cdb4f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561792357a90 con 0x5617922c4010 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x5617924423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x5617924423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52454/0 (socket says 172.21.15.110:52454) 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 -- 172.21.15.110:0/1326365880 learned_addr learned my addr 172.21.15.110:0/1326365880 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561792442900 con 0x5617922c4010 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.405+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x5617924423c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00a5c0 tx=0x7fe0bc0180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fdfb7e07640 1 -- 172.21.15.110:0/1326365880 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc0c6750 con 0x5617922c4010 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561792442b90 con 0x5617922c4010 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fdfb7e07640 1 -- 172.21.15.110:0/1326365880 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0bc0c68f0 con 0x5617922c4010 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fdfb7e07640 1 -- 172.21.15.110:0/1326365880 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0bc0c6bd0 con 0x5617922c4010 2024-04-27T03:18:13.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561792445770 con 0x5617922c4010 2024-04-27T03:18:13.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fdfb7e07640 1 -- 172.21.15.110:0/1326365880 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe0bc0ce020 con 0x5617922c4010 2024-04-27T03:18:13.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fdfb7e07640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfb0034650 0x7fdfb0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fdfb7e07640 1 -- 172.21.15.110:0/1326365880 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe0bc0cc020 con 0x5617922c4010 2024-04-27T03:18:13.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cae69640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfb0034650 0x7fdfb0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561792353470 0x561792449250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792449790 con 0x561792353470 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561792353470 0x561792449250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cae69640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfb0034650 0x7fdfb0036b10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00012f0 tx=0x7fe0c0006700 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561792353470 0x561792449250 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/1326365880 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792353470 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56179244a580 0x56179244c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179244cfb0 con 0x56179244a580 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56179244a580 0x56179244c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.409+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56179244a580 0x56179244c970 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cb66a640 1 -- 172.21.15.110:0/1326365880 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x56179244a580 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179244e860 con 0x561792353470 2024-04-27T03:18:13.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/1326365880 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792353470 2024-04-27T03:18:13.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179244da90 con 0x56179244a580 2024-04-27T03:18:13.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cb66a640 1 -- 172.21.15.110:0/1326365880 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x56179244a580 2024-04-27T03:18:13.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179244de80 con 0x561792353470 2024-04-27T03:18:13.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/1326365880 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792353470 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 Processor -- start 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 -- start start 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179245d160 0x56179245d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561792257310 con 0x56179245d160 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cbe6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179245d160 0x56179245d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cbe6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179245d160 0x56179245d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52462/0 (socket says 172.21.15.110:52462) 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/1180784341 learned_addr learned my addr 172.21.15.110:0/1180784341 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/1180784341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56179245dac0 con 0x56179245d160 2024-04-27T03:18:13.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179245d160 0x56179245d580 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400aab0 tx=0x7fe0c400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fdfb5602640 1 -- 172.21.15.110:0/1180784341 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0c4007d30 con 0x56179245d160 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56179245e560 con 0x56179245d160 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fdfb5602640 1 -- 172.21.15.110:0/1180784341 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c401d070 con 0x56179245d160 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56179245ea30 con 0x56179245d160 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.413+0000 7fdfb5602640 1 -- 172.21.15.110:0/1180784341 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0c4022460 con 0x56179245d160 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fdfb5602640 1 -- 172.21.15.110:0/1180784341 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe0c4013070 con 0x56179245d160 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fdfb5602640 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfa4034650 0x7fdfa4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fdfb5602640 1 -- 172.21.15.110:0/1180784341 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe0c404b5c0 con 0x56179245d160 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfa4034650 0x7fdfa4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 Processor -- start 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- start start 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfa4034650 0x7fdfa4036b10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x56179244d5b0 tx=0x7fe0bc0d0d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x56179246a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56179243bad0 con 0x56179246a330 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cbe6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x56179246a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cbe6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x56179246a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52476/0 (socket says 172.21.15.110:52476) 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/1435699478 learned_addr learned my addr 172.21.15.110:0/1435699478 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/1435699478 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56179246ad20 con 0x56179246a330 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/1435699478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x56179246a750 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4049910 tx=0x7fe0c4017460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28db94dcaca68a72 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fdfaeffd640 1 -- 172.21.15.110:0/1435699478 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0c4047070 con 0x56179246a330 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fdfaeffd640 1 -- 172.21.15.110:0/1435699478 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c40227e0 con 0x56179246a330 2024-04-27T03:18:13.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fdfaeffd640 1 -- 172.21.15.110:0/1435699478 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0c4018440 con 0x56179246a330 2024-04-27T03:18:13.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1435699478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 msgr2=0x56179246a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1435699478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x56179246a750 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4049910 tx=0x7fe0c4017460 comp rx=0 tx=0).stop 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1435699478 shutdown_connections 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1435699478 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x56179246a750 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1435699478 >> 172.21.15.110:0/1435699478 conn(0x561792466670 msgr2=0x5617924646d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1435699478 shutdown_connections 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1435699478 wait complete. 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 Processor -- start 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.417+0000 7fe0cdb4f980 1 -- start start 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x5617924717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56179246b4a0 con 0x56179246a330 2024-04-27T03:18:13.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cbe6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x5617924717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cbe6b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x5617924717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52490/0 (socket says 172.21.15.110:52490) 2024-04-27T03:18:13.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 learned_addr learned my addr 172.21.15.110:0/2824695165 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561792471d30 con 0x56179246a330 2024-04-27T03:18:13.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x5617924717f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40168b0 tx=0x7fe0c40498a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fdfad7fa640 1 -- 172.21.15.110:0/2824695165 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0c4030020 con 0x56179246a330 2024-04-27T03:18:13.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561792471fc0 con 0x56179246a330 2024-04-27T03:18:13.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fdfad7fa640 1 -- 172.21.15.110:0/2824695165 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0c4020c80 con 0x56179246a330 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561792474ca0 con 0x56179246a330 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fdfad7fa640 1 -- 172.21.15.110:0/2824695165 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0c4047030 con 0x56179246a330 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fdfad7fa640 1 -- 172.21.15.110:0/2824695165 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe0c4033020 con 0x56179246a330 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fdfad7fa640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdf94034650 0x7fdf94036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fdfad7fa640 1 -- 172.21.15.110:0/2824695165 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fe0c4048070 con 0x56179246a330 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdf94034650 0x7fdf94036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56179247cea0 0x56179247f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179247fab0 con 0x56179247cea0 2024-04-27T03:18:13.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cae69640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56179247cea0 0x56179247f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdf94034650 0x7fdf94036b10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc01b170 tx=0x7fe0bc0ed000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cae69640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56179247cea0 0x56179247f360 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 20 realm 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179247c2f0 con 0x56179247cea0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.421+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5617924805b0 con 0x56179247cea0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561792481390 0x561792483780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792483e00 con 0x561792481390 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561792481390 0x561792483780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561792481390 0x561792483780 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792484650 con 0x56179247cea0 2024-04-27T03:18:13.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 20 searching for the correct realm 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5617924863c0 0x5617924887b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x5617924863c0 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5617924863c0 0x5617924887b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cb66a640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5617924863c0 0x5617924887b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cbe6b640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe0c4035020 0x7fe0c404c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.425+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x7fe0c4035020 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe0c4035020 0x7fe0c404c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe0c4035020 0x7fe0c404c000 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x7fe0c4035020 2024-04-27T03:18:13.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x5617924863c0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x7fe0c4035020 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x7fe0c4035020 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x7fe0c4035020 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fe0c000f070 con 0x7fe0c4035020 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.429+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x5617924863c0 2024-04-27T03:18:13.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x5617924863c0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x5617924863c0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x7fe0c4035020 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x7fe0c4035020 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x56179247cea0 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x7fe0c4035020 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x7fe0c4035020 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x5617924863c0 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x7fe0c4035020 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c000f070 con 0x7fe0c4035020 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.433+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x5617924863c0 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x561792488e30 con 0x561792481390 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:13.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792484fa0 con 0x56179247cea0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe0c000f070 con 0x56179247cea0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.437+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792484bb0 con 0x561792481390 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792485890 con 0x56179247cea0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:13.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 zone default found 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 4 Realm: () 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 zonegroup default 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792489740 con 0x7fe0c4035020 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179248b970 con 0x5617924863c0 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x56179248bd10 con 0x5617924863c0 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 started sync module instance, tier type = 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248adb0 con 0x561792481390 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94659238474144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248b5c0 con 0x561792481390 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.441+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248cb80 con 0x56179247cea0 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94659238479840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248d250 con 0x56179247cea0 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248dd70 con 0x561792481390 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94659238474144] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94659238479840] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94659238484032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248dd30 con 0x561792481390 2024-04-27T03:18:13.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248e610 con 0x7fe0c4035020 2024-04-27T03:18:13.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94659238486368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248dd30 con 0x7fe0c4035020 2024-04-27T03:18:13.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.445+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248f080 con 0x561792481390 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94659238486368] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94659238484032] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94659238478000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248ebd0 con 0x561792481390 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248f950 con 0x5617924863c0 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94659238491296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248ebd0 con 0x5617924863c0 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5617924906b0 con 0x5617924863c0 2024-04-27T03:18:13.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94659238478000] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.449+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94659238491296] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94659238487424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792490670 con 0x5617924863c0 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792490f50 con 0x7fe0c4035020 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'141 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94659238496928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792491370 con 0x7fe0c4035020 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=0 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=1 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94659238487424] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=2 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=3 2024-04-27T03:18:13.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=4 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=5 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=6 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94659238496928] v27'142 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 20 add_watcher() i=7 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x561792493fe0 con 0x56179247cea0 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.453+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94659238493424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792491370 con 0x56179247cea0 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94659238493424] v27'140 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fdf81fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cdb4f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:13.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cdb4f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:13.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179249cab0 con 0x561792481390 2024-04-27T03:18:13.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'46 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179249cb70 con 0x7fe0c4035020 2024-04-27T03:18:13.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'23 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.457+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179249d330 con 0x5617924863c0 2024-04-27T03:18:13.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.461+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'23 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.461+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179249d7e0 con 0x56179247cea0 2024-04-27T03:18:13.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.461+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'141 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.461+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x561792481390 2024-04-27T03:18:13.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.465+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'46 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.465+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x5617924863c0 2024-04-27T03:18:13.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.465+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'37 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.465+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x561792481390 2024-04-27T03:18:13.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.469+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'24 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.469+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x56179247cea0 2024-04-27T03:18:13.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.469+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'32 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.469+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x561792481390 2024-04-27T03:18:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.473+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'16 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.473+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x561792481390 2024-04-27T03:18:13.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.473+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'22 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.473+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x561792481390 2024-04-27T03:18:13.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.477+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'47 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.477+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x5617924863c0 2024-04-27T03:18:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.477+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'18 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.477+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179249e570 con 0x561792481390 2024-04-27T03:18:13.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.481+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'22 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.481+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179249e570 con 0x56179247cea0 2024-04-27T03:18:13.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.481+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'43 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.481+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498100 con 0x56179247cea0 2024-04-27T03:18:13.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.485+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'142 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.485+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498100 con 0x561792481390 2024-04-27T03:18:13.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.489+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'38 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.489+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x561792481390 2024-04-27T03:18:13.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.489+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'48 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.489+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x5617924863c0 2024-04-27T03:18:13.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.493+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'29 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.493+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x561792481390 2024-04-27T03:18:13.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.497+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'24 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.497+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498780 con 0x56179247cea0 2024-04-27T03:18:13.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.497+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'143 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.497+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x561792481390 2024-04-27T03:18:13.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.501+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'39 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.501+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x561792481390 2024-04-27T03:18:13.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.501+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'32 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.501+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x5617924863c0 2024-04-27T03:18:13.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.505+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'24 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.505+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x561792481390 2024-04-27T03:18:13.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.505+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'34 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.505+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x5617924863c0 2024-04-27T03:18:13.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.509+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'18 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.509+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x5617924863c0 2024-04-27T03:18:13.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.509+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'38 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.509+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x5617924863c0 2024-04-27T03:18:13.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.513+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'24 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.513+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x561792481390 2024-04-27T03:18:13.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.513+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'28 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.513+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498950 con 0x56179247cea0 2024-04-27T03:18:13.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.517+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'33 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.517+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179249e9b0 con 0x561792481390 2024-04-27T03:18:13.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.521+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'40 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.521+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498100 con 0x56179247cea0 2024-04-27T03:18:13.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.521+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'44 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.521+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x561792498100 con 0x7fe0c4035020 2024-04-27T03:18:13.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'22 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fdf80fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 20 Filter name: none 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fdefa7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser : miss 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:94 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5617924b9b00 con 0x5617924863c0 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(94 testx$tenanteduser [call out=48b,read 0~419 out=419b,getxattrs out=75b] v0'0 uv7 ondisk = 0) ==== 246+0+542 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 20 rados_obj.operate() r=0 bl.length=419 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser info.flags=0x13 2024-04-27T03:18:13.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 10 adding default.rgw.meta+users.uid+testx$tenanteduser to cache LRU end 2024-04-27T03:18:13.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:13.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5617924bbfd0 con 0x56179247cea0 2024-04-27T03:18:13.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(95 testx$tenanteduser.buckets [call out=15b] v0'0 uv2 ondisk = 0) ==== 170+0+15 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 10 removing key index: p5YnriCv1nAtykxBrupQ 2024-04-27T03:18:13.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:96 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5617924b9fb0 con 0x5617924863c0 2024-04-27T03:18:13.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(96 p5YnriCv1nAtykxBrupQ [delete] v27'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [notify cookie 94659238669120 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5617924bc870 con 0x561792481390 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== watch-notify(notify (1) cookie 94659238484032 notify 115964117019 ret 0) ==== 212+0+0 (crc 0 0 0) 0x5617924984a0 con 0x561792481390 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fdfaeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117019 cookie 94659238484032 notifier 4555 bl.length()=170 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(97 notify.2 [notify cookie 94659238669120 out=8b] v0'0 uv141 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.525+0000 7fdfaeffd640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fdf90003d70 con 0x561792481390 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(98 notify.2 [notify-ack] v0'0 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== watch-notify(notify_complete (2) cookie 94659238669120 notify 115964117019 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7fe0c4064020 con 0x561792481390 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cdb4f980 10 removing email index: 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cdb4f980 10 removing user buckets index 2024-04-27T03:18:13.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:99 8.c 8:37c7a57e:users.uid::testx$tenanteduser.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5617924b9fb0 con 0x56179247cea0 2024-04-27T03:18:13.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(99 testx$tenanteduser.buckets [delete] v27'3 uv3 ondisk = 0) ==== 170+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cdb4f980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.buckets, ofs0, ns] 2024-04-27T03:18:13.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 7.d 7:b4812045:::notify.4:head [notify cookie 94659238664960 in=187b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5617924b9f20 con 0x561792481390 2024-04-27T03:18:13.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== watch-notify(notify (1) cookie 94659238478000 notify 115964117020 ret 0) ==== 217+0+0 (crc 0 0 0) 0x7fe0c4064020 con 0x561792481390 2024-04-27T03:18:13.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fdfaeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117020 cookie 94659238478000 notifier 4555 bl.length()=175 2024-04-27T03:18:13.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(100 notify.4 [notify cookie 94659238664960 out=8b] v0'0 uv141 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fdfaeffd640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fdf90003d70 con 0x561792481390 2024-04-27T03:18:13.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== watch-notify(notify_complete (2) cookie 94659238664960 notify 115964117020 ret 0) ==== 42+0+48 (crc 0 0 0) 0x561792471fc0 con 0x561792481390 2024-04-27T03:18:13.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(101 notify.4 [notify-ack] v0'0 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cdb4f980 10 removing user index: testx$tenanteduser 2024-04-27T03:18:13.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.529+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 8.13 8:cdfb9947:users.uid::testx$tenanteduser:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5617924bb3c0 con 0x5617924863c0 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(102 testx$tenanteduser [call,call,delete] v27'10 uv10 ondisk = 0) ==== 246+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cdb4f980 10 removing default.rgw.meta+users.uid+testx$tenanteduser from cache 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cdb4f980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser, ofs0, ns] 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:103 7.e 7:7759931f:::notify.3:head [notify cookie 94659238672576 in=179b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5617924bb3c0 con 0x7fe0c4035020 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== watch-notify(notify (1) cookie 94659238486368 notify 115964117001 ret 0) ==== 209+0+0 (crc 0 0 0) 0x56179249e570 con 0x7fe0c4035020 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fdfaeffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117001 cookie 94659238486368 notifier 4555 bl.length()=167 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(103 notify.3 [notify cookie 94659238672576 out=8b] v0'0 uv141 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fdfaeffd640 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:104 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fdf90003d70 con 0x7fe0c4035020 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(104 notify.3 [notify-ack] v0'0 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== watch-notify(notify_complete (2) cookie 94659238672576 notify 115964117001 ret 0) ==== 42+0+48 (crc 0 0 0) 0x56179249c690 con 0x7fe0c4035020 2024-04-27T03:18:13.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fdf81fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94659238493424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5617924b9b00 con 0x56179247cea0 2024-04-27T03:18:13.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(105 data_loggenerations_metadata [watch unwatch cookie 94659238493424] v27'144 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.533+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94659238474144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248fcf0 con 0x561792481390 2024-04-27T03:18:13.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.537+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(106 notify.0 [watch unwatch cookie 94659238474144] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.537+0000 7fe0cdb4f980 20 remove_watcher() i=0 2024-04-27T03:18:13.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.537+0000 7fe0cdb4f980 2 removed watcher, disabling cache 2024-04-27T03:18:13.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.537+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:107 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94659238479840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248fcf0 con 0x56179247cea0 2024-04-27T03:18:13.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.537+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(107 notify.1 [watch unwatch cookie 94659238479840] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x56179247cea0 2024-04-27T03:18:13.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.537+0000 7fe0cdb4f980 20 remove_watcher() i=1 2024-04-27T03:18:13.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.537+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94659238484032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248c7e0 con 0x561792481390 2024-04-27T03:18:13.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(108 notify.2 [watch unwatch cookie 94659238484032] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cdb4f980 20 remove_watcher() i=2 2024-04-27T03:18:13.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:109 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94659238486368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248fcf0 con 0x7fe0c4035020 2024-04-27T03:18:13.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(109 notify.3 [watch unwatch cookie 94659238486368] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cdb4f980 20 remove_watcher() i=3 2024-04-27T03:18:13.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:110 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94659238478000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248d840 con 0x561792481390 2024-04-27T03:18:13.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cbe6b640 1 -- 172.21.15.110:0/2824695165 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(110 notify.4 [watch unwatch cookie 94659238478000] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c40083f0 con 0x561792481390 2024-04-27T03:18:13.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cdb4f980 20 remove_watcher() i=4 2024-04-27T03:18:13.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.541+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:111 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94659238491296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248c7e0 con 0x5617924863c0 2024-04-27T03:18:13.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(111 notify.5 [watch unwatch cookie 94659238491296] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 20 remove_watcher() i=5 2024-04-27T03:18:13.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:112 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94659238487424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248e160 con 0x5617924863c0 2024-04-27T03:18:13.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cb66a640 1 -- 172.21.15.110:0/2824695165 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(112 notify.6 [watch unwatch cookie 94659238487424] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0bc0c6d70 con 0x5617924863c0 2024-04-27T03:18:13.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 20 remove_watcher() i=6 2024-04-27T03:18:13.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:113 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94659238496928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x56179248fcf0 con 0x7fe0c4035020 2024-04-27T03:18:13.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(113 notify.7 [watch unwatch cookie 94659238496928] v27'143 uv141 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe0c00401c0 con 0x7fe0c4035020 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 20 remove_watcher() i=7 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561792481390 msgr2=0x561792483780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561792481390 0x561792483780 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56179247cea0 msgr2=0x56179247f360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56179247cea0 0x56179247f360 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe0c4035020 msgr2=0x7fe0c404c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe0c4035020 0x7fe0c404c000 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5617924863c0 msgr2=0x5617924887b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5617924863c0 0x5617924887b0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdf94034650 msgr2=0x7fdf94036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdf94034650 0x7fdf94036b10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc01b170 tx=0x7fe0bc0ed000 comp rx=0 tx=0).stop 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 msgr2=0x5617924717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/2824695165 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179246a330 0x5617924717f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40168b0 tx=0x7fe0c40498a0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 reap_dead start 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cae69640 1 -- 172.21.15.110:0/2824695165 reap_dead start 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 shutdown_connections 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 >> 172.21.15.110:0/2824695165 conn(0x561792466670 msgr2=0x561792463cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 shutdown_connections 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/2824695165 wait complete. 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfa4034650 msgr2=0x7fdfa4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfa4034650 0x7fdfa4036b10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x56179244d5b0 tx=0x7fe0bc0d0d50 comp rx=0 tx=0).stop 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179245d160 msgr2=0x56179245d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179245d160 0x56179245d580 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400aab0 tx=0x7fe0c400af70 comp rx=0 tx=0).stop 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 shutdown_connections 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfa4034650 0x7fdfa4036b10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1180784341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56179245d160 0x56179245d580 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 >> 172.21.15.110:0/1180784341 conn(0x561792451f70 msgr2=0x561792452a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 shutdown_connections 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.545+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1180784341 wait complete. 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56179244a580 msgr2=0x56179244c970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56179244a580 0x56179244c970 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561792353470 msgr2=0x561792449250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561792353470 0x561792449250 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfb0034650 msgr2=0x7fdfb0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfb0034650 0x7fdfb0036b10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00012f0 tx=0x7fe0c0006700 comp rx=0 tx=0).stop 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 msgr2=0x5617924423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x5617924423c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc00a5c0 tx=0x7fe0bc0180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 shutdown_connections 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56179244a580 0x56179244c970 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561792353470 0x561792449250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdfb0034650 0x7fdfb0036b10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 --2- 172.21.15.110:0/1326365880 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5617922c4010 0x5617924423c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 >> 172.21.15.110:0/1326365880 conn(0x56179234aa50 msgr2=0x561792435ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 shutdown_connections 2024-04-27T03:18:13.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.549+0000 7fe0cdb4f980 1 -- 172.21.15.110:0/1326365880 wait complete. 2024-04-27T03:18:13.572 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=4 bytes_in=0 failed=False 2024-04-27T03:18:13.575 INFO:tasks.radosgw_admin:RL: get_obj myfoo foo bytes_out=0 bytes_in=4 failed=False 2024-04-27T03:18:13.619 INFO:tasks.radosgw_admin:RL: list_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:13.619 INFO:tasks.util.rgw:rgwadmin: client.0 : ['object', 'rm', '--bucket', 'myfoo', '--object', 'four'] 2024-04-27T03:18:13.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'object', 'rm', '--bucket', 'myfoo', '--object', 'four'] 2024-04-27T03:18:13.619 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph object rm --bucket myfoo --object four 2024-04-27T03:18:13.651 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:13.651 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa808aac980 1 Processor -- start 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa808aac980 1 -- start start 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa808aac980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ef4f5ab9d0 0x55ef4f5addc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa808aac980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55ef4f4640d0 con 0x55ef4f5a74c0 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa808aac980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55ef4f459430 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa805dc6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ef4f5ab9d0 0x55ef4f5addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa805dc6640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ef4f5ab9d0 0x55ef4f5addc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52506/0 (socket says 172.21.15.110:52506) 2024-04-27T03:18:13.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa805dc6640 1 -- 172.21.15.110:0/3084517750 learned_addr learned my addr 172.21.15.110:0/3084517750 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa805dc6640 1 -- 172.21.15.110:0/3084517750 >> v1:172.21.15.77:6789/0 conn(0x55ef4f5a74c0 legacy=0x55ef4f5a78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.645+0000 7fa805dc6640 1 -- 172.21.15.110:0/3084517750 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4f452510 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa805dc6640 1 --2- 172.21.15.110:0/3084517750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f5addc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800a4e0 tx=0x7fa7f801b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3c9bd5b74763c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa8055c5640 1 -- 172.21.15.110:0/3084517750 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f80aa540 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa8055c5640 1 -- 172.21.15.110:0/3084517750 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7f80aa6e0 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa8055c5640 1 -- 172.21.15.110:0/3084517750 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f80aaa40 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa808aac980 1 -- 172.21.15.110:0/3084517750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 msgr2=0x55ef4f5addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa808aac980 1 --2- 172.21.15.110:0/3084517750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f5addc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa7f800a4e0 tx=0x7fa7f801b090 comp rx=0 tx=0).stop 2024-04-27T03:18:13.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa808aac980 1 -- 172.21.15.110:0/3084517750 shutdown_connections 2024-04-27T03:18:13.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa808aac980 1 --2- 172.21.15.110:0/3084517750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f5addc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa808aac980 1 -- 172.21.15.110:0/3084517750 >> 172.21.15.110:0/3084517750 conn(0x55ef4f51aba0 msgr2=0x55ef4f51afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa808aac980 1 -- 172.21.15.110:0/3084517750 shutdown_connections 2024-04-27T03:18:13.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.649+0000 7fa808aac980 1 -- 172.21.15.110:0/3084517750 wait complete. 2024-04-27T03:18:13.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.661+0000 7fa808aac980 1 Processor -- start 2024-04-27T03:18:13.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.661+0000 7fa808aac980 1 -- start start 2024-04-27T03:18:13.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.661+0000 7fa808aac980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5a74c0 0x55ef4f6925e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.661+0000 7fa808aac980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef4f4a82d0 con 0x55ef4f5a74c0 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5a74c0 0x55ef4f6925e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5a74c0 0x55ef4f6925e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52508/0 (socket says 172.21.15.110:52508) 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 -- 172.21.15.110:0/2706272586 learned_addr learned my addr 172.21.15.110:0/2706272586 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 -- 172.21.15.110:0/2706272586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4f42eae0 con 0x55ef4f5a74c0 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 --2- 172.21.15.110:0/2706272586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5a74c0 0x55ef4f6925e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0001ea0 tx=0x7fa7f000b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab8bb1321221fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa6f4608640 1 -- 172.21.15.110:0/2706272586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f000a040 con 0x55ef4f5a74c0 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa6f4608640 1 -- 172.21.15.110:0/2706272586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7f000f620 con 0x55ef4f5a74c0 2024-04-27T03:18:13.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa6f4608640 1 -- 172.21.15.110:0/2706272586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f000f890 con 0x55ef4f5a74c0 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 -- 172.21.15.110:0/2706272586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5a74c0 msgr2=0x55ef4f6925e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 --2- 172.21.15.110:0/2706272586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5a74c0 0x55ef4f6925e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0001ea0 tx=0x7fa7f000b680 comp rx=0 tx=0).stop 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 -- 172.21.15.110:0/2706272586 shutdown_connections 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 --2- 172.21.15.110:0/2706272586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5a74c0 0x55ef4f6925e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 -- 172.21.15.110:0/2706272586 >> 172.21.15.110:0/2706272586 conn(0x55ef4f51aba0 msgr2=0x55ef4f68e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 -- 172.21.15.110:0/2706272586 shutdown_connections 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 -- 172.21.15.110:0/2706272586 wait complete. 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 Processor -- start 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 -- start start 2024-04-27T03:18:13.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f699880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa808aac980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef4f5b0df0 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f699880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f699880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52524/0 (socket says 172.21.15.110:52524) 2024-04-27T03:18:13.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 -- 172.21.15.110:0/3431971965 learned_addr learned my addr 172.21.15.110:0/3431971965 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4f699dc0 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.665+0000 7fa8065c7640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f699880 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0001fd0 tx=0x7fa7f0000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa6f2e05640 1 -- 172.21.15.110:0/3431971965 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f0001520 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa6f2e05640 1 -- 172.21.15.110:0/3431971965 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7f001c970 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ef4f69a050 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa6f2e05640 1 -- 172.21.15.110:0/3431971965 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7f001ccb0 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4f69ccd0 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa6f2e05640 1 -- 172.21.15.110:0/3431971965 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa7f0017050 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa6f2e05640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6ec034650 0x7fa6ec036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa6f2e05640 1 -- 172.21.15.110:0/3431971965 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fa7f0015020 con 0x55ef4f5ab9d0 2024-04-27T03:18:13.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa805dc6640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6ec034650 0x7fa6ec036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f69e910 0x55ef4f6a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6a1240 con 0x55ef4f69e910 2024-04-27T03:18:13.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa806dc8640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f69e910 0x55ef4f6a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa806dc8640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f69e910 0x55ef4f6a0d00 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa805dc6640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6ec034650 0x7fa6ec036b10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80ab000 tx=0x7fa7f80a5000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa806dc8640 1 -- 172.21.15.110:0/3431971965 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa8000083f0 con 0x55ef4f69e910 2024-04-27T03:18:13.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6a1fa0 0x55ef4f6a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6a49c0 con 0x55ef4f6a1fa0 2024-04-27T03:18:13.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa8065c7640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6a1fa0 0x55ef4f6a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.669+0000 7fa8065c7640 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6a1fa0 0x55ef4f6a4390 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa8065c7640 1 -- 172.21.15.110:0/3431971965 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fa7f002e070 con 0x55ef4f6a1fa0 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6a6270 con 0x55ef4f69e910 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa806dc8640 1 -- 172.21.15.110:0/3431971965 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fa8000083f0 con 0x55ef4f69e910 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6a54a0 con 0x55ef4f6a1fa0 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa8065c7640 1 -- 172.21.15.110:0/3431971965 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fa7f002e070 con 0x55ef4f6a1fa0 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6a5890 con 0x55ef4f69e910 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa806dc8640 1 -- 172.21.15.110:0/3431971965 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fa8000083f0 con 0x55ef4f69e910 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa808aac980 1 Processor -- start 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa808aac980 1 -- start start 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa808aac980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6b4b70 0x55ef4f6b4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa808aac980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef4f4ac310 con 0x55ef4f6b4b70 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa806dc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6b4b70 0x55ef4f6b4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa806dc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6b4b70 0x55ef4f6b4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52526/0 (socket says 172.21.15.110:52526) 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa806dc8640 1 -- 172.21.15.110:0/2386563724 learned_addr learned my addr 172.21.15.110:0/2386563724 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa806dc8640 1 -- 172.21.15.110:0/2386563724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4f6b54d0 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa806dc8640 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6b4b70 0x55ef4f6b4f90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa80000aab0 tx=0x7fa80000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.673+0000 7fa6ebfff640 1 -- 172.21.15.110:0/2386563724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800007d30 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6ebfff640 1 -- 172.21.15.110:0/2386563724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa80001d070 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ef4f6b5f70 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6ebfff640 1 -- 172.21.15.110:0/2386563724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800018440 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4f6b6440 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6ebfff640 1 -- 172.21.15.110:0/2386563724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa800013070 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6ebfff640 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6e0034650 0x7fa6e0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6ebfff640 1 -- 172.21.15.110:0/2386563724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fa80004b5e0 con 0x55ef4f6b4b70 2024-04-27T03:18:13.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa8065c7640 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6e0034650 0x7fa6e0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa8065c7640 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6e0034650 0x7fa6e0036b10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55ef4f6a4fc0 tx=0x7fa7f0001280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa808aac980 1 Processor -- start 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa808aac980 1 -- start start 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa808aac980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa808aac980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef4f692d30 con 0x55ef4f6c1d20 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa806dc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa806dc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52542/0 (socket says 172.21.15.110:52542) 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa806dc8640 1 -- 172.21.15.110:0/644974842 learned_addr learned my addr 172.21.15.110:0/644974842 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa806dc8640 1 -- 172.21.15.110:0/644974842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4f6c2710 con 0x55ef4f6c1d20 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa806dc8640 1 --2- 172.21.15.110:0/644974842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c2140 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa800049930 tx=0x7fa800049b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09066c80e5963d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6e9ffb640 1 -- 172.21.15.110:0/644974842 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800047070 con 0x55ef4f6c1d20 2024-04-27T03:18:13.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6e9ffb640 1 -- 172.21.15.110:0/644974842 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa800022460 con 0x55ef4f6c1d20 2024-04-27T03:18:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.677+0000 7fa6e9ffb640 1 -- 172.21.15.110:0/644974842 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800018bc0 con 0x55ef4f6c1d20 2024-04-27T03:18:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- 172.21.15.110:0/644974842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 msgr2=0x55ef4f6c2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 --2- 172.21.15.110:0/644974842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c2140 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa800049930 tx=0x7fa800049b30 comp rx=0 tx=0).stop 2024-04-27T03:18:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- 172.21.15.110:0/644974842 shutdown_connections 2024-04-27T03:18:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 --2- 172.21.15.110:0/644974842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c2140 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- 172.21.15.110:0/644974842 >> 172.21.15.110:0/644974842 conn(0x55ef4f6be060 msgr2=0x55ef4f6bc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- 172.21.15.110:0/644974842 shutdown_connections 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- 172.21.15.110:0/644974842 wait complete. 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 Processor -- start 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- start start 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ef4f6c2e00 con 0x55ef4f6c1d20 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa806dc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa806dc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52556/0 (socket says 172.21.15.110:52556) 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 learned_addr learned my addr 172.21.15.110:0/674200367 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef4f6c9700 con 0x55ef4f6c1d20 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa806dc8640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c91c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x55ef4f6be780 tx=0x7fa8000498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa6d7fff640 1 -- 172.21.15.110:0/674200367 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800031070 con 0x55ef4f6c1d20 2024-04-27T03:18:13.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa6d7fff640 1 -- 172.21.15.110:0/674200367 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa800016d60 con 0x55ef4f6c1d20 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ef4f6c9990 con 0x55ef4f6c1d20 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa6d7fff640 1 -- 172.21.15.110:0/674200367 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa800020840 con 0x55ef4f6c1d20 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef4f6cc670 con 0x55ef4f6c1d20 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa6d7fff640 1 -- 172.21.15.110:0/674200367 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa800034070 con 0x55ef4f6c1d20 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.681+0000 7fa6d7fff640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6d0034650 0x7fa6d0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa6d7fff640 1 -- 172.21.15.110:0/674200367 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fa800037c90 con 0x55ef4f6c1d20 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa8065c7640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6d0034650 0x7fa6d0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f6d4860 0x55ef4f6d6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6d7450 con 0x55ef4f6d4860 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa805dc6640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f6d4860 0x55ef4f6d6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa8065c7640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6d0034650 0x7fa6d0036b10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa7f000c010 tx=0x7fa7f0037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa805dc6640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f6d4860 0x55ef4f6d6d00 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 realm 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6d7820 con 0x55ef4f6d4860 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6d80f0 con 0x55ef4f6d4860 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6d8b10 0x55ef4f6daf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6db6b0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa806dc8640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6d8b10 0x55ef4f6daf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa806dc8640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6d8b10 0x55ef4f6daf40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6dbf50 con 0x55ef4f6d4860 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 20 searching for the correct realm 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ef4f6ddcc0 0x55ef4f6e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa8065c7640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ef4f6ddcc0 0x55ef4f6e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.685+0000 7fa8065c7640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ef4f6ddcc0 0x55ef4f6e00b0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa806dc8640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa800036070 0x7fa8000218a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x7fa800036070 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa800036070 0x7fa8000218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa800036070 0x7fa8000218a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.689+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x7fa800036070 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x7fa800036070 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.693+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x7fa800036070 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d4860 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x7fa800036070 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x7fa800036070 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.697+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e06c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6dc8d0 con 0x55ef4f6d4860 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6dd170 con 0x55ef4f6d8b10 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6dc510 con 0x55ef4f6d4860 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 zone default found 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 4 Realm: () 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:13.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 zonegroup default 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e0ff0 con 0x7fa800036070 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e32a0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e3640 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 started sync module instance, tier type = 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:13.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e26e0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94486318164688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e2ef0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.701+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e44b0 con 0x55ef4f6d4860 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94486318170384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e4ba0 con 0x55ef4f6d4860 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e56c0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94486318164688] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94486318170384] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94486318174608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e5680 con 0x55ef4f6d8b10 2024-04-27T03:18:13.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e5f60 con 0x7fa800036070 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94486318176944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e5680 con 0x7fa800036070 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.705+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e6a10 con 0x55ef4f6d8b10 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94486318174608] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94486318176944] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94486318178000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e6a10 con 0x55ef4f6d8b10 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e6e00 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94486318168544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e7650 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e8050 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.709+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94486318178000] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94486318168544] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94486318179616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e71a0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e88f0 con 0x7fa800036070 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94486318179616] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'144 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94486318185312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e71a0 con 0x7fa800036070 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=0 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=1 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=2 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=3 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=4 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=5 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=6 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94486318185312] v27'145 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 20 add_watcher() i=7 2024-04-27T03:18:13.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6eb980 con 0x55ef4f6d4860 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.713+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94486318187584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e8e00 con 0x55ef4f6d4860 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.717+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94486318187584] v27'145 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.717+0000 7fa6bcfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.717+0000 7fa808aac980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.717+0000 7fa808aac980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:13.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.717+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f4430 con 0x55ef4f6d8b10 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.717+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'47 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.717+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f4530 con 0x7fa800036070 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.721+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'24 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.721+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f4c50 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.721+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'24 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.721+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f4c50 con 0x55ef4f6d4860 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.721+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'146 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.721+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f51e0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.725+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'49 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.725+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f3e90 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.725+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'39 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.725+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5690 con 0x55ef4f6d8b10 2024-04-27T03:18:13.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.729+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'25 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.729+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5690 con 0x55ef4f6d4860 2024-04-27T03:18:13.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.729+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'34 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.729+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5c00 con 0x55ef4f6d8b10 2024-04-27T03:18:13.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.733+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'17 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.733+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5c00 con 0x55ef4f6d8b10 2024-04-27T03:18:13.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.733+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'23 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.733+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5c00 con 0x55ef4f6d8b10 2024-04-27T03:18:13.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.737+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'50 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.737+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5c00 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.737+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'19 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.737+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5fe0 con 0x55ef4f6d8b10 2024-04-27T03:18:13.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.737+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'23 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.737+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5690 con 0x55ef4f6d4860 2024-04-27T03:18:13.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.741+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'45 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.741+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5860 con 0x55ef4f6d4860 2024-04-27T03:18:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.741+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'147 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.741+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5860 con 0x55ef4f6d8b10 2024-04-27T03:18:13.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.745+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'41 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.745+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d8b10 2024-04-27T03:18:13.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.745+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'51 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.745+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.749+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'30 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.749+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d8b10 2024-04-27T03:18:13.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.749+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'25 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.749+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d4860 2024-04-27T03:18:13.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.753+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'148 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.753+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d8b10 2024-04-27T03:18:13.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.753+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'42 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.753+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d8b10 2024-04-27T03:18:13.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.757+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'33 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.757+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.757+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'25 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.757+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d8b10 2024-04-27T03:18:13.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.761+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'35 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.761+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.761+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'19 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.761+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.761+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'40 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.761+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.765+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'25 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.765+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d8b10 2024-04-27T03:18:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.769+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'29 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.769+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d4860 2024-04-27T03:18:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.769+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'35 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.769+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d8b10 2024-04-27T03:18:13.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.773+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'43 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.773+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x55ef4f6d4860 2024-04-27T03:18:13.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.773+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'46 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.773+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6f5a30 con 0x7fa800036070 2024-04-27T03:18:13.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'23 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa6377fe640 20 reqs_thread_entry: start 2024-04-27T03:18:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 Filter name: none 2024-04-27T03:18:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa6357fa640 20 reqs_thread_entry: start 2024-04-27T03:18:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f7113e0 con 0x7fa800036070 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(94 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv27 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:95 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f712c80 con 0x7fa800036070 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(95 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv7 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 get_obj_state: octx=0x7fa7f80aaa70 obj=myfoo:four state=0x55ef4f7139f8 s->prefetch_data=0 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:96 3.18s0 3:1b05c7ba:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1_four:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f714c70 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(96 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1_four [getxattrs out=851b,stat out=16b] v0'0 uv1 ondisk = 0) ==== 234+0+867 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 10 manifest: total_size = 4 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.14337541202213412430 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 get_obj_state: octx=0x7fa7f80aaa70 obj=myfoo:four state=0x55ef4f7139f8 s->prefetch_data=0 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 prepare_atomic_modification: state is not atomic. state=0x55ef4f7139f8 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 2024-04-27T03:18:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.777+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:97 4.8 4:10a25087:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=182b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f7196b0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.781+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(97 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 [stat,call,call] v27'4 uv4 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.781+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:98 3.18s0 3:1b05c7ba:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1_four:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f7196b0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.781+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(98 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1_four [call] v27'2 uv2 ondisk = 0) ==== 192+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.781+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:99 4.8 4:10a25087:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=259b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f71a140 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.781+0000 7fa6bcfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.781+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:100 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94486318187584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f71a140 con 0x55ef4f6d4860 2024-04-27T03:18:13.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(99 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 [stat,call,call] v27'5 uv5 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa6e9ffb640 20 handle_completion(): completion ok for obj=four 2024-04-27T03:18:13.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(100 data_loggenerations_metadata [watch unwatch cookie 94486318187584] v27'149 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94486318164688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f714b70 con 0x55ef4f6d8b10 2024-04-27T03:18:13.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(101 notify.0 [watch unwatch cookie 94486318164688] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa808aac980 20 remove_watcher() i=0 2024-04-27T03:18:13.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa808aac980 2 removed watcher, disabling cache 2024-04-27T03:18:13.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.785+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:102 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94486318170384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e2ad0 con 0x55ef4f6d4860 2024-04-27T03:18:13.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(102 notify.1 [watch unwatch cookie 94486318170384] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x55ef4f6d4860 2024-04-27T03:18:13.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa808aac980 20 remove_watcher() i=1 2024-04-27T03:18:13.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:103 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94486318174608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e4110 con 0x55ef4f6d8b10 2024-04-27T03:18:13.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(103 notify.2 [watch unwatch cookie 94486318174608] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa808aac980 20 remove_watcher() i=2 2024-04-27T03:18:13.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:104 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94486318176944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e2ad0 con 0x7fa800036070 2024-04-27T03:18:13.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(104 notify.3 [watch unwatch cookie 94486318176944] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa808aac980 20 remove_watcher() i=3 2024-04-27T03:18:13.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.789+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:105 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94486318178000] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e4110 con 0x55ef4f6d8b10 2024-04-27T03:18:13.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa806dc8640 1 -- 172.21.15.110:0/674200367 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(105 notify.4 [watch unwatch cookie 94486318178000] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa800033030 con 0x55ef4f6d8b10 2024-04-27T03:18:13.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa808aac980 20 remove_watcher() i=4 2024-04-27T03:18:13.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:106 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94486318168544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e2ad0 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(106 notify.5 [watch unwatch cookie 94486318168544] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa808aac980 20 remove_watcher() i=5 2024-04-27T03:18:13.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:107 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94486318179616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e4110 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa8065c7640 1 -- 172.21.15.110:0/674200367 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(107 notify.6 [watch unwatch cookie 94486318179616] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f004b030 con 0x55ef4f6ddcc0 2024-04-27T03:18:13.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa808aac980 20 remove_watcher() i=6 2024-04-27T03:18:13.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.793+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:108 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94486318185312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55ef4f6e2ad0 con 0x7fa800036070 2024-04-27T03:18:13.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(108 notify.7 [watch unwatch cookie 94486318185312] v27'146 uv144 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa7f80aaaf0 con 0x7fa800036070 2024-04-27T03:18:13.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 20 remove_watcher() i=7 2024-04-27T03:18:13.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6d8b10 msgr2=0x55ef4f6daf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6d8b10 0x55ef4f6daf40 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f6d4860 msgr2=0x55ef4f6d6d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f6d4860 0x55ef4f6d6d00 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa800036070 msgr2=0x7fa8000218a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa800036070 0x7fa8000218a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ef4f6ddcc0 msgr2=0x55ef4f6e00b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ef4f6ddcc0 0x55ef4f6e00b0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6d0034650 msgr2=0x7fa6d0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6d0034650 0x7fa6d0036b10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa7f000c010 tx=0x7fa7f0037000 comp rx=0 tx=0).stop 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 msgr2=0x55ef4f6c91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/674200367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6c1d20 0x55ef4f6c91c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x55ef4f6be780 tx=0x7fa8000498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 reap_dead start 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa805dc6640 1 -- 172.21.15.110:0/674200367 reap_dead start 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 shutdown_connections 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 >> 172.21.15.110:0/674200367 conn(0x55ef4f6be060 msgr2=0x55ef4f6bb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 shutdown_connections 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/674200367 wait complete. 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6e0034650 msgr2=0x7fa6e0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6e0034650 0x7fa6e0036b10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55ef4f6a4fc0 tx=0x7fa7f0001280 comp rx=0 tx=0).stop 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6b4b70 msgr2=0x55ef4f6b4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6b4b70 0x55ef4f6b4f90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa80000aab0 tx=0x7fa80000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 shutdown_connections 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6e0034650 0x7fa6e0036b10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/2386563724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f6b4b70 0x55ef4f6b4f90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 >> 172.21.15.110:0/2386563724 conn(0x55ef4f6a9980 msgr2=0x55ef4f6aa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 shutdown_connections 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/2386563724 wait complete. 2024-04-27T03:18:13.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6a1fa0 msgr2=0x55ef4f6a4390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6a1fa0 0x55ef4f6a4390 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f69e910 msgr2=0x55ef4f6a0d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f69e910 0x55ef4f6a0d00 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6ec034650 msgr2=0x7fa6ec036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6ec034650 0x7fa6ec036b10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80ab000 tx=0x7fa7f80a5000 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 msgr2=0x55ef4f699880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f699880 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0001fd0 tx=0x7fa7f0000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 shutdown_connections 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ef4f6a1fa0 0x55ef4f6a4390 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ef4f69e910 0x55ef4f6a0d00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa6ec034650 0x7fa6ec036b10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 --2- 172.21.15.110:0/3431971965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ef4f5ab9d0 0x55ef4f699880 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 >> 172.21.15.110:0/3431971965 conn(0x55ef4f51aba0 msgr2=0x55ef4f68cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 shutdown_connections 2024-04-27T03:18:13.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.797+0000 7fa808aac980 1 -- 172.21.15.110:0/3431971965 wait complete. 2024-04-27T03:18:13.814 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'stats', '--bucket', 'myfoo'] 2024-04-27T03:18:13.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'stats', '--bucket', 'myfoo'] 2024-04-27T03:18:13.815 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket stats --bucket myfoo 2024-04-27T03:18:13.893 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:13.893 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:13.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc3ab66980 1 Processor -- start 2024-04-27T03:18:13.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc3ab66980 1 -- start start 2024-04-27T03:18:13.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc3ab66980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x563e9f3ac9b0 0x563e9f3aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc3ab66980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x563e9f2650d0 con 0x563e9f3a8490 2024-04-27T03:18:13.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc3ab66980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x563e9f25a430 con 0x563e9f3ac9b0 2024-04-27T03:18:13.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc37e80640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x563e9f3ac9b0 0x563e9f3aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc38681640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x563e9f3a8490 0x563e9f3a8870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:50698/0 (socket says 172.21.15.110:50698) 2024-04-27T03:18:13.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.885+0000 7fcc38681640 1 -- 172.21.15.110:0/3292432822 learned_addr learned my addr 172.21.15.110:0/3292432822 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:13.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc37e80640 1 -- 172.21.15.110:0/3292432822 >> v1:172.21.15.77:6789/0 conn(0x563e9f3a8490 legacy=0x563e9f3a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc37e80640 1 -- 172.21.15.110:0/3292432822 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e9f253510 con 0x563e9f3ac9b0 2024-04-27T03:18:13.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc37e80640 1 --2- 172.21.15.110:0/3292432822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3ac9b0 0x563e9f3aeda0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00a480 tx=0x7fcc2c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cf150c0da3f4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3767f640 1 -- 172.21.15.110:0/3292432822 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc2c0b0070 con 0x563e9f3ac9b0 2024-04-27T03:18:13.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3767f640 1 -- 172.21.15.110:0/3292432822 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc2c0ab540 con 0x563e9f3ac9b0 2024-04-27T03:18:13.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3767f640 1 -- 172.21.15.110:0/3292432822 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc2c0ab840 con 0x563e9f3ac9b0 2024-04-27T03:18:13.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3ab66980 1 -- 172.21.15.110:0/3292432822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3ac9b0 msgr2=0x563e9f3aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/3292432822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3ac9b0 0x563e9f3aeda0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c00a480 tx=0x7fcc2c01b280 comp rx=0 tx=0).stop 2024-04-27T03:18:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3ab66980 1 -- 172.21.15.110:0/3292432822 shutdown_connections 2024-04-27T03:18:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/3292432822 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3ac9b0 0x563e9f3aeda0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3ab66980 1 -- 172.21.15.110:0/3292432822 >> 172.21.15.110:0/3292432822 conn(0x563e9f31bb90 msgr2=0x563e9f31bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3ab66980 1 -- 172.21.15.110:0/3292432822 shutdown_connections 2024-04-27T03:18:13.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.889+0000 7fcc3ab66980 1 -- 172.21.15.110:0/3292432822 wait complete. 2024-04-27T03:18:13.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 Processor -- start 2024-04-27T03:18:13.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- start start 2024-04-27T03:18:13.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f4935e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e9f2a92d0 con 0x563e9f3a8490 2024-04-27T03:18:13.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f4935e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f4935e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:44980/0 (socket says 172.21.15.110:44980) 2024-04-27T03:18:13.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 -- 172.21.15.110:0/2064062379 learned_addr learned my addr 172.21.15.110:0/2064062379 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 -- 172.21.15.110:0/2064062379 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e9f22fae0 con 0x563e9f3a8490 2024-04-27T03:18:13.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 --2- 172.21.15.110:0/2064062379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f4935e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcc300015f0 tx=0x7fcc30001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d10f04d11de84f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcb2660c640 1 -- 172.21.15.110:0/2064062379 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc30004530 con 0x563e9f3a8490 2024-04-27T03:18:13.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcb2660c640 1 -- 172.21.15.110:0/2064062379 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc300046f0 con 0x563e9f3a8490 2024-04-27T03:18:13.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcb2660c640 1 -- 172.21.15.110:0/2064062379 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc30004a90 con 0x563e9f3a8490 2024-04-27T03:18:13.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2064062379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 msgr2=0x563e9f4935e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2064062379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f4935e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcc300015f0 tx=0x7fcc30001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2064062379 shutdown_connections 2024-04-27T03:18:13.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2064062379 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f4935e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2064062379 >> 172.21.15.110:0/2064062379 conn(0x563e9f31bb90 msgr2=0x563e9f48f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2064062379 shutdown_connections 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2064062379 wait complete. 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 Processor -- start 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- start start 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f49a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc3ab66980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e9f3b1e00 con 0x563e9f3a8490 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f49a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f49a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:44990/0 (socket says 172.21.15.110:44990) 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.905+0000 7fcc38681640 1 -- 172.21.15.110:0/2870041623 learned_addr learned my addr 172.21.15.110:0/2870041623 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc38681640 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e9f49ab60 con 0x563e9f3a8490 2024-04-27T03:18:13.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc38681640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f49a620 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x563e9f494470 tx=0x7fcc30000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcb24e09640 1 -- 172.21.15.110:0/2870041623 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc30013760 con 0x563e9f3a8490 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563e9f49adf0 con 0x563e9f3a8490 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcb24e09640 1 -- 172.21.15.110:0/2870041623 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc30013900 con 0x563e9f3a8490 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcb24e09640 1 -- 172.21.15.110:0/2870041623 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc30013be0 con 0x563e9f3a8490 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e9f49da70 con 0x563e9f3a8490 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcb24e09640 1 -- 172.21.15.110:0/2870041623 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fcc30010070 con 0x563e9f3a8490 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcb24e09640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb20034650 0x7fcb20036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcb24e09640 1 -- 172.21.15.110:0/2870041623 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fcc30018070 con 0x563e9f3a8490 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc37e80640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb20034650 0x7fcb20036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f3ac9b0 0x563e9f4a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4a1b70 con 0x563e9f3ac9b0 2024-04-27T03:18:13.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc38e82640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f3ac9b0 0x563e9f4a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc37e80640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb20034650 0x7fcb20036b10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0ad000 tx=0x7fcc2c0a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc38e82640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f3ac9b0 0x563e9f4a1550 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc38e82640 1 -- 172.21.15.110:0/2870041623 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fcc280083f0 con 0x563e9f3ac9b0 2024-04-27T03:18:13.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4a2960 0x563e9f4a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4a5390 con 0x563e9f4a2960 2024-04-27T03:18:13.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.909+0000 7fcc38681640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4a2960 0x563e9f4a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38681640 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4a2960 0x563e9f4a4d50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38681640 1 -- 172.21.15.110:0/2870041623 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fcc3001b030 con 0x563e9f4a2960 2024-04-27T03:18:13.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4a6c40 con 0x563e9f3ac9b0 2024-04-27T03:18:13.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38e82640 1 -- 172.21.15.110:0/2870041623 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fcc280083f0 con 0x563e9f3ac9b0 2024-04-27T03:18:13.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4a5e70 con 0x563e9f4a2960 2024-04-27T03:18:13.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38681640 1 -- 172.21.15.110:0/2870041623 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fcc3001b030 con 0x563e9f4a2960 2024-04-27T03:18:13.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4a6260 con 0x563e9f3ac9b0 2024-04-27T03:18:13.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38e82640 1 -- 172.21.15.110:0/2870041623 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fcc280083f0 con 0x563e9f3ac9b0 2024-04-27T03:18:13.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc3ab66980 1 Processor -- start 2024-04-27T03:18:13.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc3ab66980 1 -- start start 2024-04-27T03:18:13.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc3ab66980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4b5540 0x563e9f4b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc3ab66980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e9f2ad310 con 0x563e9f4b5540 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38e82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4b5540 0x563e9f4b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38e82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4b5540 0x563e9f4b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45004/0 (socket says 172.21.15.110:45004) 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38e82640 1 -- 172.21.15.110:0/4232333430 learned_addr learned my addr 172.21.15.110:0/4232333430 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.913+0000 7fcc38e82640 1 -- 172.21.15.110:0/4232333430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e9f4b5ea0 con 0x563e9f4b5540 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38e82640 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4b5540 0x563e9f4b5960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800aab0 tx=0x7fcc2800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb1dffb640 1 -- 172.21.15.110:0/4232333430 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28007d30 con 0x563e9f4b5540 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563e9f4b6940 con 0x563e9f4b5540 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e9f4b6e10 con 0x563e9f4b5540 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb1dffb640 1 -- 172.21.15.110:0/4232333430 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc2801d070 con 0x563e9f4b5540 2024-04-27T03:18:13.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb1dffb640 1 -- 172.21.15.110:0/4232333430 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28022460 con 0x563e9f4b5540 2024-04-27T03:18:13.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb1dffb640 1 -- 172.21.15.110:0/4232333430 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fcc28013070 con 0x563e9f4b5540 2024-04-27T03:18:13.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb1dffb640 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb10034650 0x7fcb10036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb1dffb640 1 -- 172.21.15.110:0/4232333430 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fcc2804b500 con 0x563e9f4b5540 2024-04-27T03:18:13.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38681640 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb10034650 0x7fcb10036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc3ab66980 1 Processor -- start 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc3ab66980 1 -- start start 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38681640 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb10034650 0x7fcb10036b10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x563e9f4a5990 tx=0x7fcc30010440 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc3ab66980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc3ab66980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e9f493d30 con 0x563e9f4c2740 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38e82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38e82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45018/0 (socket says 172.21.15.110:45018) 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38e82640 1 -- 172.21.15.110:0/2076869090 learned_addr learned my addr 172.21.15.110:0/2076869090 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38e82640 1 -- 172.21.15.110:0/2076869090 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e9f4c3130 con 0x563e9f4c2740 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcc38e82640 1 --2- 172.21.15.110:0/2076869090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c2b60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcc280204a0 tx=0x7fcc280205e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e07a7aff906df83 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb177fe640 1 -- 172.21.15.110:0/2076869090 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28047070 con 0x563e9f4c2740 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb177fe640 1 -- 172.21.15.110:0/2076869090 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc28020840 con 0x563e9f4c2740 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.917+0000 7fcb177fe640 1 -- 172.21.15.110:0/2076869090 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28021e00 con 0x563e9f4c2740 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2076869090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 msgr2=0x563e9f4c2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2076869090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c2b60 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcc280204a0 tx=0x7fcc280205e0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2076869090 shutdown_connections 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2076869090 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c2b60 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2076869090 >> 172.21.15.110:0/2076869090 conn(0x563e9f4bea80 msgr2=0x563e9f4bcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2076869090 shutdown_connections 2024-04-27T03:18:13.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2076869090 wait complete. 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 Processor -- start 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- start start 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x563e9f4c3840 con 0x563e9f4c2740 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc38e82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc38e82640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45022/0 (socket says 172.21.15.110:45022) 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 learned_addr learned my addr 172.21.15.110:0/752268344 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e9f4ca110 con 0x563e9f4c2740 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc38e82640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c9bd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcc28016590 tx=0x7fcc280498a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcb15ffb640 1 -- 172.21.15.110:0/752268344 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28030070 con 0x563e9f4c2740 2024-04-27T03:18:13.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563e9f4ca3a0 con 0x563e9f4c2740 2024-04-27T03:18:13.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcb15ffb640 1 -- 172.21.15.110:0/752268344 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc28022ad0 con 0x563e9f4c2740 2024-04-27T03:18:13.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcb15ffb640 1 -- 172.21.15.110:0/752268344 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc28021530 con 0x563e9f4c2740 2024-04-27T03:18:13.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e9f4cd080 con 0x563e9f4c2740 2024-04-27T03:18:13.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcb15ffb640 1 -- 172.21.15.110:0/752268344 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fcc28033070 con 0x563e9f4c2740 2024-04-27T03:18:13.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcb15ffb640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb08034600 0x7fcb08036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.921+0000 7fcb15ffb640 1 -- 172.21.15.110:0/752268344 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fcc2801a870 con 0x563e9f4c2740 2024-04-27T03:18:13.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc38681640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb08034600 0x7fcb08036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f4d5250 0x563e9f4d76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4d7e40 con 0x563e9f4d5250 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc37e80640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f4d5250 0x563e9f4d76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc38681640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb08034600 0x7fcb08036ac0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcc30048440 tx=0x7fcc30049000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc37e80640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f4d5250 0x563e9f4d76f0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 realm 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4d8270 con 0x563e9f4d5250 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4d8b40 con 0x563e9f4d5250 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4d9560 0x563e9f4db950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4dc0c0 con 0x563e9f4d9560 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc38e82640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4d9560 0x563e9f4db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc38e82640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4d9560 0x563e9f4db950 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4dc960 con 0x563e9f4d5250 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 20 searching for the correct realm 2024-04-27T03:18:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563e9f4de6d0 0x563e9f4e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4de6d0 2024-04-27T03:18:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc38681640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563e9f4de6d0 0x563e9f4e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.925+0000 7fcc38681640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563e9f4de6d0 0x563e9f4e0ac0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38e82640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fcc28035070 0x7fcc2803b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x7fcc28035070 2024-04-27T03:18:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fcc28035070 0x7fcc2803b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fcc28035070 0x7fcc2803b820 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:13.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4de6d0 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.929+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x7fcc28035070 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x7fcc28035070 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4de6d0 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4de6d0 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.933+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4de6d0 2024-04-27T03:18:13.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x7fcc28035070 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d5250 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x7fcc28035070 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4de6d0 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x7fcc28035070 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4de6d0 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.937+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e10a0 con 0x563e9f4d9560 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:13.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4dd2e0 con 0x563e9f4d5250 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4dd760 con 0x563e9f4d9560 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4ddbc0 con 0x563e9f4d5250 2024-04-27T03:18:13.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 zone default found 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 4 Realm: () 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 zonegroup default 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e19d0 con 0x7fcc28035070 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e33b0 con 0x563e9f4de6d0 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e3870 con 0x563e9f4de6d0 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 started sync module instance, tier type = 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.941+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e2840 con 0x563e9f4d9560 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94826960661552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e24a0 con 0x563e9f4d9560 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e47a0 con 0x563e9f4d5250 2024-04-27T03:18:13.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94826960667376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e4710 con 0x563e9f4d5250 2024-04-27T03:18:13.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e5700 con 0x563e9f4d9560 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94826960661552] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94826960667376] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94826960669584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e5700 con 0x563e9f4d9560 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.945+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e5b30 con 0x7fcc28035070 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94826960671200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e5af0 con 0x7fcc28035070 2024-04-27T03:18:13.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e69f0 con 0x563e9f4d9560 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94826960669584] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94826960671200] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94826960674592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e6540 con 0x563e9f4d9560 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e7290 con 0x563e9f4de6d0 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.949+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94826960678368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e6540 con 0x563e9f4de6d0 2024-04-27T03:18:13.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e8010 con 0x563e9f4de6d0 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94826960681696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e7fd0 con 0x563e9f4de6d0 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e8fd0 con 0x7fcc28035070 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94826960678368] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94826960674592] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'147 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94826960665808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e8fd0 con 0x7fcc28035070 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=0 2024-04-27T03:18:13.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=1 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=2 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=3 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=4 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=5 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94826960681696] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=6 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94826960665808] v27'148 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 20 add_watcher() i=7 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:13.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e8400 con 0x563e9f4d5250 2024-04-27T03:18:13.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.953+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94826960685824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f4e91f0 con 0x563e9f4d5250 2024-04-27T03:18:13.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94826960685824] v27'150 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcaee7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 5 note: GC not initialized 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcaed7e2640 20 reqs_thread_entry: start 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 Filter name: none 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fca5effd640 20 reqs_thread_entry: start 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50f0a0 con 0x7fcc28035070 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv27 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50f9d0 con 0x7fcc28035070 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv7 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:13.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50fa10 con 0x7fcc28035070 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(64 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv27 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados->read ofs=0 len=0 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f5110b0 con 0x7fcc28035070 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(65 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv7 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 20 couldn't put binfo cache entry, might have raced with data changes 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:66 4.11 4:8a36b39c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f510790 con 0x563e9f4de6d0 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 4.39 4:9c046854:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f515c60 con 0x563e9f4de6d0 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 4.15 4:aaf269a6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f5184d0 con 0x563e9f4de6d0 2024-04-27T03:18:13.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 4.24 4:241b82a3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f51ad40 con 0x563e9f4d5250 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 4.e 4:709f8441:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f51d710 con 0x7fcc28035070 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.957+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 4.3d 4:bc3f4761:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f51d710 con 0x563e9f4d9560 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 4.8 4:10a25087:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f51ad40 con 0x563e9f4de6d0 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(66 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 4.1 4:836fd4f5:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f5252c0 con 0x7fcc28035070 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(68 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fcc30050040 con 0x563e9f4de6d0 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:74 4.2e 4:773ce9c4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f51ad40 con 0x563e9f4de6d0 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fcc2c0b3040 con 0x563e9f4d5250 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:75 4.1 4:80e658b2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f518130 con 0x7fcc28035070 2024-04-27T03:18:13.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(70 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 4.b 4:d2da04c3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x563e9f5147a0 con 0x7fcc28035070 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(73 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fcc2c0b3040 con 0x7fcc28035070 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(72 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.6 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7fcc30065090 con 0x563e9f4de6d0 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(74 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.8 [call out=124b] v0'0 uv7 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7fcc30065090 con 0x563e9f4de6d0 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(71 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.5 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(76 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7fcc2c0ec060 con 0x7fcc28035070 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(75 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.1.9 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fcc2c0ec060 con 0x7fcc28035070 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcaee7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:13.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:77 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94826960685824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x563e9f4d5250 2024-04-27T03:18:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(77 data_loggenerations_metadata [watch unwatch cookie 94826960685824] v27'151 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fcc2c0ec060 con 0x563e9f4d5250 2024-04-27T03:18:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.961+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94826960661552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x563e9f4d9560 2024-04-27T03:18:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.965+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(78 notify.0 [watch unwatch cookie 94826960661552] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.965+0000 7fcc3ab66980 20 remove_watcher() i=0 2024-04-27T03:18:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.965+0000 7fcc3ab66980 2 removed watcher, disabling cache 2024-04-27T03:18:13.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.965+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94826960667376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x563e9f4d5250 2024-04-27T03:18:13.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.965+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(79 notify.1 [watch unwatch cookie 94826960667376] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0ec060 con 0x563e9f4d5250 2024-04-27T03:18:13.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.965+0000 7fcc3ab66980 20 remove_watcher() i=1 2024-04-27T03:18:13.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.965+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94826960669584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x563e9f4d9560 2024-04-27T03:18:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(80 notify.2 [watch unwatch cookie 94826960669584] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc3ab66980 20 remove_watcher() i=2 2024-04-27T03:18:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:81 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94826960671200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x7fcc28035070 2024-04-27T03:18:13.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(81 notify.3 [watch unwatch cookie 94826960671200] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0ec060 con 0x7fcc28035070 2024-04-27T03:18:13.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc3ab66980 20 remove_watcher() i=3 2024-04-27T03:18:13.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94826960674592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x563e9f4d9560 2024-04-27T03:18:13.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc38e82640 1 -- 172.21.15.110:0/752268344 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(82 notify.4 [watch unwatch cookie 94826960674592] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc28032030 con 0x563e9f4d9560 2024-04-27T03:18:13.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc3ab66980 20 remove_watcher() i=4 2024-04-27T03:18:13.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.969+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:83 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94826960678368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x563e9f4de6d0 2024-04-27T03:18:13.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(83 notify.5 [watch unwatch cookie 94826960678368] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc30065090 con 0x563e9f4de6d0 2024-04-27T03:18:13.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 20 remove_watcher() i=5 2024-04-27T03:18:13.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94826960681696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x563e9f4de6d0 2024-04-27T03:18:13.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc38681640 1 -- 172.21.15.110:0/752268344 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(84 notify.6 [watch unwatch cookie 94826960681696] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc30065090 con 0x563e9f4de6d0 2024-04-27T03:18:13.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 20 remove_watcher() i=6 2024-04-27T03:18:13.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:85 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94826960665808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x563e9f50ca60 con 0x7fcc28035070 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(85 notify.7 [watch unwatch cookie 94826960665808] v27'149 uv147 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fcc2c0ec060 con 0x7fcc28035070 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 20 remove_watcher() i=7 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4d9560 msgr2=0x563e9f4db950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4d9560 0x563e9f4db950 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f4d5250 msgr2=0x563e9f4d76f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f4d5250 0x563e9f4d76f0 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fcc28035070 msgr2=0x7fcc2803b820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fcc28035070 0x7fcc2803b820 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563e9f4de6d0 msgr2=0x563e9f4e0ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x563e9f4de6d0 0x563e9f4e0ac0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb08034600 msgr2=0x7fcb08036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb08034600 0x7fcb08036ac0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcc30048440 tx=0x7fcc30049000 comp rx=0 tx=0).stop 2024-04-27T03:18:13.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 msgr2=0x563e9f4c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/752268344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4c2740 0x563e9f4c9bd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcc28016590 tx=0x7fcc280498a0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 reap_dead start 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc37e80640 1 -- 172.21.15.110:0/752268344 reap_dead start 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 shutdown_connections 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 >> 172.21.15.110:0/752268344 conn(0x563e9f4bea80 msgr2=0x563e9f4bc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 shutdown_connections 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/752268344 wait complete. 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb10034650 msgr2=0x7fcb10036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb10034650 0x7fcb10036b10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x563e9f4a5990 tx=0x7fcc30010440 comp rx=0 tx=0).stop 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4b5540 msgr2=0x563e9f4b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4b5540 0x563e9f4b5960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800aab0 tx=0x7fcc2800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 shutdown_connections 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb10034650 0x7fcb10036b10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/4232333430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f4b5540 0x563e9f4b5960 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 >> 172.21.15.110:0/4232333430 conn(0x563e9f4aa350 msgr2=0x563e9f4aae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 shutdown_connections 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/4232333430 wait complete. 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4a2960 msgr2=0x563e9f4a4d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4a2960 0x563e9f4a4d50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f3ac9b0 msgr2=0x563e9f4a1550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f3ac9b0 0x563e9f4a1550 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb20034650 msgr2=0x7fcb20036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb20034650 0x7fcb20036b10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c0ad000 tx=0x7fcc2c0a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 msgr2=0x563e9f49a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f49a620 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x563e9f494470 tx=0x7fcc30000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 shutdown_connections 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x563e9f4a2960 0x563e9f4a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x563e9f3ac9b0 0x563e9f4a1550 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fcb20034650 0x7fcb20036b10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.973+0000 7fcc3ab66980 1 --2- 172.21.15.110:0/2870041623 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x563e9f3a8490 0x563e9f49a620 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:13.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.977+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 >> 172.21.15.110:0/2870041623 conn(0x563e9f31bb90 msgr2=0x563e9f48dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:13.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.977+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 shutdown_connections 2024-04-27T03:18:13.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:13.977+0000 7fcc3ab66980 1 -- 172.21.15.110:0/2870041623 wait complete. 2024-04-27T03:18:13.992 INFO:teuthology.orchestra.run.smithi110.stdout:{"bucket":"myfoo","tenant":"","versioning":"off","zonegroup":"3fb181a9-2f1b-4a6e-85a1-ca8305b0483c","placement_rule":"default-placement","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""},"id":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","marker":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1","index_type":"Normal","index_generation":0,"num_shards":11,"object_lock_enabled":false,"mfa_enabled":false,"owner":"foo","ver":"0#1,1#1,2#1,3#1,4#1,5#3,6#3,7#1,8#3,9#1,10#1","master_ver":"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0","mtime":"2024-04-27T03:18:13.274887Z","creation_time":"2024-04-27T03:18:09.694744Z","max_marker":"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#","usage":{"rgw.main":{"size":0,"size_actual":0,"size_utilized":0,"size_kb":0,"size_kb_actual":0,"size_kb_utilized":0,"num_objects":0}},"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}} 2024-04-27T03:18:13.992 DEBUG:tasks.util.rgw: json result: {'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#3,6#3,7#1,8#3,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:18:13.274887Z', 'creation_time': '2024-04-27T03:18:09.694744Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2024-04-27T03:18:13.992 INFO:tasks.util.rgw:rgwadmin: client.0 : ['log', 'list'] 2024-04-27T03:18:13.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'log', 'list'] 2024-04-27T03:18:13.993 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph log list 2024-04-27T03:18:14.063 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:14.063 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:14.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e6bfc980 1 Processor -- start 2024-04-27T03:18:14.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e6bfc980 1 -- start start 2024-04-27T03:18:14.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e6bfc980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55fa2a2184a0 0x55fa2a218880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e6bfc980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55fa2a0d50d0 con 0x55fa2a218dc0 2024-04-27T03:18:14.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e6bfc980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55fa2a0ca430 con 0x55fa2a2184a0 2024-04-27T03:18:14.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e4717640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55fa2a2184a0 0x55fa2a218880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e4717640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55fa2a2184a0 0x55fa2a218880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45030/0 (socket says 172.21.15.110:45030) 2024-04-27T03:18:14.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e3f16640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55fa2a218dc0 0x55fa2a219780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:50710/0 (socket says 172.21.15.110:50710) 2024-04-27T03:18:14.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e4717640 1 -- 172.21.15.110:0/4073260559 learned_addr learned my addr 172.21.15.110:0/4073260559 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e4717640 1 -- 172.21.15.110:0/4073260559 >> v1:172.21.15.77:6789/0 conn(0x55fa2a218dc0 legacy=0x55fa2a219780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.057+0000 7f77e4717640 1 -- 172.21.15.110:0/4073260559 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2a0c3510 con 0x55fa2a2184a0 2024-04-27T03:18:14.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e4717640 1 --2- 172.21.15.110:0/4073260559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a218880 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00a5c0 tx=0x7f77cc01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f63c5fee1601c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e3715640 1 -- 172.21.15.110:0/4073260559 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77cc0a6070 con 0x55fa2a2184a0 2024-04-27T03:18:14.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e3715640 1 -- 172.21.15.110:0/4073260559 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77cc0af070 con 0x55fa2a2184a0 2024-04-27T03:18:14.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e3715640 1 -- 172.21.15.110:0/4073260559 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77cc0aa5c0 con 0x55fa2a2184a0 2024-04-27T03:18:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4073260559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 msgr2=0x55fa2a218880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/4073260559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a218880 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00a5c0 tx=0x7f77cc01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4073260559 shutdown_connections 2024-04-27T03:18:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/4073260559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a218880 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4073260559 >> 172.21.15.110:0/4073260559 conn(0x55fa2a18bb80 msgr2=0x55fa2a18bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4073260559 shutdown_connections 2024-04-27T03:18:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.061+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4073260559 wait complete. 2024-04-27T03:18:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e6bfc980 1 Processor -- start 2024-04-27T03:18:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e6bfc980 1 -- start start 2024-04-27T03:18:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e6bfc980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a303d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e6bfc980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fa2a1192d0 con 0x55fa2a2184a0 2024-04-27T03:18:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e4717640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a303d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e4717640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a303d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45042/0 (socket says 172.21.15.110:45042) 2024-04-27T03:18:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e4717640 1 -- 172.21.15.110:0/1146507640 learned_addr learned my addr 172.21.15.110:0/1146507640 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e4717640 1 -- 172.21.15.110:0/1146507640 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2a09fae0 con 0x55fa2a2184a0 2024-04-27T03:18:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.073+0000 7f77e4717640 1 --2- 172.21.15.110:0/1146507640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a303d00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00a5c0 tx=0x7f77cc003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d749ec990f75233 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f76d260c640 1 -- 172.21.15.110:0/1146507640 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77cc0aa9b0 con 0x55fa2a2184a0 2024-04-27T03:18:14.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f76d260c640 1 -- 172.21.15.110:0/1146507640 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77cc0aab50 con 0x55fa2a2184a0 2024-04-27T03:18:14.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f76d260c640 1 -- 172.21.15.110:0/1146507640 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77cc0aae50 con 0x55fa2a2184a0 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- 172.21.15.110:0/1146507640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 msgr2=0x55fa2a303d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/1146507640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a303d00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00a5c0 tx=0x7f77cc003030 comp rx=0 tx=0).stop 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- 172.21.15.110:0/1146507640 shutdown_connections 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/1146507640 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a2184a0 0x55fa2a303d00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- 172.21.15.110:0/1146507640 >> 172.21.15.110:0/1146507640 conn(0x55fa2a18bb80 msgr2=0x55fa2a2ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- 172.21.15.110:0/1146507640 shutdown_connections 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- 172.21.15.110:0/1146507640 wait complete. 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 Processor -- start 2024-04-27T03:18:14.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- start start 2024-04-27T03:18:14.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a218dc0 0x55fa2a30ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fa2a2224f0 con 0x55fa2a218dc0 2024-04-27T03:18:14.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e4717640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a218dc0 0x55fa2a30ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e4717640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a218dc0 0x55fa2a30ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45052/0 (socket says 172.21.15.110:45052) 2024-04-27T03:18:14.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e4717640 1 -- 172.21.15.110:0/2978962113 learned_addr learned my addr 172.21.15.110:0/2978962113 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e4717640 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2a30b3b0 con 0x55fa2a218dc0 2024-04-27T03:18:14.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e4717640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a218dc0 0x55fa2a30ae70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00a5c0 tx=0x7f77cc0180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f76d0e09640 1 -- 172.21.15.110:0/2978962113 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77cc0c6690 con 0x55fa2a218dc0 2024-04-27T03:18:14.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fa2a30b640 con 0x55fa2a218dc0 2024-04-27T03:18:14.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f76d0e09640 1 -- 172.21.15.110:0/2978962113 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77cc0c6830 con 0x55fa2a218dc0 2024-04-27T03:18:14.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f76d0e09640 1 -- 172.21.15.110:0/2978962113 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77cc0c6b10 con 0x55fa2a218dc0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.077+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2a30e270 con 0x55fa2a218dc0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f76d0e09640 1 -- 172.21.15.110:0/2978962113 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f77cc0ce020 con 0x55fa2a218dc0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f76d0e09640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76cc034650 0x7f76cc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f76d0e09640 1 -- 172.21.15.110:0/2978962113 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f77cc0cc020 con 0x55fa2a218dc0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e3f16640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76cc034650 0x7f76cc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a30fd40 0x55fa2a312130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3126b0 con 0x55fa2a30fd40 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4f18640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a30fd40 0x55fa2a312130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e3f16640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76cc034650 0x7f76cc036b10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f77d4000f70 tx=0x7f77d4001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4f18640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a30fd40 0x55fa2a312130 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4f18640 1 -- 172.21.15.110:0/2978962113 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f77dc00a4d0 con 0x55fa2a30fd40 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a313480 0x55fa2a315870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a315eb0 con 0x55fa2a313480 2024-04-27T03:18:14.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4717640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a313480 0x55fa2a315870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4717640 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a313480 0x55fa2a315870 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4717640 1 -- 172.21.15.110:0/2978962113 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a313480 2024-04-27T03:18:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a317760 con 0x55fa2a30fd40 2024-04-27T03:18:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4f18640 1 -- 172.21.15.110:0/2978962113 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f77dc00a4d0 con 0x55fa2a30fd40 2024-04-27T03:18:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a316990 con 0x55fa2a313480 2024-04-27T03:18:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e4717640 1 -- 172.21.15.110:0/2978962113 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a313480 2024-04-27T03:18:14.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.081+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a316d80 con 0x55fa2a30fd40 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e4f18640 1 -- 172.21.15.110:0/2978962113 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f77dc00a4d0 con 0x55fa2a30fd40 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e6bfc980 1 Processor -- start 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e6bfc980 1 -- start start 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e6bfc980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a326080 0x55fa2a3264a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e6bfc980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fa2a11d310 con 0x55fa2a326080 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e4f18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a326080 0x55fa2a3264a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e4f18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a326080 0x55fa2a3264a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45066/0 (socket says 172.21.15.110:45066) 2024-04-27T03:18:14.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e4f18640 1 -- 172.21.15.110:0/3626995386 learned_addr learned my addr 172.21.15.110:0/3626995386 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e4f18640 1 -- 172.21.15.110:0/3626995386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2a3269e0 con 0x55fa2a326080 2024-04-27T03:18:14.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e4f18640 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a326080 0x55fa2a3264a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00dfc0 tx=0x7f77dc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f76c9ffb640 1 -- 172.21.15.110:0/3626995386 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77dc0157d0 con 0x55fa2a326080 2024-04-27T03:18:14.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fa2a327480 con 0x55fa2a326080 2024-04-27T03:18:14.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2a327950 con 0x55fa2a326080 2024-04-27T03:18:14.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f76c9ffb640 1 -- 172.21.15.110:0/3626995386 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77dc015970 con 0x55fa2a326080 2024-04-27T03:18:14.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f76c9ffb640 1 -- 172.21.15.110:0/3626995386 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77dc015c50 con 0x55fa2a326080 2024-04-27T03:18:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f76c9ffb640 1 -- 172.21.15.110:0/3626995386 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f77dc020440 con 0x55fa2a326080 2024-04-27T03:18:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f76c9ffb640 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76bc034650 0x7f76bc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f76c9ffb640 1 -- 172.21.15.110:0/3626995386 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f77dc00e040 con 0x55fa2a326080 2024-04-27T03:18:14.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.085+0000 7f77e4717640 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76bc034650 0x7f76bc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 Processor -- start 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- start start 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4717640 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76bc034650 0x7f76bc036b10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55fa2a3164b0 tx=0x7f77cc0d0ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a333610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fa2a304390 con 0x55fa2a3331f0 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a333610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a333610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45082/0 (socket says 172.21.15.110:45082) 2024-04-27T03:18:14.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 -- 172.21.15.110:0/4137707857 learned_addr learned my addr 172.21.15.110:0/4137707857 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 -- 172.21.15.110:0/4137707857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2a333be0 con 0x55fa2a3331f0 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 --2- 172.21.15.110:0/4137707857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a333610 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f77dc04aaa0 tx=0x7f77dc020890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f1288733741c6a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f76c37fe640 1 -- 172.21.15.110:0/4137707857 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77dc048030 con 0x55fa2a3331f0 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f76c37fe640 1 -- 172.21.15.110:0/4137707857 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77dc020c80 con 0x55fa2a3331f0 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f76c37fe640 1 -- 172.21.15.110:0/4137707857 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77dc013de0 con 0x55fa2a3331f0 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4137707857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 msgr2=0x55fa2a333610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/4137707857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a333610 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f77dc04aaa0 tx=0x7f77dc020890 comp rx=0 tx=0).stop 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4137707857 shutdown_connections 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/4137707857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a333610 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4137707857 >> 172.21.15.110:0/4137707857 conn(0x55fa2a32f530 msgr2=0x55fa2a32d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4137707857 shutdown_connections 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- 172.21.15.110:0/4137707857 wait complete. 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 Processor -- start 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- start start 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a33a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e6bfc980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fa2a334360 con 0x55fa2a3331f0 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a33a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a33a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45096/0 (socket says 172.21.15.110:45096) 2024-04-27T03:18:14.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 learned_addr learned my addr 172.21.15.110:0/2860614277 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa2a33abf0 con 0x55fa2a3331f0 2024-04-27T03:18:14.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.089+0000 7f77e4f18640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a33a6b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f77dc012750 tx=0x7f77dc04aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f76c1ffb640 1 -- 172.21.15.110:0/2860614277 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77dc038020 con 0x55fa2a3331f0 2024-04-27T03:18:14.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fa2a33ae80 con 0x55fa2a3331f0 2024-04-27T03:18:14.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f76c1ffb640 1 -- 172.21.15.110:0/2860614277 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77dc03c3b0 con 0x55fa2a3331f0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2a33db60 con 0x55fa2a3331f0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f76c1ffb640 1 -- 172.21.15.110:0/2860614277 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77dc04c070 con 0x55fa2a3331f0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f76c1ffb640 1 -- 172.21.15.110:0/2860614277 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f77dc03b020 con 0x55fa2a3331f0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f76c1ffb640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76b4034650 0x7f76b4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f76c1ffb640 1 -- 172.21.15.110:0/2860614277 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f77dc02f050 con 0x55fa2a3331f0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e4717640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76b4034650 0x7f76b4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a345d00 0x55fa2a348190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3488e0 con 0x55fa2a345d00 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e3f16640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a345d00 0x55fa2a348190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e4717640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76b4034650 0x7f76b4036b10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0d2fd0 tx=0x7f77cc0ea000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e3f16640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a345d00 0x55fa2a348190 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f77d400f620 con 0x55fa2a345d00 2024-04-27T03:18:14.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 realm 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a345210 con 0x55fa2a345d00 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f77d400f620 con 0x55fa2a345d00 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3493e0 con 0x55fa2a345d00 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f77d400f620 con 0x55fa2a345d00 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a349fc0 0x55fa2a34c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34cbf0 con 0x55fa2a349fc0 2024-04-27T03:18:14.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.093+0000 7f77e4f18640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a349fc0 0x55fa2a34c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4f18640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a349fc0 0x55fa2a34c480 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e6bfc980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34d490 con 0x55fa2a345d00 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f77d400f620 con 0x55fa2a345d00 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e6bfc980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e6bfc980 20 searching for the correct realm 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fa2a34f200 0x55fa2a3515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a34f200 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4717640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fa2a34f200 0x55fa2a3515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4717640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fa2a34f200 0x55fa2a3515f0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4f18640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f77dc014030 0x7f77dc0133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x7f77dc014030 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f77dc014030 0x7f77dc0133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f77dc014030 0x7f77dc0133b0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d400f620 con 0x7f77dc014030 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d400f620 con 0x55fa2a345d00 2024-04-27T03:18:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a34f200 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.097+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x7f77dc014030 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x7f77dc014030 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x7f77dc014030 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f77d402b040 con 0x7f77dc014030 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a34f200 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.101+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a34f200 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a34f200 2024-04-27T03:18:14.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x7f77dc014030 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x7f77dc014030 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a345d00 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d402b040 con 0x55fa2a345d00 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x7f77dc014030 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a34f200 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x7f77dc014030 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.105+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a34f200 2024-04-27T03:18:14.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a351c00 con 0x55fa2a349fc0 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34de40 con 0x55fa2a345d00 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34e290 con 0x55fa2a349fc0 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34e6b0 con 0x55fa2a345d00 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 zone default found 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 4 Realm: () 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 zonegroup default 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34da90 con 0x7f77dc014030 2024-04-27T03:18:14.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a354440 con 0x55fa2a34f200 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3547e0 con 0x55fa2a34f200 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 started sync module instance, tier type = 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.109+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3538d0 con 0x55fa2a349fc0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94532938317040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a353530 con 0x55fa2a349fc0 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3556a0 con 0x55fa2a345d00 2024-04-27T03:18:14.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94532938322416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a355610 con 0x55fa2a345d00 2024-04-27T03:18:14.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3565e0 con 0x55fa2a349fc0 2024-04-27T03:18:14.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94532938317040] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.113+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94532938322416] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94532938324624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3565a0 con 0x55fa2a349fc0 2024-04-27T03:18:14.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3569f0 con 0x7f77dc014030 2024-04-27T03:18:14.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94532938320848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3569f0 con 0x7f77dc014030 2024-04-27T03:18:14.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3578b0 con 0x55fa2a349fc0 2024-04-27T03:18:14.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94532938324624] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94532938320848] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94532938329568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a358150 con 0x55fa2a34f200 2024-04-27T03:18:14.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.117+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94532938333344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a358eb0 con 0x55fa2a34f200 2024-04-27T03:18:14.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94532938329568] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94532938333344] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94532938326192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3589a0 con 0x55fa2a34f200 2024-04-27T03:18:14.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a359750 con 0x7f77dc014030 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'150 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94532938338976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3596c0 con 0x7f77dc014030 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=0 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=1 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=2 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=3 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=4 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=5 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94532938326192] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=6 2024-04-27T03:18:14.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94532938338976] v27'151 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 20 add_watcher() i=7 2024-04-27T03:18:14.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:14.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3584f0 con 0x55fa2a345d00 2024-04-27T03:18:14.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.121+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94532938335472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a35a1d0 con 0x55fa2a345d00 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94532938335472] v27'152 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f769a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e6bfc980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e6bfc980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e6bfc980 5 note: GC not initialized 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f76997e2640 20 reqs_thread_entry: start 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e6bfc980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:14.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e6bfc980 20 Filter name: none 2024-04-27T03:18:14.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f760affd640 20 reqs_thread_entry: start 2024-04-27T03:18:14.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 6.0 6:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37f960 con 0x55fa2a345d00 2024-04-27T03:18:14.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 [pgnls start_epoch 27 out=254b] v27'46 uv30 ondisk = 1) ==== 144+0+254 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.10 6:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34e6b0 con 0x55fa2a345d00 2024-04-27T03:18:14.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 [pgnls start_epoch 27 out=131b] v27'4 uv4 ondisk = 1) ==== 144+0+131 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.125+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 6.8 6:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a34e6b0 con 0x55fa2a349fc0 2024-04-27T03:18:14.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 [pgnls start_epoch 27 out=295b] v27'35 uv27 ondisk = 1) ==== 144+0+295 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 6.18 6:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3538d0 con 0x7f77dc014030 2024-04-27T03:18:14.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(65 [pgnls start_epoch 27 out=172b] v27'24 uv15 ondisk = 1) ==== 144+0+172 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3538d0 con 0x55fa2a345d00 2024-04-27T03:18:14.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(66 [pgnls start_epoch 27 out=335b] v27'152 uv29 ondisk = 1) ==== 144+0+335 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 6.14 6:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3538d0 con 0x7f77dc014030 2024-04-27T03:18:14.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(67 [pgnls start_epoch 27 out=213b] v27'22 uv22 ondisk = 1) ==== 144+0+213 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 6.c 6:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3538d0 con 0x55fa2a34f200 2024-04-27T03:18:14.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(68 [pgnls start_epoch 27 out=254b] v27'10 uv10 ondisk = 1) ==== 144+0+254 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 6.1c 6:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(69 [pgnls start_epoch 27 out=295b] v27'23 uv15 ondisk = 1) ==== 144+0+295 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 6.2 6:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 [pgnls start_epoch 27 out=213b] v27'16 uv16 ondisk = 1) ==== 144+0+213 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.12 6:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(71 [pgnls start_epoch 27 out=172b] v27'17 uv9 ondisk = 1) ==== 144+0+172 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.a 6:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(72 [pgnls start_epoch 27 out=336b] v27'33 uv25 ondisk = 1) ==== 144+0+336 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.129+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 6.1a 6:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(73 [pgnls start_epoch 27 out=131b] v27'29 uv21 ondisk = 1) ==== 144+0+131 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:74 6.6 6:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(74 [pgnls start_epoch 27 out=254b] v27'12 uv12 ondisk = 1) ==== 144+0+254 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 6.16 6:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(75 [pgnls start_epoch 27 out=90b] v27'30 uv21 ondisk = 1) ==== 144+0+90 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:76 6.e 6:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(76 [pgnls start_epoch 27 out=213b] v27'25 uv17 ondisk = 1) ==== 144+0+213 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 6.1e 6:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(77 [pgnls start_epoch 27 out=254b] v27'16 uv16 ondisk = 1) ==== 144+0+254 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:78 6.1 6:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a345d00 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(78 [pgnls start_epoch 27 out=254b] v27'20 uv20 ondisk = 1) ==== 144+0+254 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.11 6:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(79 [pgnls start_epoch 27 out=213b] v27'25 uv17 ondisk = 1) ==== 144+0+213 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:80 6.9 6:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(80 [pgnls start_epoch 27 out=172b] v27'40 uv24 ondisk = 1) ==== 144+0+172 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 6.19 6:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(81 [pgnls start_epoch 27 out=295b] v27'47 uv39 ondisk = 1) ==== 144+0+295 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.133+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.5 6:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(82 [pgnls start_epoch 27 out=276b] v27'15 uv15 ondisk = 1) ==== 144+0+276 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.15 6:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(83 [pgnls start_epoch 27 out=172b] v27'23 uv15 ondisk = 1) ==== 144+0+172 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:84 6.d 6:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(84 [pgnls start_epoch 27 out=295b] v27'25 uv17 ondisk = 1) ==== 144+0+295 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.1d 6:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a345d00 2024-04-27T03:18:14.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 [pgnls start_epoch 27 out=172b] v27'35 uv19 ondisk = 1) ==== 144+0+172 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:86 6.3 6:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(86 [pgnls start_epoch 27 out=213b] v27'51 uv27 ondisk = 1) ==== 144+0+213 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.137+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.13 6:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(87 [pgnls start_epoch 27 out=172b] v27'19 uv11 ondisk = 1) ==== 144+0+172 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.b 6:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(88 [pgnls start_epoch 27 out=172b] v27'24 uv15 ondisk = 1) ==== 144+0+172 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:89 6.1b 6:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(89 [pgnls start_epoch 27 out=90b] v27'19 uv11 ondisk = 1) ==== 144+0+90 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:90 6.7 6:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a349fc0 2024-04-27T03:18:14.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(90 [pgnls start_epoch 27 out=90b] v27'43 uv19 ondisk = 1) ==== 144+0+90 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:91 6.17 6:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a345d00 2024-04-27T03:18:14.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(91 [pgnls start_epoch 27 out=131b] v27'12 uv12 ondisk = 1) ==== 144+0+131 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:92 6.f 6:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x7f77dc014030 2024-04-27T03:18:14.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(92 [pgnls start_epoch 27 out=295b] v27'23 uv15 ondisk = 1) ==== 144+0+295 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:93 6.1f 6:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a357400 con 0x55fa2a34f200 2024-04-27T03:18:14.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.141+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(93 [pgnls start_epoch 27 out=213b] v27'25 uv17 ondisk = 1) ==== 144+0+213 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.150 INFO:teuthology.orchestra.run.smithi110.stdout:["obj_delete_at_hint.0000000078","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_loggenerations_metadata","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","obj_delete_at_hint.0000000027","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2024-04-27T03:18:14.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.145+0000 7f769a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:14.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.145+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94532938335472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37f920 con 0x55fa2a345d00 2024-04-27T03:18:14.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.145+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(94 data_loggenerations_metadata [watch unwatch cookie 94532938335472] v27'153 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.145+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94532938317040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a3538d0 con 0x55fa2a349fc0 2024-04-27T03:18:14.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(95 notify.0 [watch unwatch cookie 94532938317040] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e6bfc980 20 remove_watcher() i=0 2024-04-27T03:18:14.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e6bfc980 2 removed watcher, disabling cache 2024-04-27T03:18:14.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94532938322416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37dd90 con 0x55fa2a345d00 2024-04-27T03:18:14.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(96 notify.1 [watch unwatch cookie 94532938322416] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x55fa2a345d00 2024-04-27T03:18:14.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e6bfc980 20 remove_watcher() i=1 2024-04-27T03:18:14.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94532938324624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37dd90 con 0x55fa2a349fc0 2024-04-27T03:18:14.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(97 notify.2 [watch unwatch cookie 94532938324624] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e6bfc980 20 remove_watcher() i=2 2024-04-27T03:18:14.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.149+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94532938320848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37dd90 con 0x7f77dc014030 2024-04-27T03:18:14.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(98 notify.3 [watch unwatch cookie 94532938320848] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e6bfc980 20 remove_watcher() i=3 2024-04-27T03:18:14.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94532938329568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37dd90 con 0x55fa2a349fc0 2024-04-27T03:18:14.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e4f18640 1 -- 172.21.15.110:0/2860614277 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(99 notify.4 [watch unwatch cookie 94532938329568] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77dc053070 con 0x55fa2a349fc0 2024-04-27T03:18:14.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e6bfc980 20 remove_watcher() i=4 2024-04-27T03:18:14.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94532938333344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37dd90 con 0x55fa2a34f200 2024-04-27T03:18:14.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(100 notify.5 [watch unwatch cookie 94532938333344] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e6bfc980 20 remove_watcher() i=5 2024-04-27T03:18:14.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.153+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94532938326192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37dd90 con 0x55fa2a34f200 2024-04-27T03:18:14.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e4717640 1 -- 172.21.15.110:0/2860614277 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(101 notify.6 [watch unwatch cookie 94532938326192] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77cc0d09a0 con 0x55fa2a34f200 2024-04-27T03:18:14.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 20 remove_watcher() i=6 2024-04-27T03:18:14.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94532938338976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fa2a37dd90 con 0x7f77dc014030 2024-04-27T03:18:14.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(102 notify.7 [watch unwatch cookie 94532938338976] v27'152 uv150 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f77d403e040 con 0x7f77dc014030 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 20 remove_watcher() i=7 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a349fc0 msgr2=0x55fa2a34c480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a349fc0 0x55fa2a34c480 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a345d00 msgr2=0x55fa2a348190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a345d00 0x55fa2a348190 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f77dc014030 msgr2=0x7f77dc0133b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f77dc014030 0x7f77dc0133b0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fa2a34f200 msgr2=0x55fa2a3515f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fa2a34f200 0x55fa2a3515f0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76b4034650 msgr2=0x7f76b4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76b4034650 0x7f76b4036b10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0d2fd0 tx=0x7f77cc0ea000 comp rx=0 tx=0).stop 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e3f16640 1 -- 172.21.15.110:0/2860614277 reap_dead start 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 msgr2=0x55fa2a33a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a33a6b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f77dc012750 tx=0x7f77dc04aa30 comp rx=0 tx=0).stop 2024-04-27T03:18:14.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 shutdown_connections 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2860614277 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a3331f0 0x55fa2a33a6b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 >> 172.21.15.110:0/2860614277 conn(0x55fa2a32f530 msgr2=0x55fa2a32cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 shutdown_connections 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2860614277 wait complete. 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76bc034650 msgr2=0x7f76bc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76bc034650 0x7f76bc036b10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x55fa2a3164b0 tx=0x7f77cc0d0ee0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a326080 msgr2=0x55fa2a3264a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a326080 0x55fa2a3264a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00dfc0 tx=0x7f77dc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 shutdown_connections 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76bc034650 0x7f76bc036b10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/3626995386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a326080 0x55fa2a3264a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 >> 172.21.15.110:0/3626995386 conn(0x55fa2a31ae90 msgr2=0x55fa2a31b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 shutdown_connections 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/3626995386 wait complete. 2024-04-27T03:18:14.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a313480 msgr2=0x55fa2a315870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a313480 0x55fa2a315870 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a30fd40 msgr2=0x55fa2a312130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a30fd40 0x55fa2a312130 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76cc034650 msgr2=0x7f76cc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76cc034650 0x7f76cc036b10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f77d4000f70 tx=0x7f77d4001380 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a218dc0 msgr2=0x55fa2a30ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a218dc0 0x55fa2a30ae70 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00a5c0 tx=0x7f77cc0180e0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 shutdown_connections 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fa2a313480 0x55fa2a315870 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fa2a30fd40 0x55fa2a312130 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f76cc034650 0x7f76cc036b10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 --2- 172.21.15.110:0/2978962113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fa2a218dc0 0x55fa2a30ae70 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 >> 172.21.15.110:0/2978962113 conn(0x55fa2a18bb80 msgr2=0x55fa2a2fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 shutdown_connections 2024-04-27T03:18:14.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.157+0000 7f77e6bfc980 1 -- 172.21.15.110:0/2978962113 wait complete. 2024-04-27T03:18:14.175 DEBUG:tasks.util.rgw: json result: ['obj_delete_at_hint.0000000078', 'obj_delete_at_hint.0000000070', 'obj_delete_at_hint.0000000104', 'obj_delete_at_hint.0000000026', 'obj_delete_at_hint.0000000028', 'obj_delete_at_hint.0000000040', 'obj_delete_at_hint.0000000015', 'obj_delete_at_hint.0000000069', 'obj_delete_at_hint.0000000095', 'obj_delete_at_hint.0000000003', 'obj_delete_at_hint.0000000047', 'obj_delete_at_hint.0000000035', 'obj_delete_at_hint.0000000037', 'obj_delete_at_hint.0000000024', 'obj_delete_at_hint.0000000000', 'obj_delete_at_hint.0000000031', 'data_loggenerations_metadata', 'obj_delete_at_hint.0000000076', 'obj_delete_at_hint.0000000113', 'obj_delete_at_hint.0000000005', 'obj_delete_at_hint.0000000011', 'obj_delete_at_hint.0000000058', 'obj_delete_at_hint.0000000052', 'obj_delete_at_hint.0000000088', 'obj_delete_at_hint.0000000080', 'obj_delete_at_hint.0000000090', 'obj_delete_at_hint.0000000110', 'obj_delete_at_hint.0000000096', 'obj_delete_at_hint.0000000087', 'obj_delete_at_hint.0000000008', 'obj_delete_at_hint.0000000006', 'obj_delete_at_hint.0000000029', 'obj_delete_at_hint.0000000089', 'obj_delete_at_hint.0000000022', 'obj_delete_at_hint.0000000016', 'obj_delete_at_hint.0000000041', 'obj_delete_at_hint.0000000018', 'obj_delete_at_hint.0000000092', 'obj_delete_at_hint.0000000014', 'obj_delete_at_hint.0000000112', 'obj_delete_at_hint.0000000007', 'obj_delete_at_hint.0000000021', 'obj_delete_at_hint.0000000064', 'obj_delete_at_hint.0000000071', 'obj_delete_at_hint.0000000074', 'obj_delete_at_hint.0000000081', 'obj_delete_at_hint.0000000009', 'obj_delete_at_hint.0000000121', 'obj_delete_at_hint.0000000125', 'obj_delete_at_hint.0000000082', 'obj_delete_at_hint.0000000105', 'obj_delete_at_hint.0000000059', 'obj_delete_at_hint.0000000077', 'obj_delete_at_hint.0000000032', 'obj_delete_at_hint.0000000053', 'obj_delete_at_hint.0000000091', 'obj_delete_at_hint.0000000065', 'obj_delete_at_hint.0000000083', 'obj_delete_at_hint.0000000010', 'obj_delete_at_hint.0000000045', 'obj_delete_at_hint.0000000002', 'obj_delete_at_hint.0000000116', 'obj_delete_at_hint.0000000034', 'obj_delete_at_hint.0000000101', 'obj_delete_at_hint.0000000079', 'obj_delete_at_hint.0000000049', 'obj_delete_at_hint.0000000117', 'obj_delete_at_hint.0000000044', 'obj_delete_at_hint.0000000066', 'obj_delete_at_hint.0000000068', 'obj_delete_at_hint.0000000085', 'obj_delete_at_hint.0000000073', 'obj_delete_at_hint.0000000038', 'obj_delete_at_hint.0000000118', 'obj_delete_at_hint.0000000036', 'obj_delete_at_hint.0000000103', 'obj_delete_at_hint.0000000119', 'obj_delete_at_hint.0000000098', 'obj_delete_at_hint.0000000027', 'obj_delete_at_hint.0000000019', 'obj_delete_at_hint.0000000039', 'obj_delete_at_hint.0000000100', 'obj_delete_at_hint.0000000093', 'obj_delete_at_hint.0000000004', 'obj_delete_at_hint.0000000063', 'obj_delete_at_hint.0000000122', 'obj_delete_at_hint.0000000057', 'obj_delete_at_hint.0000000054', 'obj_delete_at_hint.0000000114', 'data_log.0', 'obj_delete_at_hint.0000000012', 'obj_delete_at_hint.0000000084', 'obj_delete_at_hint.0000000043', 'obj_delete_at_hint.0000000111', 'obj_delete_at_hint.0000000048', 'obj_delete_at_hint.0000000020', 'obj_delete_at_hint.0000000099', 'obj_delete_at_hint.0000000056', 'obj_delete_at_hint.0000000072', 'obj_delete_at_hint.0000000062', 'obj_delete_at_hint.0000000115', 'obj_delete_at_hint.0000000033', 'obj_delete_at_hint.0000000025', 'obj_delete_at_hint.0000000123', 'obj_delete_at_hint.0000000108', 'obj_delete_at_hint.0000000094', 'obj_delete_at_hint.0000000060', 'obj_delete_at_hint.0000000109', 'obj_delete_at_hint.0000000013', 'obj_delete_at_hint.0000000051', 'obj_delete_at_hint.0000000106', 'obj_delete_at_hint.0000000017', 'obj_delete_at_hint.0000000023', 'obj_delete_at_hint.0000000107', 'obj_delete_at_hint.0000000046', 'obj_delete_at_hint.0000000067', 'obj_delete_at_hint.0000000086', 'obj_delete_at_hint.0000000050', 'obj_delete_at_hint.0000000102', 'obj_delete_at_hint.0000000124', 'obj_delete_at_hint.0000000001', 'obj_delete_at_hint.0000000126', 'obj_delete_at_hint.0000000055', 'obj_delete_at_hint.0000000061', 'obj_delete_at_hint.0000000097', 'obj_delete_at_hint.0000000042', 'obj_delete_at_hint.0000000075', 'obj_delete_at_hint.0000000120', 'obj_delete_at_hint.0000000030'] 2024-04-27T03:18:14.175 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'suspend', '--uid', 'foo'] 2024-04-27T03:18:14.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'suspend', '--uid', 'foo'] 2024-04-27T03:18:14.175 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user suspend --uid foo 2024-04-27T03:18:14.251 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:14.251 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:14.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae4631980 1 Processor -- start 2024-04-27T03:18:14.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae4631980 1 -- start start 2024-04-27T03:18:14.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae4631980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55959e30fa60 0x55959e311e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae4631980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55959e1c80d0 con 0x55959e27d850 2024-04-27T03:18:14.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae4631980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55959e1bd430 con 0x55959e30fa60 2024-04-27T03:18:14.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae194b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55959e30fa60 0x55959e311e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae214c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55959e27d850 0x55959e30e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:50722/0 (socket says 172.21.15.110:50722) 2024-04-27T03:18:14.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae214c640 1 -- 172.21.15.110:0/3517549346 learned_addr learned my addr 172.21.15.110:0/3517549346 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:14.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae194b640 1 --2- 172.21.15.110:0/3517549346 >> v2:172.21.15.77:3300/0 conn(0x55959e30fa60 0x55959e311e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45100/0 (socket says 172.21.15.110:45100) 2024-04-27T03:18:14.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae194b640 1 -- 172.21.15.110:0/3517549346 >> v1:172.21.15.77:6789/0 conn(0x55959e27d850 legacy=0x55959e30e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.245+0000 7f2ae194b640 1 -- 172.21.15.110:0/3517549346 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55959e1b6510 con 0x55959e30fa60 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae194b640 1 --2- 172.21.15.110:0/3517549346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e30fa60 0x55959e311e50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800a480 tx=0x7f2ad801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd294e5351082ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae114a640 1 -- 172.21.15.110:0/3517549346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad80b0070 con 0x55959e30fa60 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae114a640 1 -- 172.21.15.110:0/3517549346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad80ab540 con 0x55959e30fa60 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae114a640 1 -- 172.21.15.110:0/3517549346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad80ab840 con 0x55959e30fa60 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae4631980 1 -- 172.21.15.110:0/3517549346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e30fa60 msgr2=0x55959e311e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae4631980 1 --2- 172.21.15.110:0/3517549346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e30fa60 0x55959e311e50 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2ad800a480 tx=0x7f2ad801b280 comp rx=0 tx=0).stop 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae4631980 1 -- 172.21.15.110:0/3517549346 shutdown_connections 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae4631980 1 --2- 172.21.15.110:0/3517549346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e30fa60 0x55959e311e50 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae4631980 1 -- 172.21.15.110:0/3517549346 >> 172.21.15.110:0/3517549346 conn(0x55959e303a20 msgr2=0x55959e305e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae4631980 1 -- 172.21.15.110:0/3517549346 shutdown_connections 2024-04-27T03:18:14.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.249+0000 7f2ae4631980 1 -- 172.21.15.110:0/3517549346 wait complete. 2024-04-27T03:18:14.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.261+0000 7f2ae4631980 1 Processor -- start 2024-04-27T03:18:14.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.261+0000 7f2ae4631980 1 -- start start 2024-04-27T03:18:14.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.261+0000 7f2ae4631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.261+0000 7f2ae4631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55959e20c2d0 con 0x55959e27d850 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45116/0 (socket says 172.21.15.110:45116) 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 -- 172.21.15.110:0/1357255114 learned_addr learned my addr 172.21.15.110:0/1357255114 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 -- 172.21.15.110:0/1357255114 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55959e192ae0 con 0x55959e27d850 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 --2- 172.21.15.110:0/1357255114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3f65f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4001d90 tx=0x7f2ad4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436a08b0bf1a2044 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f29d0608640 1 -- 172.21.15.110:0/1357255114 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad4013030 con 0x55959e27d850 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f29d0608640 1 -- 172.21.15.110:0/1357255114 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad400e690 con 0x55959e27d850 2024-04-27T03:18:14.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f29d0608640 1 -- 172.21.15.110:0/1357255114 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad400ea10 con 0x55959e27d850 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- 172.21.15.110:0/1357255114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 msgr2=0x55959e3f65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 --2- 172.21.15.110:0/1357255114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3f65f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4001d90 tx=0x7f2ad4001e80 comp rx=0 tx=0).stop 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- 172.21.15.110:0/1357255114 shutdown_connections 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 --2- 172.21.15.110:0/1357255114 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3f65f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- 172.21.15.110:0/1357255114 >> 172.21.15.110:0/1357255114 conn(0x55959e303a20 msgr2=0x55959e3f2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- 172.21.15.110:0/1357255114 shutdown_connections 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- 172.21.15.110:0/1357255114 wait complete. 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 Processor -- start 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- start start 2024-04-27T03:18:14.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3fd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55959e314df0 con 0x55959e27d850 2024-04-27T03:18:14.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3fd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3fd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45120/0 (socket says 172.21.15.110:45120) 2024-04-27T03:18:14.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 -- 172.21.15.110:0/194513652 learned_addr learned my addr 172.21.15.110:0/194513652 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55959e3fdb70 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae214c640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3fd630 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x55959e3f7480 tx=0x7f2ad4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f29cee05640 1 -- 172.21.15.110:0/194513652 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad4027070 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55959e3fde00 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f29cee05640 1 -- 172.21.15.110:0/194513652 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad4019900 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f29cee05640 1 -- 172.21.15.110:0/194513652 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ad4019c20 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.265+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55959e400a80 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f29cee05640 1 -- 172.21.15.110:0/194513652 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2ad402a070 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f29cee05640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29c8034650 0x7f29c8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f29cee05640 1 -- 172.21.15.110:0/194513652 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2ad401f070 con 0x55959e27d850 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae194b640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29c8034650 0x7f29c8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e30fa60 0x55959e404560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e404b60 con 0x55959e30fa60 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae294d640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e30fa60 0x55959e404560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae194b640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29c8034650 0x7f29c8036b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80ad000 tx=0x7f2ad80a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae294d640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e30fa60 0x55959e404560 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae294d640 1 -- 172.21.15.110:0/194513652 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2adc00a4d0 con 0x55959e30fa60 2024-04-27T03:18:14.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e405930 0x55959e407d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e408360 con 0x55959e405930 2024-04-27T03:18:14.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae214c640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e405930 0x55959e407d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae214c640 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e405930 0x55959e407d20 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae214c640 1 -- 172.21.15.110:0/194513652 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e405930 2024-04-27T03:18:14.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.269+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e409c10 con 0x55959e30fa60 2024-04-27T03:18:14.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae294d640 1 -- 172.21.15.110:0/194513652 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f2adc00a4d0 con 0x55959e30fa60 2024-04-27T03:18:14.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e408e40 con 0x55959e405930 2024-04-27T03:18:14.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae214c640 1 -- 172.21.15.110:0/194513652 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e405930 2024-04-27T03:18:14.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e409260 con 0x55959e30fa60 2024-04-27T03:18:14.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae294d640 1 -- 172.21.15.110:0/194513652 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f2adc00a4d0 con 0x55959e30fa60 2024-04-27T03:18:14.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 Processor -- start 2024-04-27T03:18:14.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 -- start start 2024-04-27T03:18:14.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e418520 0x55959e418940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55959e210310 con 0x55959e418520 2024-04-27T03:18:14.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e418520 0x55959e418940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e418520 0x55959e418940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45124/0 (socket says 172.21.15.110:45124) 2024-04-27T03:18:14.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae294d640 1 -- 172.21.15.110:0/2533132901 learned_addr learned my addr 172.21.15.110:0/2533132901 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae294d640 1 -- 172.21.15.110:0/2533132901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55959e418e80 con 0x55959e418520 2024-04-27T03:18:14.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae294d640 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e418520 0x55959e418940 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00df90 tx=0x7f2adc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f29c7fff640 1 -- 172.21.15.110:0/2533132901 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2adc0157d0 con 0x55959e418520 2024-04-27T03:18:14.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55959e419920 con 0x55959e418520 2024-04-27T03:18:14.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f29c7fff640 1 -- 172.21.15.110:0/2533132901 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2adc015970 con 0x55959e418520 2024-04-27T03:18:14.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f29c7fff640 1 -- 172.21.15.110:0/2533132901 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2adc015c50 con 0x55959e418520 2024-04-27T03:18:14.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.273+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55959e419df0 con 0x55959e418520 2024-04-27T03:18:14.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f29c7fff640 1 -- 172.21.15.110:0/2533132901 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2adc020440 con 0x55959e418520 2024-04-27T03:18:14.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f29c7fff640 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29bc0346a0 0x7f29bc036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f29c7fff640 1 -- 172.21.15.110:0/2533132901 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2adc00e040 con 0x55959e418520 2024-04-27T03:18:14.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae214c640 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29bc0346a0 0x7f29bc036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 Processor -- start 2024-04-27T03:18:14.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 -- start start 2024-04-27T03:18:14.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae214c640 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29bc0346a0 0x7f29bc036b60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55959e4050f0 tx=0x7f2ad4019e60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e425b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55959e3f6d40 con 0x55959e425770 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e425b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e425b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45140/0 (socket says 172.21.15.110:45140) 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae294d640 1 -- 172.21.15.110:0/1146126959 learned_addr learned my addr 172.21.15.110:0/1146126959 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae294d640 1 -- 172.21.15.110:0/1146126959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55959e426160 con 0x55959e425770 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae294d640 1 --2- 172.21.15.110:0/1146126959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e425b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2adc04aa10 tx=0x7f2adc01e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee7bc58c45fc725 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f29c5ffb640 1 -- 172.21.15.110:0/1146126959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2adc012440 con 0x55959e425770 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f29c5ffb640 1 -- 172.21.15.110:0/1146126959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2adc020b50 con 0x55959e425770 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f29c5ffb640 1 -- 172.21.15.110:0/1146126959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2adc0125e0 con 0x55959e425770 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 -- 172.21.15.110:0/1146126959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 msgr2=0x55959e425b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 --2- 172.21.15.110:0/1146126959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e425b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2adc04aa10 tx=0x7f2adc01e680 comp rx=0 tx=0).stop 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 -- 172.21.15.110:0/1146126959 shutdown_connections 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 --2- 172.21.15.110:0/1146126959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e425b90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 -- 172.21.15.110:0/1146126959 >> 172.21.15.110:0/1146126959 conn(0x55959e421ab0 msgr2=0x55959e41fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 -- 172.21.15.110:0/1146126959 shutdown_connections 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 -- 172.21.15.110:0/1146126959 wait complete. 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.277+0000 7f2ae4631980 1 Processor -- start 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 1 -- start start 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e42cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55959e426830 con 0x55959e425770 2024-04-27T03:18:14.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e42cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae294d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e42cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45144/0 (socket says 172.21.15.110:45144) 2024-04-27T03:18:14.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 learned_addr learned my addr 172.21.15.110:0/2858645918 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55959e42d160 con 0x55959e425770 2024-04-27T03:18:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae294d640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e42cc20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0207a0 tx=0x7f2adc04a9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f29b3fff640 1 -- 172.21.15.110:0/2858645918 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2adc038070 con 0x55959e425770 2024-04-27T03:18:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55959e42d3f0 con 0x55959e425770 2024-04-27T03:18:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f29b3fff640 1 -- 172.21.15.110:0/2858645918 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2adc04ab50 con 0x55959e425770 2024-04-27T03:18:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f29b3fff640 1 -- 172.21.15.110:0/2858645918 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2adc04ae30 con 0x55959e425770 2024-04-27T03:18:14.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55959e4300d0 con 0x55959e425770 2024-04-27T03:18:14.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f29b3fff640 1 -- 172.21.15.110:0/2858645918 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2adc04c0d0 con 0x55959e425770 2024-04-27T03:18:14.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f29b3fff640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29ac034650 0x7f29ac036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f29b3fff640 1 -- 172.21.15.110:0/2858645918 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f2adc03b710 con 0x55959e425770 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae214c640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29ac034650 0x7f29ac036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e4382a0 0x55959e43a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e43ae90 con 0x55959e4382a0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae194b640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e4382a0 0x55959e43a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae214c640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29ac034650 0x7f29ac036b10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400bc00 tx=0x7f2ad4035000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.281+0000 7f2ae194b640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e4382a0 0x55959e43a740 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 realm 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e43b2c0 con 0x55959e4382a0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e43bb90 con 0x55959e4382a0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e43c5b0 0x55959e43e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e43f150 con 0x55959e43c5b0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae294d640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e43c5b0 0x55959e43e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae294d640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e43c5b0 0x55959e43e9e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:14.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e43f9a0 con 0x55959e4382a0 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 20 searching for the correct realm 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55959e441710 0x55959e443b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e441710 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae214c640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55959e441710 0x55959e443b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae214c640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55959e441710 0x55959e443b00 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.285+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae294d640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2adc014030 0x7f2adc03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x7f2adc014030 2024-04-27T03:18:14.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2adc014030 0x7f2adc03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2adc014030 0x7f2adc03ba70 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e441710 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x7f2adc014030 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.289+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x7f2adc014030 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e441710 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e441710 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e441710 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x7f2adc014030 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e4382a0 2024-04-27T03:18:14.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x7f2adc014030 2024-04-27T03:18:14.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.293+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e441710 2024-04-27T03:18:14.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x7f2adc014030 2024-04-27T03:18:14.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e441710 2024-04-27T03:18:14.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55959e444180 con 0x55959e43c5b0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e440350 con 0x55959e4382a0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e43ff90 con 0x55959e43c5b0 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.297+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e440cb0 con 0x55959e4382a0 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 zone default found 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 4 Realm: () 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 zonegroup default 2024-04-27T03:18:14.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e444a90 con 0x7f2adc014030 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e446890 con 0x55959e441710 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e446c30 con 0x55959e441710 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 started sync module instance, tier type = 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:14.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e445d30 con 0x55959e43c5b0 2024-04-27T03:18:14.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94101093769504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e447570 con 0x55959e43c5b0 2024-04-27T03:18:14.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.301+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e447dc0 con 0x55959e4382a0 2024-04-27T03:18:14.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94101093774704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e447ea0 con 0x55959e4382a0 2024-04-27T03:18:14.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e4489c0 con 0x55959e43c5b0 2024-04-27T03:18:14.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94101093769504] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94101093774704] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94101093775760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e448980 con 0x55959e43c5b0 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e449260 con 0x7f2adc014030 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94101093780912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e448980 con 0x7f2adc014030 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e449d10 con 0x55959e43c5b0 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.305+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94101093775760] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94101093780912] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94101093781968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e449d10 con 0x55959e43c5b0 2024-04-27T03:18:14.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e44a100 con 0x55959e441710 2024-04-27T03:18:14.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94101093778576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e44a950 con 0x55959e441710 2024-04-27T03:18:14.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e44b350 con 0x55959e441710 2024-04-27T03:18:14.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94101093781968] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94101093783584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e44b310 con 0x55959e441710 2024-04-27T03:18:14.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e44bde0 con 0x7f2adc014030 2024-04-27T03:18:14.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.309+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94101093778576] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94101093783584] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'153 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94101093791920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e44ae60 con 0x7f2adc014030 2024-04-27T03:18:14.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=0 2024-04-27T03:18:14.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=1 2024-04-27T03:18:14.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=2 2024-04-27T03:18:14.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=3 2024-04-27T03:18:14.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=4 2024-04-27T03:18:14.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=5 2024-04-27T03:18:14.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=6 2024-04-27T03:18:14.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94101093791920] v27'154 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 add_watcher() i=7 2024-04-27T03:18:14.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:14.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e44ae60 con 0x55959e4382a0 2024-04-27T03:18:14.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94101093786784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e44ae60 con 0x55959e4382a0 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94101093786784] v27'154 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2998fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.313+0000 7f2ae4631980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 5 note: GC not initialized 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f29137fe640 20 reqs_thread_entry: start 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:14.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 Filter name: none 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f29117fa640 20 reqs_thread_entry: start 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e472500 con 0x55959e4382a0 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e472020 con 0x7f2adc014030 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 foo.buckets [call out=183b] v0'0 uv26 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 disabling bucket name=myfoo 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:14.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:91b71841:root::myfoo:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e472f70 con 0x7f2adc014030 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(64 myfoo [read 0~165 out=165b] v0'0 uv27 ondisk = 0) ==== 149+0+165 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x1 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e474850 con 0x7f2adc014030 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(65 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv7 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:14.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~379 in=379b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e47d420 con 0x7f2adc014030 2024-04-27T03:18:14.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(66 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [delete,create,call,call,writefull 0~379,setxattr (123)] v27'8 uv8 ondisk = 0) ==== 416+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:14.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.317+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [notify cookie 94101093971248 in=764b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e477e00 con 0x55959e43c5b0 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94101093775760 notify 115964117021 ret 0) ==== 794+0+0 (crc 0 0 0) 0x55959e4300d0 con 0x55959e43c5b0 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [notify cookie 94101093971248 out=8b] v0'0 uv153 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117021 cookie 94101093775760 notifier 4583 bl.length()=752 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f29a8003eb0 con 0x55959e43c5b0 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.2 [notify-ack] v0'0 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94101093971248 notify 115964117021 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55959e42d3f0 con 0x55959e43c5b0 2024-04-27T03:18:14.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_sources= new_sources= 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential sources added=[] removed=[] 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_dests= new_dests= 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential dests added=[] removed=[] 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e476450 con 0x55959e4382a0 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 foo [delete,create,call,call,writefull 0~391] v27'12 uv12 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:14.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [notify cookie 94101093944064 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55959e4713d0 con 0x55959e43c5b0 2024-04-27T03:18:14.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94101093781968 notify 115964117022 ret 0) ==== 609+0+0 (crc 0 0 0) 0x55959e42d3f0 con 0x55959e43c5b0 2024-04-27T03:18:14.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117022 cookie 94101093781968 notifier 4583 bl.length()=567 2024-04-27T03:18:14.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:14.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:14.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f29c5ffb640 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f29a8003eb0 con 0x55959e43c5b0 2024-04-27T03:18:14.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.321+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(70 notify.4 [notify cookie 94101093944064 out=8b] v0'0 uv153 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.325+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== watch-notify(notify_complete (2) cookie 94101093944064 notify 115964117022 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55959e42d160 con 0x55959e43c5b0 2024-04-27T03:18:14.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.325+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(71 notify.4 [notify-ack] v0'0 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.333 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":1,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:14.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.325+0000 7f2998fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:14.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.325+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94101093786784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x55959e4382a0 2024-04-27T03:18:14.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.325+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(72 data_loggenerations_metadata [watch unwatch cookie 94101093786784] v27'155 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.325+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94101093769504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x55959e43c5b0 2024-04-27T03:18:14.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.329+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(73 notify.0 [watch unwatch cookie 94101093769504] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.329+0000 7f2ae4631980 20 remove_watcher() i=0 2024-04-27T03:18:14.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.329+0000 7f2ae4631980 2 removed watcher, disabling cache 2024-04-27T03:18:14.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.329+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:74 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94101093774704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x55959e4382a0 2024-04-27T03:18:14.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.329+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(74 notify.1 [watch unwatch cookie 94101093774704] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x55959e4382a0 2024-04-27T03:18:14.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.329+0000 7f2ae4631980 20 remove_watcher() i=1 2024-04-27T03:18:14.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.329+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94101093775760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x55959e43c5b0 2024-04-27T03:18:14.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(75 notify.2 [watch unwatch cookie 94101093775760] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae4631980 20 remove_watcher() i=2 2024-04-27T03:18:14.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94101093780912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e447a00 con 0x7f2adc014030 2024-04-27T03:18:14.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(76 notify.3 [watch unwatch cookie 94101093780912] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae4631980 20 remove_watcher() i=3 2024-04-27T03:18:14.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94101093781968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x55959e43c5b0 2024-04-27T03:18:14.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae294d640 1 -- 172.21.15.110:0/2858645918 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(77 notify.4 [watch unwatch cookie 94101093781968] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2adc026020 con 0x55959e43c5b0 2024-04-27T03:18:14.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae4631980 20 remove_watcher() i=4 2024-04-27T03:18:14.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.333+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:78 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94101093778576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x55959e441710 2024-04-27T03:18:14.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(78 notify.5 [watch unwatch cookie 94101093778576] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 20 remove_watcher() i=5 2024-04-27T03:18:14.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94101093783584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x55959e441710 2024-04-27T03:18:14.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae214c640 1 -- 172.21.15.110:0/2858645918 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(79 notify.6 [watch unwatch cookie 94101093783584] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad4029030 con 0x55959e441710 2024-04-27T03:18:14.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 20 remove_watcher() i=6 2024-04-27T03:18:14.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:80 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94101093791920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55959e470230 con 0x7f2adc014030 2024-04-27T03:18:14.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(80 notify.7 [watch unwatch cookie 94101093791920] v27'155 uv153 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f2ad80b3040 con 0x7f2adc014030 2024-04-27T03:18:14.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 20 remove_watcher() i=7 2024-04-27T03:18:14.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e43c5b0 msgr2=0x55959e43e9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e43c5b0 0x55959e43e9e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e4382a0 msgr2=0x55959e43a740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e4382a0 0x55959e43a740 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2adc014030 msgr2=0x7f2adc03ba70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f2adc014030 0x7f2adc03ba70 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55959e441710 msgr2=0x55959e443b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55959e441710 0x55959e443b00 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29ac034650 msgr2=0x7f29ac036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29ac034650 0x7f29ac036b10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2ad400bc00 tx=0x7f2ad4035000 comp rx=0 tx=0).stop 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 msgr2=0x55959e42cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae194b640 1 -- 172.21.15.110:0/2858645918 reap_dead start 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e42cc20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0207a0 tx=0x7f2adc04a9a0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 shutdown_connections 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2858645918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e425770 0x55959e42cc20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 >> 172.21.15.110:0/2858645918 conn(0x55959e421ab0 msgr2=0x55959e426f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 shutdown_connections 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2858645918 wait complete. 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29bc0346a0 msgr2=0x7f29bc036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29bc0346a0 0x7f29bc036b60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55959e4050f0 tx=0x7f2ad4019e60 comp rx=0 tx=0).stop 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e418520 msgr2=0x55959e418940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e418520 0x55959e418940 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2adc00df90 tx=0x7f2adc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 shutdown_connections 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29bc0346a0 0x7f29bc036b60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/2533132901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e418520 0x55959e418940 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 >> 172.21.15.110:0/2533132901 conn(0x55959e40d330 msgr2=0x55959e40de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 shutdown_connections 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/2533132901 wait complete. 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e405930 msgr2=0x55959e407d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e405930 0x55959e407d20 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e30fa60 msgr2=0x55959e404560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e30fa60 0x55959e404560 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29c8034650 msgr2=0x7f29c8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29c8034650 0x7f29c8036b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2ad80ad000 tx=0x7f2ad80a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 msgr2=0x55959e3fd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3fd630 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x55959e3f7480 tx=0x7f2ad4000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 shutdown_connections 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55959e405930 0x55959e407d20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55959e30fa60 0x55959e404560 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f29c8034650 0x7f29c8036b10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 --2- 172.21.15.110:0/194513652 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55959e27d850 0x55959e3fd630 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 >> 172.21.15.110:0/194513652 conn(0x55959e303a20 msgr2=0x55959e3f0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 shutdown_connections 2024-04-27T03:18:14.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.337+0000 7f2ae4631980 1 -- 172.21.15.110:0/194513652 wait complete. 2024-04-27T03:18:14.356 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:14.358 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'enable', '--uid', 'foo'] 2024-04-27T03:18:14.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'enable', '--uid', 'foo'] 2024-04-27T03:18:14.358 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user enable --uid foo 2024-04-27T03:18:14.428 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:14.428 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa352df2980 1 Processor -- start 2024-04-27T03:18:14.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa352df2980 1 -- start start 2024-04-27T03:18:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa352df2980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5589951f59c0 0x5589951f7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa352df2980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5589950ae0d0 con 0x5589951f14b0 2024-04-27T03:18:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa352df2980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5589950a3430 con 0x5589951f59c0 2024-04-27T03:18:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa35010c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5589951f59c0 0x5589951f7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa35010c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5589951f59c0 0x5589951f7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45160/0 (socket says 172.21.15.110:45160) 2024-04-27T03:18:14.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa35010c640 1 -- 172.21.15.110:0/1380975919 learned_addr learned my addr 172.21.15.110:0/1380975919 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa35010c640 1 -- 172.21.15.110:0/1380975919 >> v1:172.21.15.77:6789/0 conn(0x5589951f14b0 legacy=0x5589951f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.421+0000 7fa35010c640 1 -- 172.21.15.110:0/1380975919 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55899509c510 con 0x5589951f59c0 2024-04-27T03:18:14.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa35010c640 1 --2- 172.21.15.110:0/1380975919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f59c0 0x5589951f7db0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa34400a5c0 tx=0x7fa34401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1f6001f93639f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa34f90b640 1 -- 172.21.15.110:0/1380975919 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3440a6070 con 0x5589951f59c0 2024-04-27T03:18:14.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa34f90b640 1 -- 172.21.15.110:0/1380975919 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3440af070 con 0x5589951f59c0 2024-04-27T03:18:14.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa34f90b640 1 -- 172.21.15.110:0/1380975919 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3440aa5c0 con 0x5589951f59c0 2024-04-27T03:18:14.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa352df2980 1 -- 172.21.15.110:0/1380975919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f59c0 msgr2=0x5589951f7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa352df2980 1 --2- 172.21.15.110:0/1380975919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f59c0 0x5589951f7db0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa34400a5c0 tx=0x7fa34401b340 comp rx=0 tx=0).stop 2024-04-27T03:18:14.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa352df2980 1 -- 172.21.15.110:0/1380975919 shutdown_connections 2024-04-27T03:18:14.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa352df2980 1 --2- 172.21.15.110:0/1380975919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f59c0 0x5589951f7db0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa352df2980 1 -- 172.21.15.110:0/1380975919 >> 172.21.15.110:0/1380975919 conn(0x558995164b90 msgr2=0x558995164fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa352df2980 1 -- 172.21.15.110:0/1380975919 shutdown_connections 2024-04-27T03:18:14.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.425+0000 7fa352df2980 1 -- 172.21.15.110:0/1380975919 wait complete. 2024-04-27T03:18:14.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.437+0000 7fa352df2980 1 Processor -- start 2024-04-27T03:18:14.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.437+0000 7fa352df2980 1 -- start start 2024-04-27T03:18:14.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.437+0000 7fa352df2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952dc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.437+0000 7fa352df2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589950f22d0 con 0x5589951f14b0 2024-04-27T03:18:14.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.437+0000 7fa35090d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952dc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.437+0000 7fa35090d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952dc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45170/0 (socket says 172.21.15.110:45170) 2024-04-27T03:18:14.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.437+0000 7fa35090d640 1 -- 172.21.15.110:0/1997889634 learned_addr learned my addr 172.21.15.110:0/1997889634 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa35090d640 1 -- 172.21.15.110:0/1997889634 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558995078ae0 con 0x5589951f14b0 2024-04-27T03:18:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa35090d640 1 --2- 172.21.15.110:0/1997889634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952dc5d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa3480015f0 tx=0x7fa348001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ccddd022baecbb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa23ee0d640 1 -- 172.21.15.110:0/1997889634 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa348016070 con 0x5589951f14b0 2024-04-27T03:18:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa23ee0d640 1 -- 172.21.15.110:0/1997889634 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34800f640 con 0x5589951f14b0 2024-04-27T03:18:14.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa23ee0d640 1 -- 172.21.15.110:0/1997889634 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa34800f960 con 0x5589951f14b0 2024-04-27T03:18:14.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- 172.21.15.110:0/1997889634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 msgr2=0x5589952dc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 --2- 172.21.15.110:0/1997889634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952dc5d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa3480015f0 tx=0x7fa348001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- 172.21.15.110:0/1997889634 shutdown_connections 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 --2- 172.21.15.110:0/1997889634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952dc5d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- 172.21.15.110:0/1997889634 >> 172.21.15.110:0/1997889634 conn(0x558995164b90 msgr2=0x5589952d8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- 172.21.15.110:0/1997889634 shutdown_connections 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- 172.21.15.110:0/1997889634 wait complete. 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 Processor -- start 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- start start 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952e3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589951fade0 con 0x5589951f14b0 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa35090d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952e3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa35090d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952e3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45184/0 (socket says 172.21.15.110:45184) 2024-04-27T03:18:14.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa35090d640 1 -- 172.21.15.110:0/64880863 learned_addr learned my addr 172.21.15.110:0/64880863 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa35090d640 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589952e3b50 con 0x5589951f14b0 2024-04-27T03:18:14.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa35090d640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952e3610 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa348001af0 tx=0x7fa348000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa23d60a640 1 -- 172.21.15.110:0/64880863 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa348016040 con 0x5589951f14b0 2024-04-27T03:18:14.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5589952e3de0 con 0x5589951f14b0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa23d60a640 1 -- 172.21.15.110:0/64880863 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa348022070 con 0x5589951f14b0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa23d60a640 1 -- 172.21.15.110:0/64880863 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa348006180 con 0x5589951f14b0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.441+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589952e6a60 con 0x5589951f14b0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa23d60a640 1 -- 172.21.15.110:0/64880863 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa3480117c0 con 0x5589951f14b0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa23d60a640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa2380346a0 0x7fa238036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa23d60a640 1 -- 172.21.15.110:0/64880863 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fa348049820 con 0x5589951f14b0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35010c640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa2380346a0 0x7fa238036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5589951f59c0 0x5589952ea540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589952eab10 con 0x5589951f59c0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35110e640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5589951f59c0 0x5589952ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35010c640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa2380346a0 0x7fa238036b60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa3440ae000 tx=0x7fa3440a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35110e640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5589951f59c0 0x5589952ea540 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35110e640 1 -- 172.21.15.110:0/64880863 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa3400083f0 con 0x5589951f59c0 2024-04-27T03:18:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589952eb8e0 0x5589952edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589952ee310 con 0x5589952eb8e0 2024-04-27T03:18:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35090d640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589952eb8e0 0x5589952edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35090d640 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589952eb8e0 0x5589952edcd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa35090d640 1 -- 172.21.15.110:0/64880863 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fa34800f070 con 0x5589952eb8e0 2024-04-27T03:18:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.445+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589952efbc0 con 0x5589951f59c0 2024-04-27T03:18:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35110e640 1 -- 172.21.15.110:0/64880863 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fa3400083f0 con 0x5589951f59c0 2024-04-27T03:18:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589952eedf0 con 0x5589952eb8e0 2024-04-27T03:18:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35090d640 1 -- 172.21.15.110:0/64880863 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fa34800f070 con 0x5589952eb8e0 2024-04-27T03:18:14.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589952ef1e0 con 0x5589951f59c0 2024-04-27T03:18:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35110e640 1 -- 172.21.15.110:0/64880863 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fa3400083f0 con 0x5589951f59c0 2024-04-27T03:18:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 Processor -- start 2024-04-27T03:18:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 -- start start 2024-04-27T03:18:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589952fe500 0x5589952fe920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589950f6310 con 0x5589952fe500 2024-04-27T03:18:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35110e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589952fe500 0x5589952fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35110e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589952fe500 0x5589952fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45190/0 (socket says 172.21.15.110:45190) 2024-04-27T03:18:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35110e640 1 -- 172.21.15.110:0/3368408895 learned_addr learned my addr 172.21.15.110:0/3368408895 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35110e640 1 -- 172.21.15.110:0/3368408895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589952fee60 con 0x5589952fe500 2024-04-27T03:18:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa35110e640 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589952fe500 0x5589952fe920 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa34000aab0 tx=0x7fa34000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa2367fc640 1 -- 172.21.15.110:0/3368408895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa340007d30 con 0x5589952fe500 2024-04-27T03:18:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5589952ff900 con 0x5589952fe500 2024-04-27T03:18:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589952ffdd0 con 0x5589952fe500 2024-04-27T03:18:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa2367fc640 1 -- 172.21.15.110:0/3368408895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34001d070 con 0x5589952fe500 2024-04-27T03:18:14.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.449+0000 7fa2367fc640 1 -- 172.21.15.110:0/3368408895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa340022460 con 0x5589952fe500 2024-04-27T03:18:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa2367fc640 1 -- 172.21.15.110:0/3368408895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa340013070 con 0x5589952fe500 2024-04-27T03:18:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa2367fc640 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa228034650 0x7fa228036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa2367fc640 1 -- 172.21.15.110:0/3368408895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fa34004b470 con 0x5589952fe500 2024-04-27T03:18:14.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa35090d640 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa228034650 0x7fa228036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 Processor -- start 2024-04-27T03:18:14.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- start start 2024-04-27T03:18:14.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa35090d640 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa228034650 0x7fa228036b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x5589952ee910 tx=0x7fa348047b00 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x55899530bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5589952dcd20 con 0x55899530b720 2024-04-27T03:18:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa35110e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x55899530bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa35110e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x55899530bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45204/0 (socket says 172.21.15.110:45204) 2024-04-27T03:18:14.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa35110e640 1 -- 172.21.15.110:0/2414361928 learned_addr learned my addr 172.21.15.110:0/2414361928 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa35110e640 1 -- 172.21.15.110:0/2414361928 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55899530c110 con 0x55899530b720 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa35110e640 1 --2- 172.21.15.110:0/2414361928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x55899530bb40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa340016650 tx=0x7fa3400163f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97797f14f554bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa227fff640 1 -- 172.21.15.110:0/2414361928 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa340041070 con 0x55899530b720 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa227fff640 1 -- 172.21.15.110:0/2414361928 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa340020780 con 0x55899530b720 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa227fff640 1 -- 172.21.15.110:0/2414361928 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa34003b120 con 0x55899530b720 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- 172.21.15.110:0/2414361928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 msgr2=0x55899530bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 --2- 172.21.15.110:0/2414361928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x55899530bb40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fa340016650 tx=0x7fa3400163f0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- 172.21.15.110:0/2414361928 shutdown_connections 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 --2- 172.21.15.110:0/2414361928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x55899530bb40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- 172.21.15.110:0/2414361928 >> 172.21.15.110:0/2414361928 conn(0x558995307a60 msgr2=0x558995305ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- 172.21.15.110:0/2414361928 shutdown_connections 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- 172.21.15.110:0/2414361928 wait complete. 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 Processor -- start 2024-04-27T03:18:14.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.453+0000 7fa352df2980 1 -- start start 2024-04-27T03:18:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa352df2980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x558995312b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa352df2980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55899530c7c0 con 0x55899530b720 2024-04-27T03:18:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa35110e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x558995312b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa35110e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x558995312b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45212/0 (socket says 172.21.15.110:45212) 2024-04-27T03:18:14.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 learned_addr learned my addr 172.21.15.110:0/456607337 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558995313050 con 0x55899530b720 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa35110e640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x558995312b10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa34003f040 tx=0x7fa340049750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa2267fc640 1 -- 172.21.15.110:0/456607337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa34002f070 con 0x55899530b720 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa2267fc640 1 -- 172.21.15.110:0/456607337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa340021c90 con 0x55899530b720 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5589953132e0 con 0x55899530b720 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa2267fc640 1 -- 172.21.15.110:0/456607337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa340041040 con 0x55899530b720 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558995315fc0 con 0x55899530b720 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa2267fc640 1 -- 172.21.15.110:0/456607337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa340032070 con 0x55899530b720 2024-04-27T03:18:14.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa2267fc640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa21c034650 0x7fa21c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa2267fc640 1 -- 172.21.15.110:0/456607337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fa340035c90 con 0x55899530b720 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa35090d640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa21c034650 0x7fa21c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55899531e1c0 0x558995320660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995320db0 con 0x55899531e1c0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.457+0000 7fa35010c640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55899531e1c0 0x558995320660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35090d640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa21c034650 0x7fa21c036b10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa348011c20 tx=0x7fa348034000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35010c640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55899531e1c0 0x558995320660 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 realm 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589953211e0 con 0x55899531e1c0 2024-04-27T03:18:14.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995321ab0 con 0x55899531e1c0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589953224d0 0x558995324900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995325070 con 0x5589953224d0 2024-04-27T03:18:14.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35110e640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589953224d0 0x558995324900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35110e640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589953224d0 0x558995324900 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:14.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589953258c0 con 0x55899531e1c0 2024-04-27T03:18:14.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 20 searching for the correct realm 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558995327630 0x558995329a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x558995327630 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35090d640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558995327630 0x558995329a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35090d640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558995327630 0x558995329a20 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.461+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35110e640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa340034070 0x7fa340044030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x7fa340034070 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa340034070 0x7fa340044030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:14.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa340034070 0x7fa340044030 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x558995327630 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x7fa340034070 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.465+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x7fa340034070 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x558995327630 2024-04-27T03:18:14.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x558995327630 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x558995327630 2024-04-27T03:18:14.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x7fa340034070 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x55899531e1c0 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x7fa340034070 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.469+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x558995327630 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x7fa340034070 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x558995327630 2024-04-27T03:18:14.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55899532a0a0 con 0x5589953224d0 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.473+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995326240 con 0x55899531e1c0 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995326ae0 con 0x5589953224d0 2024-04-27T03:18:14.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:14.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995325e80 con 0x55899531e1c0 2024-04-27T03:18:14.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:14.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 zone default found 2024-04-27T03:18:14.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 4 Realm: () 2024-04-27T03:18:14.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:14.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 zonegroup default 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55899532a9b0 con 0x7fa340034070 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55899532cc30 con 0x558995327630 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55899532cfd0 con 0x558995327630 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:14.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 started sync module instance, tier type = 2024-04-27T03:18:14.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:14.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.477+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532c070 con 0x5589953224d0 2024-04-27T03:18:14.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94049402012816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532d820 con 0x5589953224d0 2024-04-27T03:18:14.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532e070 con 0x55899531e1c0 2024-04-27T03:18:14.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94049402017824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532e190 con 0x55899531e1c0 2024-04-27T03:18:14.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532ecb0 con 0x5589953224d0 2024-04-27T03:18:14.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94049402012816] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.481+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94049402017824] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94049402018880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532ec70 con 0x5589953224d0 2024-04-27T03:18:14.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532f550 con 0x7fa340034070 2024-04-27T03:18:14.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94049402024096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532ec70 con 0x7fa340034070 2024-04-27T03:18:14.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532ffd0 con 0x5589953224d0 2024-04-27T03:18:14.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94049402024096] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94049402018880] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94049402021760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55899532ffd0 con 0x5589953224d0 2024-04-27T03:18:14.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5589953303c0 con 0x558995327630 2024-04-27T03:18:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94049402025152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995330c10 con 0x558995327630 2024-04-27T03:18:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.485+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5589953315d0 con 0x558995327630 2024-04-27T03:18:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94049402021760] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94049402025152] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94049402026720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5589953315d0 con 0x558995327630 2024-04-27T03:18:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995331ee0 con 0x7fa340034070 2024-04-27T03:18:14.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94049402026720] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'156 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94049402034672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5589953315d0 con 0x7fa340034070 2024-04-27T03:18:14.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=0 2024-04-27T03:18:14.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=1 2024-04-27T03:18:14.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=2 2024-04-27T03:18:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=3 2024-04-27T03:18:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=4 2024-04-27T03:18:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=5 2024-04-27T03:18:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=6 2024-04-27T03:18:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94049402034672] v27'157 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 20 add_watcher() i=7 2024-04-27T03:18:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995335330 con 0x55899531e1c0 2024-04-27T03:18:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.489+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94049402032880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995332850 con 0x55899531e1c0 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94049402032880] v27'156 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa206fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 5 note: GC not initialized 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa205fe3640 20 reqs_thread_entry: start 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 Filter name: none 2024-04-27T03:18:14.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa1777fe640 20 reqs_thread_entry: start 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x558995358b30 con 0x55899531e1c0 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv12 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55899535b5c0 con 0x7fa340034070 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 foo.buckets [call out=183b] v0'0 uv26 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 enabling bucket name=myfoo 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.9 8:91b71841:root::myfoo:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5589953595a0 con 0x7fa340034070 2024-04-27T03:18:14.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(64 myfoo [read 0~165 out=165b] v0'0 uv27 ondisk = 0) ==== 149+0+165 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x1 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 : miss 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rados->read ofs=0 len=0 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55899535be30 con 0x7fa340034070 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(65 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv8 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 2024-04-27T03:18:14.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.493+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 8.2 8:4397dfdd:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.1:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~379 in=379b,setxattr user.rgw.acl (123) in=135b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5589953639f0 con 0x7fa340034070 2024-04-27T03:18:14.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(66 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 [delete,create,call,call,writefull 0~379,setxattr (123)] v27'9 uv9 ondisk = 0) ==== 416+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:14.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 10 moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1, ofs0, ns] 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [notify cookie 94049402237584 in=764b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55899535db10 con 0x5589953224d0 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94049402018880 notify 115964117023 ret 0) ==== 794+0+0 (crc 0 0 0) 0x558995315fc0 con 0x5589953224d0 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa227fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117023 cookie 94049402018880 notifier 4590 bl.length()=752 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa227fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 info.flags=0x17 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa227fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1 to cache LRU end 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa227fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa227fff640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fa214003eb0 con 0x5589953224d0 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [notify cookie 94049402237584 out=8b] v0'0 uv156 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.2 [notify-ack] v0'0 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94049402237584 notify 115964117023 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5589953132e0 con 0x5589953224d0 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_sources= new_sources= 2024-04-27T03:18:14.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential sources added=[] removed=[] 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): orig_dests= new_dests= 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 20 handle_bi_update(): bucket=:myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.1]): potential dests added=[] removed=[] 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.497+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995362690 con 0x55899531e1c0 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 foo [delete,create,call,call,writefull 0~391] v27'13 uv13 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa352df2980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa352df2980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa352df2980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:14.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [notify cookie 94049402215184 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55899535e3e0 con 0x5589953224d0 2024-04-27T03:18:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94049402021760 notify 115964117024 ret 0) ==== 609+0+0 (crc 0 0 0) 0x5589953132e0 con 0x5589953224d0 2024-04-27T03:18:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa2277fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117024 cookie 94049402021760 notifier 4590 bl.length()=567 2024-04-27T03:18:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa2277fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa2277fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa2277fe640 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7fa184003440 con 0x5589953224d0 2024-04-27T03:18:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(70 notify.4 [notify cookie 94049402215184 out=8b] v0'0 uv156 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [notify-ack] v0'0 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== watch-notify(notify_complete (2) cookie 94049402215184 notify 115964117024 ret 0) ==== 42+0+48 (crc 0 0 0) 0x558995313050 con 0x5589953224d0 2024-04-27T03:18:14.511 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa206fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:14.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.501+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94049402032880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x55899531e1c0 2024-04-27T03:18:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.505+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(72 data_loggenerations_metadata [watch unwatch cookie 94049402032880] v27'157 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.505+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94049402012816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x5589953224d0 2024-04-27T03:18:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.505+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(73 notify.0 [watch unwatch cookie 94049402012816] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.505+0000 7fa352df2980 20 remove_watcher() i=0 2024-04-27T03:18:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.505+0000 7fa352df2980 2 removed watcher, disabling cache 2024-04-27T03:18:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.505+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:74 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94049402017824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x55899531e1c0 2024-04-27T03:18:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(74 notify.1 [watch unwatch cookie 94049402017824] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x55899531e1c0 2024-04-27T03:18:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa352df2980 20 remove_watcher() i=1 2024-04-27T03:18:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94049402018880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x5589953224d0 2024-04-27T03:18:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(75 notify.2 [watch unwatch cookie 94049402018880] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa352df2980 20 remove_watcher() i=2 2024-04-27T03:18:14.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94049402024096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x7fa340034070 2024-04-27T03:18:14.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(76 notify.3 [watch unwatch cookie 94049402024096] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa352df2980 20 remove_watcher() i=3 2024-04-27T03:18:14.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.509+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94049402021760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x5589953224d0 2024-04-27T03:18:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa35110e640 1 -- 172.21.15.110:0/456607337 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(77 notify.4 [watch unwatch cookie 94049402021760] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa340053040 con 0x5589953224d0 2024-04-27T03:18:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa352df2980 20 remove_watcher() i=4 2024-04-27T03:18:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:78 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94049402025152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x558995327630 2024-04-27T03:18:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(78 notify.5 [watch unwatch cookie 94049402025152] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa352df2980 20 remove_watcher() i=5 2024-04-27T03:18:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94049402026720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x558995327630 2024-04-27T03:18:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa35090d640 1 -- 172.21.15.110:0/456607337 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(79 notify.6 [watch unwatch cookie 94049402026720] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa34800f070 con 0x558995327630 2024-04-27T03:18:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa352df2980 20 remove_watcher() i=6 2024-04-27T03:18:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.513+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:80 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94049402034672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x558995356860 con 0x7fa340034070 2024-04-27T03:18:14.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(80 notify.7 [watch unwatch cookie 94049402034672] v27'158 uv156 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fa3440aaa20 con 0x7fa340034070 2024-04-27T03:18:14.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 20 remove_watcher() i=7 2024-04-27T03:18:14.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589953224d0 msgr2=0x558995324900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589953224d0 0x558995324900 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55899531e1c0 msgr2=0x558995320660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55899531e1c0 0x558995320660 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa340034070 msgr2=0x7fa340044030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fa340034070 0x7fa340044030 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558995327630 msgr2=0x558995329a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558995327630 0x558995329a20 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa21c034650 msgr2=0x7fa21c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa21c034650 0x7fa21c036b10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa348011c20 tx=0x7fa348034000 comp rx=0 tx=0).stop 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 msgr2=0x558995312b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x558995312b10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fa34003f040 tx=0x7fa340049750 comp rx=0 tx=0).stop 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa35010c640 1 -- 172.21.15.110:0/456607337 reap_dead start 2024-04-27T03:18:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 shutdown_connections 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/456607337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55899530b720 0x558995312b10 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 >> 172.21.15.110:0/456607337 conn(0x558995307a60 msgr2=0x55899530cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 shutdown_connections 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/456607337 wait complete. 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa228034650 msgr2=0x7fa228036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa228034650 0x7fa228036b10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x5589952ee910 tx=0x7fa348047b00 comp rx=0 tx=0).stop 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589952fe500 msgr2=0x5589952fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589952fe500 0x5589952fe920 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa34000aab0 tx=0x7fa34000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 shutdown_connections 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa228034650 0x7fa228036b10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/3368408895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589952fe500 0x5589952fe920 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 >> 172.21.15.110:0/3368408895 conn(0x5589952f3310 msgr2=0x5589952f3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 shutdown_connections 2024-04-27T03:18:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/3368408895 wait complete. 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589952eb8e0 msgr2=0x5589952edcd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589952eb8e0 0x5589952edcd0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5589951f59c0 msgr2=0x5589952ea540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5589951f59c0 0x5589952ea540 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa2380346a0 msgr2=0x7fa238036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa2380346a0 0x7fa238036b60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa3440ae000 tx=0x7fa3440a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 msgr2=0x5589952e3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952e3610 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa348001af0 tx=0x7fa348000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 shutdown_connections 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5589952eb8e0 0x5589952edcd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5589951f59c0 0x5589952ea540 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fa2380346a0 0x7fa238036b60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 --2- 172.21.15.110:0/64880863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5589951f14b0 0x5589952e3610 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 >> 172.21.15.110:0/64880863 conn(0x558995164b90 msgr2=0x5589952d6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 shutdown_connections 2024-04-27T03:18:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:14.517+0000 7fa352df2980 1 -- 172.21.15.110:0/64880863 wait complete. 2024-04-27T03:18:14.534 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:14.543 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2024-04-27T03:18:14.880 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=30000000 bytes_in=0 failed=False 2024-04-27T03:18:14.895 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:29.896 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'list', '--include-all'] 2024-04-27T03:18:29.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'list', '--include-all'] 2024-04-27T03:18:29.897 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc list --include-all 2024-04-27T03:18:29.933 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:29.933 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.924+0000 7f0863f0a980 1 Processor -- start 2024-04-27T03:18:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.924+0000 7f0863f0a980 1 -- start start 2024-04-27T03:18:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0863f0a980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55b61d2cd4b0 0x55b61d2cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0863f0a980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55b61d18a0d0 con 0x55b61d2cddd0 2024-04-27T03:18:29.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0863f0a980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55b61d17f430 con 0x55b61d2cd4b0 2024-04-27T03:18:29.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0861a25640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55b61d2cd4b0 0x55b61d2cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0861224640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55b61d2cddd0 0x55b61d2ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51188/0 (socket says 172.21.15.110:51188) 2024-04-27T03:18:29.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0861224640 1 -- 172.21.15.110:0/4059779399 learned_addr learned my addr 172.21.15.110:0/4059779399 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:29.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0861a25640 1 -- 172.21.15.110:0/4059779399 >> v1:172.21.15.77:6789/0 conn(0x55b61d2cddd0 legacy=0x55b61d2ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:29.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0861a25640 1 -- 172.21.15.110:0/4059779399 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b61d178510 con 0x55b61d2cd4b0 2024-04-27T03:18:29.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.928+0000 7f0861a25640 1 --2- 172.21.15.110:0/4059779399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d2cd890 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f085400a480 tx=0x7f085401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8307fdc2f72d3a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0860a23640 1 -- 172.21.15.110:0/4059779399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08540b0070 con 0x55b61d2cd4b0 2024-04-27T03:18:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0860a23640 1 -- 172.21.15.110:0/4059779399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08540ab540 con 0x55b61d2cd4b0 2024-04-27T03:18:29.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0860a23640 1 -- 172.21.15.110:0/4059779399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08540ab840 con 0x55b61d2cd4b0 2024-04-27T03:18:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0863f0a980 1 -- 172.21.15.110:0/4059779399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 msgr2=0x55b61d2cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0863f0a980 1 --2- 172.21.15.110:0/4059779399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d2cd890 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f085400a480 tx=0x7f085401b280 comp rx=0 tx=0).stop 2024-04-27T03:18:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0863f0a980 1 -- 172.21.15.110:0/4059779399 shutdown_connections 2024-04-27T03:18:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0863f0a980 1 --2- 172.21.15.110:0/4059779399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d2cd890 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0863f0a980 1 -- 172.21.15.110:0/4059779399 >> 172.21.15.110:0/4059779399 conn(0x55b61d240b90 msgr2=0x55b61d240fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0863f0a980 1 -- 172.21.15.110:0/4059779399 shutdown_connections 2024-04-27T03:18:29.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.932+0000 7f0863f0a980 1 -- 172.21.15.110:0/4059779399 wait complete. 2024-04-27T03:18:29.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.944+0000 7f0863f0a980 1 Processor -- start 2024-04-27T03:18:29.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.944+0000 7f0863f0a980 1 -- start start 2024-04-27T03:18:29.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b61d1ce2d0 con 0x55b61d2cd4b0 2024-04-27T03:18:29.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40734/0 (socket says 172.21.15.110:40734) 2024-04-27T03:18:29.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 -- 172.21.15.110:0/3720447133 learned_addr learned my addr 172.21.15.110:0/3720447133 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:29.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 -- 172.21.15.110:0/3720447133 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b61d154ae0 con 0x55b61d2cd4b0 2024-04-27T03:18:29.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 --2- 172.21.15.110:0/3720447133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3b8d70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f08540ae040 tx=0x7f0854003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bbb57d2e23ce29 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f074fe07640 1 -- 172.21.15.110:0/3720447133 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08540b0040 con 0x55b61d2cd4b0 2024-04-27T03:18:29.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f074fe07640 1 -- 172.21.15.110:0/3720447133 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08540ab9d0 con 0x55b61d2cd4b0 2024-04-27T03:18:29.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f074fe07640 1 -- 172.21.15.110:0/3720447133 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08540abcf0 con 0x55b61d2cd4b0 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- 172.21.15.110:0/3720447133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 msgr2=0x55b61d3b8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3720447133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3b8d70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f08540ae040 tx=0x7f0854003030 comp rx=0 tx=0).stop 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- 172.21.15.110:0/3720447133 shutdown_connections 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3720447133 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3b8d70 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- 172.21.15.110:0/3720447133 >> 172.21.15.110:0/3720447133 conn(0x55b61d240b90 msgr2=0x55b61d3b4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- 172.21.15.110:0/3720447133 shutdown_connections 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- 172.21.15.110:0/3720447133 wait complete. 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 Processor -- start 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- start start 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3bfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0863f0a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b61d2d7500 con 0x55b61d2cd4b0 2024-04-27T03:18:29.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3bfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3bfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40742/0 (socket says 172.21.15.110:40742) 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.948+0000 7f0861a25640 1 -- 172.21.15.110:0/2949003308 learned_addr learned my addr 172.21.15.110:0/2949003308 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0861a25640 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b61d3c02c0 con 0x55b61d2cd4b0 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0861a25640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3bfd80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f085401b2c0 tx=0x7f0854018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f074e604640 1 -- 172.21.15.110:0/2949003308 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08540b0050 con 0x55b61d2cd4b0 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f074e604640 1 -- 172.21.15.110:0/2949003308 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08540b6870 con 0x55b61d2cd4b0 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b61d3c0550 con 0x55b61d2cd4b0 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f074e604640 1 -- 172.21.15.110:0/2949003308 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08540b6b50 con 0x55b61d2cd4b0 2024-04-27T03:18:29.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b61d3c3180 con 0x55b61d2cd4b0 2024-04-27T03:18:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f074e604640 1 -- 172.21.15.110:0/2949003308 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f08540b7420 con 0x55b61d2cd4b0 2024-04-27T03:18:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f074e604640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0748034650 0x7f0748036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f074e604640 1 -- 172.21.15.110:0/2949003308 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f08540bc070 con 0x55b61d2cd4b0 2024-04-27T03:18:29.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0861224640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0748034650 0x7f0748036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d2cddd0 0x55b61d3c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3c71e0 con 0x55b61d2cddd0 2024-04-27T03:18:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0862226640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d2cddd0 0x55b61d3c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0861224640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0748034650 0x7f0748036b10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f08580012f0 tx=0x7f0858006700 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0862226640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d2cddd0 0x55b61d3c6c60 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0862226640 1 -- 172.21.15.110:0/2949003308 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f085c0083f0 con 0x55b61d2cddd0 2024-04-27T03:18:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3c7fb0 0x55b61d3ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3ca9e0 con 0x55b61d3c7fb0 2024-04-27T03:18:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.952+0000 7f0861a25640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3c7fb0 0x55b61d3ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0861a25640 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3c7fb0 0x55b61d3ca3a0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0861a25640 1 -- 172.21.15.110:0/2949003308 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d3c7fb0 2024-04-27T03:18:29.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3cc290 con 0x55b61d2cddd0 2024-04-27T03:18:29.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0862226640 1 -- 172.21.15.110:0/2949003308 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f085c0083f0 con 0x55b61d2cddd0 2024-04-27T03:18:29.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3cb4c0 con 0x55b61d3c7fb0 2024-04-27T03:18:29.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0861a25640 1 -- 172.21.15.110:0/2949003308 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d3c7fb0 2024-04-27T03:18:29.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3cb8b0 con 0x55b61d2cddd0 2024-04-27T03:18:29.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0862226640 1 -- 172.21.15.110:0/2949003308 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f085c0083f0 con 0x55b61d2cddd0 2024-04-27T03:18:29.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0863f0a980 1 Processor -- start 2024-04-27T03:18:29.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0863f0a980 1 -- start start 2024-04-27T03:18:29.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0863f0a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3daba0 0x55b61d3dafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0863f0a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b61d1d2310 con 0x55b61d3daba0 2024-04-27T03:18:29.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0862226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3daba0 0x55b61d3dafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0862226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3daba0 0x55b61d3dafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40758/0 (socket says 172.21.15.110:40758) 2024-04-27T03:18:29.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.956+0000 7f0862226640 1 -- 172.21.15.110:0/776270202 learned_addr learned my addr 172.21.15.110:0/776270202 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:29.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0862226640 1 -- 172.21.15.110:0/776270202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b61d3db500 con 0x55b61d3daba0 2024-04-27T03:18:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0862226640 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3daba0 0x55b61d3dafc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f085c00aab0 tx=0x7f085c00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f07477fe640 1 -- 172.21.15.110:0/776270202 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c007d30 con 0x55b61d3daba0 2024-04-27T03:18:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f07477fe640 1 -- 172.21.15.110:0/776270202 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c01d070 con 0x55b61d3daba0 2024-04-27T03:18:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b61d3dbfa0 con 0x55b61d3daba0 2024-04-27T03:18:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f07477fe640 1 -- 172.21.15.110:0/776270202 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c018440 con 0x55b61d3daba0 2024-04-27T03:18:29.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b61d3dc470 con 0x55b61d3daba0 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f07477fe640 1 -- 172.21.15.110:0/776270202 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f085c013070 con 0x55b61d3daba0 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f07477fe640 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f073c0346a0 0x7f073c036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f07477fe640 1 -- 172.21.15.110:0/776270202 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f085c04b490 con 0x55b61d3daba0 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0861a25640 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f073c0346a0 0x7f073c036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0863f0a980 1 Processor -- start 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0863f0a980 1 -- start start 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0861a25640 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f073c0346a0 0x7f073c036b60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x55b61d3cafe0 tx=0x7f08540e2de0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0863f0a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3e81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0863f0a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b61d3b9400 con 0x55b61d3e7da0 2024-04-27T03:18:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0862226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3e81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0862226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3e81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40764/0 (socket says 172.21.15.110:40764) 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.960+0000 7f0862226640 1 -- 172.21.15.110:0/1104105135 learned_addr learned my addr 172.21.15.110:0/1104105135 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0862226640 1 -- 172.21.15.110:0/1104105135 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b61d3e8790 con 0x55b61d3e7da0 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0862226640 1 --2- 172.21.15.110:0/1104105135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3e81c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f085c0498a0 tx=0x7f085c018930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97685064a54d530c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f07457fa640 1 -- 172.21.15.110:0/1104105135 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c047070 con 0x55b61d3e7da0 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f07457fa640 1 -- 172.21.15.110:0/1104105135 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c018af0 con 0x55b61d3e7da0 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f07457fa640 1 -- 172.21.15.110:0/1104105135 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c04b0d0 con 0x55b61d3e7da0 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- 172.21.15.110:0/1104105135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 msgr2=0x55b61d3e81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 --2- 172.21.15.110:0/1104105135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3e81c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f085c0498a0 tx=0x7f085c018930 comp rx=0 tx=0).stop 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- 172.21.15.110:0/1104105135 shutdown_connections 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 --2- 172.21.15.110:0/1104105135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3e81c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- 172.21.15.110:0/1104105135 >> 172.21.15.110:0/1104105135 conn(0x55b61d3e40e0 msgr2=0x55b61d3e2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:29.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- 172.21.15.110:0/1104105135 shutdown_connections 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- 172.21.15.110:0/1104105135 wait complete. 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 Processor -- start 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- start start 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b61d3e8eb0 con 0x55b61d3e7da0 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0862226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0862226640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3ef2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40772/0 (socket says 172.21.15.110:40772) 2024-04-27T03:18:29.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 learned_addr learned my addr 172.21.15.110:0/3726386053 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:29.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b61d3ef800 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0862226640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3ef2c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f085c022770 tx=0x7f085c049830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f07337fe640 1 -- 172.21.15.110:0/3726386053 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c031070 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f07337fe640 1 -- 172.21.15.110:0/3726386053 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c016d60 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b61d3efa90 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f07337fe640 1 -- 172.21.15.110:0/3726386053 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f085c020860 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.964+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b61d3f2770 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f07337fe640 1 -- 172.21.15.110:0/3726386053 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f085c034070 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f07337fe640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f072c034650 0x7f072c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f07337fe640 1 -- 172.21.15.110:0/3726386053 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f085c037c90 con 0x55b61d3e7da0 2024-04-27T03:18:29.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0861a25640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f072c034650 0x7f072c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d3fa940 0x55b61d3fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3fd530 con 0x55b61d3fa940 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0861224640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d3fa940 0x55b61d3fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0861a25640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f072c034650 0x7f072c036b10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f08540aaa00 tx=0x7f08540e1000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0861224640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d3fa940 0x55b61d3fcde0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 realm 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3f9e10 con 0x55b61d3fa940 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:29.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d3fe030 con 0x55b61d3fa940 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3fec10 0x55b61d4010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d401840 con 0x55b61d3fec10 2024-04-27T03:18:29.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0862226640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3fec10 0x55b61d4010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.968+0000 7f0862226640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3fec10 0x55b61d4010d0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0863f0a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4020e0 con 0x55b61d3fa940 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0863f0a980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0863f0a980 20 searching for the correct realm 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b61d403e50 0x55b61d406240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d403e50 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861a25640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b61d403e50 0x55b61d406240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861a25640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b61d403e50 0x55b61d406240 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:29.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0862226640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f085c036070 0x7f085c021950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x7f085c036070 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861224640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f085c036070 0x7f085c021950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861224640 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f085c036070 0x7f085c021950 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x7f085c036070 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.972+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d403e50 2024-04-27T03:18:29.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:29.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x7f085c036070 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x7f085c036070 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x7f085c036070 2024-04-27T03:18:29.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f085800f070 con 0x7f085c036070 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.976+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d403e50 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d403e50 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:29.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d403e50 2024-04-27T03:18:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:29.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x7f085c036070 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x7f085c036070 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fa940 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x7f085c036070 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.980+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x7f085c036070 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d403e50 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x7f085c036070 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085800f070 con 0x7f085c036070 2024-04-27T03:18:29.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d403e50 2024-04-27T03:18:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:29.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55b61d406820 con 0x55b61d3fec10 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.984+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d402a60 con 0x55b61d3fa940 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f085800f070 con 0x55b61d3fa940 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d402eb0 con 0x55b61d3fec10 2024-04-27T03:18:29.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4032d0 con 0x55b61d3fa940 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 zone default found 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 4 Realm: () 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 zonegroup default 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4026c0 con 0x7f085c036070 2024-04-27T03:18:30.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d409060 con 0x55b61d403e50 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d409400 con 0x55b61d403e50 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 started sync module instance, tier type = 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:30.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.988+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4084f0 con 0x55b61d3fec10 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94240663177488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d409ca0 con 0x55b61d3fec10 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40a4f0 con 0x55b61d3fa940 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94240663182496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40a650 con 0x55b61d3fa940 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40b170 con 0x55b61d3fec10 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94240663177488] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.992+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94240663182496] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94240663186496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40b130 con 0x55b61d3fec10 2024-04-27T03:18:30.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40ba10 con 0x7f085c036070 2024-04-27T03:18:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94240663188832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40b130 con 0x7f085c036070 2024-04-27T03:18:30.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40c4a0 con 0x55b61d3fec10 2024-04-27T03:18:30.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94240663188832] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94240663186496] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94240663181296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40c4a0 con 0x55b61d3fec10 2024-04-27T03:18:30.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:29.996+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40c8c0 con 0x55b61d403e50 2024-04-27T03:18:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94240663189888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40d110 con 0x55b61d403e50 2024-04-27T03:18:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40dad0 con 0x55b61d403e50 2024-04-27T03:18:30.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94240663181296] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94240663191504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40dad0 con 0x55b61d403e50 2024-04-27T03:18:30.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40e510 con 0x7f085c036070 2024-04-27T03:18:30.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94240663189888] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.000+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94240663191504] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'159 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94240663199712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40d620 con 0x7f085c036070 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=0 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=1 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=2 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=3 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=4 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=5 2024-04-27T03:18:30.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=6 2024-04-27T03:18:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94240663199712] v27'160 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 20 add_watcher() i=7 2024-04-27T03:18:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40d620 con 0x55b61d3fa940 2024-04-27T03:18:30.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.004+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94240663194720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40d620 con 0x55b61d3fa940 2024-04-27T03:18:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94240663194720] v27'158 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f07137fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0863f0a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0863f0a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:30.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d419c60 con 0x55b61d3fec10 2024-04-27T03:18:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'48 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d419c60 con 0x7f085c036070 2024-04-27T03:18:30.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'25 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.008+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41a1f0 con 0x55b61d403e50 2024-04-27T03:18:30.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.012+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'25 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.012+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41a680 con 0x55b61d3fa940 2024-04-27T03:18:30.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.012+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'159 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.016+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415ac0 con 0x55b61d3fec10 2024-04-27T03:18:30.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.016+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'52 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.016+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415ac0 con 0x55b61d403e50 2024-04-27T03:18:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.016+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'41 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.016+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415ac0 con 0x55b61d3fec10 2024-04-27T03:18:30.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.020+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'27 uv26 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.020+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415ac0 con 0x55b61d3fa940 2024-04-27T03:18:30.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.020+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'36 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.020+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fec10 2024-04-27T03:18:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.024+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'18 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.024+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fec10 2024-04-27T03:18:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.028+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'24 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.028+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fec10 2024-04-27T03:18:30.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.028+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'53 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.028+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d403e50 2024-04-27T03:18:30.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.028+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'21 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.028+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fec10 2024-04-27T03:18:30.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.032+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'24 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.032+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fa940 2024-04-27T03:18:30.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.032+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'47 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.032+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fa940 2024-04-27T03:18:30.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.036+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'160 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.036+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fec10 2024-04-27T03:18:30.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.040+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'44 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.040+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d3fec10 2024-04-27T03:18:30.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.040+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'54 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.040+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d415c90 con 0x55b61d403e50 2024-04-27T03:18:30.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.044+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'31 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.044+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fec10 2024-04-27T03:18:30.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.044+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'26 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.044+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fa940 2024-04-27T03:18:30.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.048+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'161 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.048+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fec10 2024-04-27T03:18:30.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.052+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'45 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.052+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fec10 2024-04-27T03:18:30.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.052+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'34 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.052+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d403e50 2024-04-27T03:18:30.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.056+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'26 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.056+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fec10 2024-04-27T03:18:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.056+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'36 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.056+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d403e50 2024-04-27T03:18:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.060+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'20 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.060+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d403e50 2024-04-27T03:18:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.060+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'42 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.060+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d403e50 2024-04-27T03:18:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.064+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'26 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.064+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fec10 2024-04-27T03:18:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.068+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'30 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.068+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fa940 2024-04-27T03:18:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.068+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'37 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.068+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fec10 2024-04-27T03:18:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.072+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'46 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.072+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x55b61d3fa940 2024-04-27T03:18:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.072+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'48 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.072+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d41b3d0 con 0x7f085c036070 2024-04-27T03:18:30.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'24 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f07127fc640 20 reqs_thread_entry: start 2024-04-27T03:18:30.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0863f0a980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:30.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0863f0a980 20 Filter name: none 2024-04-27T03:18:30.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0683fff640 20 reqs_thread_entry: start 2024-04-27T03:18:30.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:94 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(94 gc.0 [call out=15b] v0'0 uv17 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 6.19 6:9a6dd8c0:gc::gc.0:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.076+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(95 gc.0 [call out=24b] v0'0 uv17 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(96 gc.0 [call out=15b] v0'0 uv17 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:97 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x7f085c036070 2024-04-27T03:18:30.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(97 gc.1 [call out=15b] v0'0 uv11 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 6.18 6:1e85421f:gc::gc.1:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x7f085c036070 2024-04-27T03:18:30.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(98 gc.1 [call out=24b] v0'0 uv11 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:99 6.18 6:1e85421f:gc::gc.1:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x7f085c036070 2024-04-27T03:18:30.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(99 gc.1 [call out=15b] v0'0 uv11 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d403e50 2024-04-27T03:18:30.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(100 gc.2 [call out=15b] v0'0 uv7 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 6.b 6:d242335b:gc::gc.2:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d403e50 2024-04-27T03:18:30.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(101 gc.2 [call out=24b] v0'0 uv7 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 6.b 6:d242335b:gc::gc.2:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d403e50 2024-04-27T03:18:30.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(102 gc.2 [call out=15b] v0'0 uv7 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fa940 2024-04-27T03:18:30.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(103 gc.3 [call out=15b] v0'0 uv17 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.080+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:104 6.4 6:2166d8e1:gc::gc.3:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fa940 2024-04-27T03:18:30.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(104 gc.3 [call out=24b] v0'0 uv17 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 6.4 6:2166d8e1:gc::gc.3:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fa940 2024-04-27T03:18:30.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(105 gc.3 [call out=15b] v0'0 uv17 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(106 gc.4 [call out=15b] v0'0 uv19 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:107 6.3 6:c60da066:gc::gc.4:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(107 gc.4 [call out=24b] v0'0 uv19 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 6.3 6:c60da066:gc::gc.4:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(108 gc.4 [call out=15b] v0'0 uv19 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:109 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d403e50 2024-04-27T03:18:30.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(109 gc.5 [call out=15b] v0'0 uv20 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:110 6.9 6:954be8d2:gc::gc.5:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d403e50 2024-04-27T03:18:30.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(110 gc.5 [call out=24b] v0'0 uv20 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:111 6.9 6:954be8d2:gc::gc.5:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d403e50 2024-04-27T03:18:30.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(111 gc.5 [call out=15b] v0'0 uv20 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.084+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:112 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(112 gc.6 [call out=15b] v0'0 uv26 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:113 6.e 6:748722ff:gc::gc.6:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(113 gc.6 [call out=24b] v0'0 uv26 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:114 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d418e30 con 0x55b61d3fec10 2024-04-27T03:18:30.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(114 gc.6 [call out=1702b] v0'0 uv26 ondisk = 0) ==== 148+0+1702 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:115 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(115 gc.7 [call out=15b] v0'0 uv15 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:116 6.1d 6:beaa4be4:gc::gc.7:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fa940 2024-04-27T03:18:30.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(116 gc.7 [call out=24b] v0'0 uv15 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:117 6.1d 6:beaa4be4:gc::gc.7:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(117 gc.7 [call out=15b] v0'0 uv15 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.088+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:118 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(118 gc.8 [call out=15b] v0'0 uv3 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:119 6.12 6:4f78e9a2:gc::gc.8:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(119 gc.8 [call out=24b] v0'0 uv3 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:120 6.12 6:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== osd_op_reply(120 gc.8 [call out=15b] v0'0 uv3 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:121 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 45 ==== osd_op_reply(121 gc.9 [call out=15b] v0'0 uv9 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:122 6.1c 6:3e426232:gc::gc.9:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 46 ==== osd_op_reply(122 gc.9 [call out=24b] v0'0 uv9 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.092+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:123 6.1c 6:3e426232:gc::gc.9:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 47 ==== osd_op_reply(123 gc.9 [call out=15b] v0'0 uv9 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:124 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 48 ==== osd_op_reply(124 gc.10 [call out=15b] v0'0 uv21 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:125 6.3 6:c31f41d7:gc::gc.10:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 49 ==== osd_op_reply(125 gc.10 [call out=24b] v0'0 uv21 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:126 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 50 ==== osd_op_reply(126 gc.10 [call out=15b] v0'0 uv21 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:127 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(127 gc.11 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:128 6.13 6:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d403e50 2024-04-27T03:18:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(128 gc.11 [call out=24b] v0'0 uv7 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:129 6.13 6:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(129 gc.11 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.096+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:130 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 51 ==== osd_op_reply(130 gc.12 [call out=15b] v0'0 uv11 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:131 6.15 6:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 52 ==== osd_op_reply(131 gc.12 [call out=24b] v0'0 uv11 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:132 6.15 6:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 53 ==== osd_op_reply(132 gc.12 [call out=15b] v0'0 uv11 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:133 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(133 gc.13 [call out=15b] v0'0 uv20 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:134 6.0 6:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fa940 2024-04-27T03:18:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(134 gc.13 [call out=24b] v0'0 uv20 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:135 6.0 6:0667bb13:gc::gc.13:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(135 gc.13 [call out=15b] v0'0 uv20 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:136 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(136 gc.14 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.100+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:137 6.4 6:227f63b5:gc::gc.14:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fa940 2024-04-27T03:18:30.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(137 gc.14 [call out=24b] v0'0 uv7 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:138 6.4 6:227f63b5:gc::gc.14:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(138 gc.14 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:139 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 54 ==== osd_op_reply(139 gc.15 [call out=15b] v0'0 uv5 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:140 6.7 6:e07a6548:gc::gc.15:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 55 ==== osd_op_reply(140 gc.15 [call out=24b] v0'0 uv5 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:141 6.7 6:e07a6548:gc::gc.15:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 56 ==== osd_op_reply(141 gc.15 [call out=15b] v0'0 uv5 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.104+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:142 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 57 ==== osd_op_reply(142 gc.16 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:143 6.3 6:c623b173:gc::gc.16:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 58 ==== osd_op_reply(143 gc.16 [call out=24b] v0'0 uv7 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:144 6.3 6:c623b173:gc::gc.16:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 59 ==== osd_op_reply(144 gc.16 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:145 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(145 gc.17 [call out=15b] v0'0 uv5 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:146 6.16 6:6849fd06:gc::gc.17:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d403e50 2024-04-27T03:18:30.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(146 gc.17 [call out=24b] v0'0 uv5 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:147 6.16 6:6849fd06:gc::gc.17:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== osd_op_reply(147 gc.17 [call out=15b] v0'0 uv5 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.108+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:148 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 60 ==== osd_op_reply(148 gc.18 [call out=15b] v0'0 uv3 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:149 6.d 6:b019ea83:gc::gc.18:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 61 ==== osd_op_reply(149 gc.18 [call out=24b] v0'0 uv3 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:150 6.d 6:b019ea83:gc::gc.18:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 62 ==== osd_op_reply(150 gc.18 [call out=15b] v0'0 uv3 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:151 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(151 gc.19 [call out=15b] v0'0 uv11 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:152 6.4 6:24c8e1ae:gc::gc.19:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fa940 2024-04-27T03:18:30.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(152 gc.19 [call out=24b] v0'0 uv11 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:153 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 42 ==== osd_op_reply(153 gc.19 [call out=15b] v0'0 uv11 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:154 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 63 ==== osd_op_reply(154 gc.20 [call out=15b] v0'0 uv9 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.112+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:155 6.7 6:e622f76a:gc::gc.20:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 64 ==== osd_op_reply(155 gc.20 [call out=24b] v0'0 uv9 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:156 6.7 6:e622f76a:gc::gc.20:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 65 ==== osd_op_reply(156 gc.20 [call out=15b] v0'0 uv9 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:157 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 66 ==== osd_op_reply(157 gc.21 [call out=15b] v0'0 uv9 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:158 6.a 6:5127e642:gc::gc.21:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 67 ==== osd_op_reply(158 gc.21 [call out=24b] v0'0 uv9 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:159 6.a 6:5127e642:gc::gc.21:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 68 ==== osd_op_reply(159 gc.21 [call out=15b] v0'0 uv9 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.116+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:160 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(160 gc.22 [call out=15b] v0'0 uv3 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:161 6.11 6:8eb99c53:gc::gc.22:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d403e50 2024-04-27T03:18:30.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(161 gc.22 [call out=24b] v0'0 uv3 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:162 6.11 6:8eb99c53:gc::gc.22:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== osd_op_reply(162 gc.22 [call out=15b] v0'0 uv3 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:163 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 69 ==== osd_op_reply(163 gc.23 [call out=15b] v0'0 uv9 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:164 6.8 6:17d7e8ed:gc::gc.23:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 70 ==== osd_op_reply(164 gc.23 [call out=24b] v0'0 uv9 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:165 6.8 6:17d7e8ed:gc::gc.23:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 71 ==== osd_op_reply(165 gc.23 [call out=15b] v0'0 uv9 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:166 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(166 gc.24 [call out=15b] v0'0 uv3 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:167 6.1b 6:db9d130d:gc::gc.24:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d403e50 2024-04-27T03:18:30.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== osd_op_reply(167 gc.24 [call out=24b] v0'0 uv3 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.120+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:168 6.1b 6:db9d130d:gc::gc.24:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== osd_op_reply(168 gc.24 [call out=15b] v0'0 uv3 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:169 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== osd_op_reply(169 gc.25 [call out=15b] v0'0 uv8 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:170 6.9 6:921ecdb1:gc::gc.25:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d403e50 2024-04-27T03:18:30.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== osd_op_reply(170 gc.25 [call out=24b] v0'0 uv8 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:171 6.9 6:921ecdb1:gc::gc.25:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== osd_op_reply(171 gc.25 [call out=15b] v0'0 uv8 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:172 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== osd_op_reply(172 gc.26 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:173 6.1f 6:fde4dd55:gc::gc.26:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d403e50 2024-04-27T03:18:30.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(173 gc.26 [call out=24b] v0'0 uv7 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:174 6.1f 6:fde4dd55:gc::gc.26:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d403e50 2024-04-27T03:18:30.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== osd_op_reply(174 gc.26 [call out=15b] v0'0 uv7 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:175 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 72 ==== osd_op_reply(175 gc.27 [call out=15b] v0'0 uv11 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.124+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:176 6.1a 6:5909e671:gc::gc.27:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 73 ==== osd_op_reply(176 gc.27 [call out=24b] v0'0 uv11 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:177 6.1a 6:5909e671:gc::gc.27:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 74 ==== osd_op_reply(177 gc.27 [call out=15b] v0'0 uv11 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:178 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 43 ==== osd_op_reply(178 gc.28 [call out=15b] v0'0 uv6 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:179 6.1d 6:b88e3271:gc::gc.28:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fa940 2024-04-27T03:18:30.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 44 ==== osd_op_reply(179 gc.28 [call out=24b] v0'0 uv6 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:180 6.1d 6:b88e3271:gc::gc.28:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 45 ==== osd_op_reply(180 gc.28 [call out=15b] v0'0 uv6 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:181 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 75 ==== osd_op_reply(181 gc.29 [call out=15b] v0'0 uv15 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:182 6.7 6:e47c9322:gc::gc.29:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fec10 2024-04-27T03:18:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 76 ==== osd_op_reply(182 gc.29 [call out=24b] v0'0 uv15 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.128+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:183 6.7 6:e47c9322:gc::gc.29:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fec10 2024-04-27T03:18:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 77 ==== osd_op_reply(183 gc.29 [call out=15b] v0'0 uv15 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:184 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 46 ==== osd_op_reply(184 gc.30 [call out=15b] v0'0 uv8 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:185 6.0 6:013d6c33:gc::gc.30:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x55b61d3fa940 2024-04-27T03:18:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 47 ==== osd_op_reply(185 gc.30 [call out=24b] v0'0 uv8 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:186 6.0 6:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x55b61d3fa940 2024-04-27T03:18:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 48 ==== osd_op_reply(186 gc.30 [call out=15b] v0'0 uv8 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:187 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x7f085c036070 2024-04-27T03:18:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(187 gc.31 [call out=15b] v0'0 uv5 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:188 6.f 6:f24cb4a8:gc::gc.31:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4368b0 con 0x7f085c036070 2024-04-27T03:18:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(188 gc.31 [call out=24b] v0'0 uv5 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:189 6.f 6:f24cb4a8:gc::gc.31:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4378c0 con 0x7f085c036070 2024-04-27T03:18:30.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.132+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(189 gc.31 [call out=15b] v0'0 uv5 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.136+0000 7f07137fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:30.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.136+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:190 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94240663194720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d436400 con 0x55b61d3fa940 2024-04-27T03:18:30.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.136+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 49 ==== osd_op_reply(190 data_loggenerations_metadata [watch unwatch cookie 94240663194720] v27'162 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.136+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:191 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94240663177488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40cc60 con 0x55b61d3fec10 2024-04-27T03:18:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.140+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 78 ==== osd_op_reply(191 notify.0 [watch unwatch cookie 94240663177488] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.140+0000 7f0863f0a980 20 remove_watcher() i=0 2024-04-27T03:18:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.140+0000 7f0863f0a980 2 removed watcher, disabling cache 2024-04-27T03:18:30.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.140+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:192 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94240663182496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40cc60 con 0x55b61d3fa940 2024-04-27T03:18:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.140+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.1 v2:172.21.15.110:6800/1860462708 50 ==== osd_op_reply(192 notify.1 [watch unwatch cookie 94240663182496] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x55b61d3fa940 2024-04-27T03:18:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.140+0000 7f0863f0a980 20 remove_watcher() i=1 2024-04-27T03:18:30.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.140+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:193 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94240663186496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40cc60 con 0x55b61d3fec10 2024-04-27T03:18:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.144+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 79 ==== osd_op_reply(193 notify.2 [watch unwatch cookie 94240663186496] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.144+0000 7f0863f0a980 20 remove_watcher() i=2 2024-04-27T03:18:30.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.144+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:194 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94240663188832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40cc60 con 0x7f085c036070 2024-04-27T03:18:30.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.144+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(194 notify.3 [watch unwatch cookie 94240663188832] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.144+0000 7f0863f0a980 20 remove_watcher() i=3 2024-04-27T03:18:30.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.144+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:195 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94240663181296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40b560 con 0x55b61d3fec10 2024-04-27T03:18:30.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0862226640 1 -- 172.21.15.110:0/3726386053 <== osd.0 v2:172.21.15.77:6800/3496324319 80 ==== osd_op_reply(195 notify.4 [watch unwatch cookie 94240663181296] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f085c033030 con 0x55b61d3fec10 2024-04-27T03:18:30.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0863f0a980 20 remove_watcher() i=4 2024-04-27T03:18:30.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:196 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94240663189888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40cc60 con 0x55b61d403e50 2024-04-27T03:18:30.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 46 ==== osd_op_reply(196 notify.5 [watch unwatch cookie 94240663189888] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0863f0a980 20 remove_watcher() i=5 2024-04-27T03:18:30.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:197 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94240663191504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d4097f0 con 0x55b61d403e50 2024-04-27T03:18:30.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0861a25640 1 -- 172.21.15.110:0/3726386053 <== osd.3 v2:172.21.15.110:6816/3916647200 47 ==== osd_op_reply(197 notify.6 [watch unwatch cookie 94240663191504] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08540c7020 con 0x55b61d403e50 2024-04-27T03:18:30.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0863f0a980 20 remove_watcher() i=6 2024-04-27T03:18:30.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.148+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:198 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94240663199712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55b61d40cc60 con 0x7f085c036070 2024-04-27T03:18:30.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(198 notify.7 [watch unwatch cookie 94240663199712] v27'161 uv159 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f08580401c0 con 0x7f085c036070 2024-04-27T03:18:30.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 20 remove_watcher() i=7 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3fec10 msgr2=0x55b61d4010d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3fec10 0x55b61d4010d0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d3fa940 msgr2=0x55b61d3fcde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d3fa940 0x55b61d3fcde0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f085c036070 msgr2=0x7f085c021950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f085c036070 0x7f085c021950 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b61d403e50 msgr2=0x55b61d406240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b61d403e50 0x55b61d406240 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f072c034650 msgr2=0x7f072c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f072c034650 0x7f072c036b10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f08540aaa00 tx=0x7f08540e1000 comp rx=0 tx=0).stop 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0861224640 1 -- 172.21.15.110:0/3726386053 reap_dead start 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 msgr2=0x55b61d3ef2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3ef2c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f085c022770 tx=0x7f085c049830 comp rx=0 tx=0).stop 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 shutdown_connections 2024-04-27T03:18:30.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/3726386053 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3e7da0 0x55b61d3ef2c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 >> 172.21.15.110:0/3726386053 conn(0x55b61d3e40e0 msgr2=0x55b61d3e1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 shutdown_connections 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/3726386053 wait complete. 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f073c0346a0 msgr2=0x7f073c036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f073c0346a0 0x7f073c036b60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x55b61d3cafe0 tx=0x7f08540e2de0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3daba0 msgr2=0x55b61d3dafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3daba0 0x55b61d3dafc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f085c00aab0 tx=0x7f085c00af70 comp rx=0 tx=0).stop 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 shutdown_connections 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f073c0346a0 0x7f073c036b60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/776270202 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d3daba0 0x55b61d3dafc0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 >> 172.21.15.110:0/776270202 conn(0x55b61d3cf9b0 msgr2=0x55b61d3d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 shutdown_connections 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/776270202 wait complete. 2024-04-27T03:18:30.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3c7fb0 msgr2=0x55b61d3ca3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3c7fb0 0x55b61d3ca3a0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d2cddd0 msgr2=0x55b61d3c6c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d2cddd0 0x55b61d3c6c60 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0748034650 msgr2=0x7f0748036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0748034650 0x7f0748036b10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f08580012f0 tx=0x7f0858006700 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 msgr2=0x55b61d3bfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3bfd80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f085401b2c0 tx=0x7f0854018020 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 shutdown_connections 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b61d3c7fb0 0x55b61d3ca3a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b61d2cddd0 0x55b61d3c6c60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0748034650 0x7f0748036b10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 --2- 172.21.15.110:0/2949003308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b61d2cd4b0 0x55b61d3bfd80 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 >> 172.21.15.110:0/2949003308 conn(0x55b61d240b90 msgr2=0x55b61d3b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 shutdown_connections 2024-04-27T03:18:30.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.152+0000 7f0863f0a980 1 -- 172.21.15.110:0/2949003308 wait complete. 2024-04-27T03:18:30.174 INFO:teuthology.orchestra.run.smithi110.stdout:[{"tag":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.16249673198333708825\u0000","time":"2024-04-27T03:18:29.889059+0000","objs":[{"pool":"default.rgw.buckets.data","oid":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_1","key":"","instance":""},{"pool":"default.rgw.buckets.data","oid":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_2","key":"","instance":""},{"pool":"default.rgw.buckets.data","oid":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_3","key":"","instance":""},{"pool":"default.rgw.buckets.data","oid":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_4","key":"","instance":""},{"pool":"default.rgw.buckets.data","oid":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_5","key":"","instance":""},{"pool":"default.rgw.buckets.data","oid":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_6","key":"","instance":""},{"pool":"default.rgw.buckets.data","oid":"27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_7","key":"","instance":""}]}] 2024-04-27T03:18:30.174 DEBUG:tasks.util.rgw: json result: [{'tag': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.16249673198333708825\x00', 'time': '2024-04-27T03:18:29.889059+0000', 'objs': [{'pool': 'default.rgw.buckets.data', 'oid': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_1', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_2', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_3', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_4', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_5', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_6', 'key': '', 'instance': ''}, {'pool': 'default.rgw.buckets.data', 'oid': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_7', 'key': '', 'instance': ''}]}] 2024-04-27T03:18:30.174 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'process'] 2024-04-27T03:18:30.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'process'] 2024-04-27T03:18:30.175 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process 2024-04-27T03:18:30.246 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:30.246 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:30.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed9d81980 1 Processor -- start 2024-04-27T03:18:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed9d81980 1 -- start start 2024-04-27T03:18:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed9d81980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a4465c5280 0x55a4465c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed9d81980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a4464840d0 con 0x55a4465c5ba0 2024-04-27T03:18:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed9d81980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a446479430 con 0x55a4465c5280 2024-04-27T03:18:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed789c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a4465c5280 0x55a4465c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed709b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55a4465c5ba0 0x55a4465c6560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51198/0 (socket says 172.21.15.110:51198) 2024-04-27T03:18:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed709b640 1 -- 172.21.15.110:0/3927504613 learned_addr learned my addr 172.21.15.110:0/3927504613 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:30.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed789c640 1 -- 172.21.15.110:0/3927504613 >> v1:172.21.15.77:6789/0 conn(0x55a4465c5ba0 legacy=0x55a4465c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.240+0000 7f5ed789c640 1 -- 172.21.15.110:0/3927504613 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a446472510 con 0x55a4465c5280 2024-04-27T03:18:30.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3927504613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4465c5660 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000a420 tx=0x7f5ed001afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2658887ae36ecd8a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed689a640 1 -- 172.21.15.110:0/3927504613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed00af070 con 0x55a4465c5280 2024-04-27T03:18:30.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed689a640 1 -- 172.21.15.110:0/3927504613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed00aa530 con 0x55a4465c5280 2024-04-27T03:18:30.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed689a640 1 -- 172.21.15.110:0/3927504613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed00aa850 con 0x55a4465c5280 2024-04-27T03:18:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3927504613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 msgr2=0x55a4465c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3927504613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4465c5660 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5ed000a420 tx=0x7f5ed001afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3927504613 shutdown_connections 2024-04-27T03:18:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3927504613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4465c5660 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3927504613 >> 172.21.15.110:0/3927504613 conn(0x55a4465c0b50 msgr2=0x55a4465c2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3927504613 shutdown_connections 2024-04-27T03:18:30.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.244+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3927504613 wait complete. 2024-04-27T03:18:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed9d81980 1 Processor -- start 2024-04-27T03:18:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed9d81980 1 -- start start 2024-04-27T03:18:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed9d81980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4466b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed9d81980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a4464c82d0 con 0x55a4465c5280 2024-04-27T03:18:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed789c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4466b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed789c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4466b0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40794/0 (socket says 172.21.15.110:40794) 2024-04-27T03:18:30.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed789c640 1 -- 172.21.15.110:0/2171171237 learned_addr learned my addr 172.21.15.110:0/2171171237 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed789c640 1 -- 172.21.15.110:0/2171171237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a44644eae0 con 0x55a4465c5280 2024-04-27T03:18:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5ed789c640 1 --2- 172.21.15.110:0/2171171237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4466b0b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00ad040 tx=0x7f5ed00033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6386cb9ebf65a0d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5dc5e0b640 1 -- 172.21.15.110:0/2171171237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed00af050 con 0x55a4465c5280 2024-04-27T03:18:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5dc5e0b640 1 -- 172.21.15.110:0/2171171237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed00b0040 con 0x55a4465c5280 2024-04-27T03:18:30.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.260+0000 7f5dc5e0b640 1 -- 172.21.15.110:0/2171171237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed00aaca0 con 0x55a4465c5280 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- 172.21.15.110:0/2171171237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 msgr2=0x55a4466b0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/2171171237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4466b0b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00ad040 tx=0x7f5ed00033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- 172.21.15.110:0/2171171237 shutdown_connections 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/2171171237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5280 0x55a4466b0b60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- 172.21.15.110:0/2171171237 >> 172.21.15.110:0/2171171237 conn(0x55a4465c0b50 msgr2=0x55a4466ac6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- 172.21.15.110:0/2171171237 shutdown_connections 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- 172.21.15.110:0/2171171237 wait complete. 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 Processor -- start 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- start start 2024-04-27T03:18:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5ba0 0x55a4466b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a4465cf2d0 con 0x55a4465c5ba0 2024-04-27T03:18:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed789c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5ba0 0x55a4466b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed789c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5ba0 0x55a4466b7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40810/0 (socket says 172.21.15.110:40810) 2024-04-27T03:18:30.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed789c640 1 -- 172.21.15.110:0/3880842008 learned_addr learned my addr 172.21.15.110:0/3880842008 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed789c640 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4466b8210 con 0x55a4465c5ba0 2024-04-27T03:18:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5ba0 0x55a4466b7cd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0000f90 tx=0x7f5ed0019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5dbffff640 1 -- 172.21.15.110:0/3880842008 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed00c15b0 con 0x55a4465c5ba0 2024-04-27T03:18:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a4466b84a0 con 0x55a4465c5ba0 2024-04-27T03:18:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5dbffff640 1 -- 172.21.15.110:0/3880842008 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ed00c1750 con 0x55a4465c5ba0 2024-04-27T03:18:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5dbffff640 1 -- 172.21.15.110:0/3880842008 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ed00c1a70 con 0x55a4465c5ba0 2024-04-27T03:18:30.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4466bb0d0 con 0x55a4465c5ba0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5dbffff640 1 -- 172.21.15.110:0/3880842008 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5ed00c1c10 con 0x55a4465c5ba0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.264+0000 7f5dbffff640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5dc0034650 0x7f5dc0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5dbffff640 1 -- 172.21.15.110:0/3880842008 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5ed00bb070 con 0x55a4465c5ba0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed709b640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5dc0034650 0x7f5dc0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466bcba0 0x55a4466bef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466bf560 con 0x55a4466bcba0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed809d640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466bcba0 0x55a4466bef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed709b640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5dc0034650 0x7f5dc0036b10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc000f70 tx=0x7f5ecc0012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed809d640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466bcba0 0x55a4466bef90 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed809d640 1 -- 172.21.15.110:0/3880842008 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5ec80083f0 con 0x55a4466bcba0 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466c0330 0x55a4466c2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466c2d50 con 0x55a4466c0330 2024-04-27T03:18:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466c0330 0x55a4466c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466c0330 0x55a4466c2720 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed789c640 1 -- 172.21.15.110:0/3880842008 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466c0330 2024-04-27T03:18:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466c4600 con 0x55a4466bcba0 2024-04-27T03:18:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed809d640 1 -- 172.21.15.110:0/3880842008 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f5ec80083f0 con 0x55a4466bcba0 2024-04-27T03:18:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466c3740 con 0x55a4466c0330 2024-04-27T03:18:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.268+0000 7f5ed789c640 1 -- 172.21.15.110:0/3880842008 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466c0330 2024-04-27T03:18:30.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466c3b60 con 0x55a4466bcba0 2024-04-27T03:18:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed809d640 1 -- 172.21.15.110:0/3880842008 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f5ec80083f0 con 0x55a4466bcba0 2024-04-27T03:18:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed9d81980 1 Processor -- start 2024-04-27T03:18:30.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed9d81980 1 -- start start 2024-04-27T03:18:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed9d81980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a44653baa0 0x55a446539f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed9d81980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a4464cc310 con 0x55a44653baa0 2024-04-27T03:18:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed809d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a44653baa0 0x55a446539f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed809d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a44653baa0 0x55a446539f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40822/0 (socket says 172.21.15.110:40822) 2024-04-27T03:18:30.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed809d640 1 -- 172.21.15.110:0/90298851 learned_addr learned my addr 172.21.15.110:0/90298851 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed809d640 1 -- 172.21.15.110:0/90298851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a44653a480 con 0x55a44653baa0 2024-04-27T03:18:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed809d640 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a44653baa0 0x55a446539f40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800aab0 tx=0x7f5ec800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5dbd7fa640 1 -- 172.21.15.110:0/90298851 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ec8007d30 con 0x55a44653baa0 2024-04-27T03:18:30.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5dbd7fa640 1 -- 172.21.15.110:0/90298851 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec8018440 con 0x55a44653baa0 2024-04-27T03:18:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a44653a770 con 0x55a44653baa0 2024-04-27T03:18:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5dbd7fa640 1 -- 172.21.15.110:0/90298851 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ec8018760 con 0x55a44653baa0 2024-04-27T03:18:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a44653ac40 con 0x55a44653baa0 2024-04-27T03:18:30.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.272+0000 7f5dbd7fa640 1 -- 172.21.15.110:0/90298851 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5ec8013070 con 0x55a44653baa0 2024-04-27T03:18:30.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5dbd7fa640 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5db0034650 0x7f5db0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5dbd7fa640 1 -- 172.21.15.110:0/90298851 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5ec804b720 con 0x55a44653baa0 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed789c640 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5db0034650 0x7f5db0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 Processor -- start 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- start start 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed789c640 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5db0034650 0x7f5db0036b10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55a4466bfab0 tx=0x7f5ed00033b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a4466b11f0 con 0x55a4466e22d0 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40832/0 (socket says 172.21.15.110:40832) 2024-04-27T03:18:30.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 -- 172.21.15.110:0/3919633952 learned_addr learned my addr 172.21.15.110:0/3919633952 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 -- 172.21.15.110:0/3919633952 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4466e2cc0 con 0x55a4466e22d0 2024-04-27T03:18:30.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 --2- 172.21.15.110:0/3919633952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e26f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8015020 tx=0x7f5ec8016bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db34e4386e16532c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5db6ffd640 1 -- 172.21.15.110:0/3919633952 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ec8040070 con 0x55a4466e22d0 2024-04-27T03:18:30.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5db6ffd640 1 -- 172.21.15.110:0/3919633952 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec8016330 con 0x55a4466e22d0 2024-04-27T03:18:30.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5db6ffd640 1 -- 172.21.15.110:0/3919633952 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ec8021c70 con 0x55a4466e22d0 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3919633952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 msgr2=0x55a4466e26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3919633952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e26f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8015020 tx=0x7f5ec8016bc0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3919633952 shutdown_connections 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3919633952 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e26f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3919633952 >> 172.21.15.110:0/3919633952 conn(0x55a4466de610 msgr2=0x55a4466dc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3919633952 shutdown_connections 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3919633952 wait complete. 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 Processor -- start 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- start start 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed9d81980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a4466e3440 con 0x55a4466e22d0 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40848/0 (socket says 172.21.15.110:40848) 2024-04-27T03:18:30.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.276+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 learned_addr learned my addr 172.21.15.110:0/3125917484 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4466e9cd0 con 0x55a4466e22d0 2024-04-27T03:18:30.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed809d640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e9790 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8016f60 tx=0x7f5ec8049ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5db57fa640 1 -- 172.21.15.110:0/3125917484 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ec8040040 con 0x55a4466e22d0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5db57fa640 1 -- 172.21.15.110:0/3125917484 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec804b300 con 0x55a4466e22d0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a4466e9f60 con 0x55a4466e22d0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5db57fa640 1 -- 172.21.15.110:0/3125917484 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ec8018e60 con 0x55a4466e22d0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4466ecc40 con 0x55a4466e22d0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5db57fa640 1 -- 172.21.15.110:0/3125917484 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5ec8033020 con 0x55a4466e22d0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5db57fa640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5da00346a0 0x7f5da0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5db57fa640 1 -- 172.21.15.110:0/3125917484 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5ec8031020 con 0x55a4466e22d0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5da00346a0 0x7f5da0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466f4de0 0x55a4466f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466f79f0 con 0x55a4466f4de0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed709b640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466f4de0 0x55a4466f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5da00346a0 0x7f5da0036b60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0008030 tx=0x7f5ed00db000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed709b640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466f4de0 0x55a4466f72a0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5ecc00e5a0 con 0x55a4466f4de0 2024-04-27T03:18:30.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 realm 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466f4230 con 0x55a4466f4de0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5ecc00e5a0 con 0x55a4466f4de0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466f84f0 con 0x55a4466f4de0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5ecc00e5a0 con 0x55a4466f4de0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466f92d0 0x55a4466fb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.280+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466fbd70 con 0x55a4466f92d0 2024-04-27T03:18:30.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed809d640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466f92d0 0x55a4466fb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed809d640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466f92d0 0x55a4466fb6c0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed9d81980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466fc5c0 con 0x55a4466f4de0 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f5ecc00e5a0 con 0x55a4466f4de0 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed9d81980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed9d81980 20 searching for the correct realm 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a4466fe330 0x55a446700720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466fe330 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a4466fe330 0x55a446700720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed789c640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a4466fe330 0x55a446700720 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.284+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed809d640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5ec8022c20 0x7f5ec8021370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x7f5ec8022c20 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5ec8022c20 0x7f5ec8021370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5ec8022c20 0x7f5ec8021370 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc00e5a0 con 0x7f5ec8022c20 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc00e5a0 con 0x55a4466f4de0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466fe330 2024-04-27T03:18:30.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.288+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x7f5ec8022c20 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x7f5ec8022c20 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x7f5ec8022c20 2024-04-27T03:18:30.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f5ecc029040 con 0x7f5ec8022c20 2024-04-27T03:18:30.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466fe330 2024-04-27T03:18:30.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466fe330 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.292+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466fe330 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x7f5ec8022c20 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x7f5ec8022c20 2024-04-27T03:18:30.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f4de0 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x7f5ec8022c20 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x7f5ec8022c20 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466fe330 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x7f5ec8022c20 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ecc029040 con 0x7f5ec8022c20 2024-04-27T03:18:30.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466fe330 2024-04-27T03:18:30.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.296+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a446700da0 con 0x55a4466f92d0 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466fcf70 con 0x55a4466f4de0 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5ecc029040 con 0x55a4466f4de0 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466fd810 con 0x55a4466f92d0 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a4466fcbb0 con 0x55a4466f4de0 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 zone default found 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 4 Realm: () 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:30.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 zonegroup default 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446701200 con 0x7f5ec8022c20 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446702c40 con 0x55a4466fe330 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446703110 con 0x55a4466fe330 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 started sync module instance, tier type = 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:30.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.300+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446702080 con 0x55a4466f92d0 2024-04-27T03:18:30.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.304+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.304+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94164044752288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446702080 con 0x55a4466f92d0 2024-04-27T03:18:30.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.304+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446703bf0 con 0x55a4466f4de0 2024-04-27T03:18:30.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.304+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.304+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94164044756704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446703bb0 con 0x55a4466f4de0 2024-04-27T03:18:30.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.304+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446704af0 con 0x55a4466f92d0 2024-04-27T03:18:30.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94164044752288] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94164044756704] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94164044759008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446704600 con 0x55a4466f92d0 2024-04-27T03:18:30.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467053c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94164044762896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446704600 con 0x7f5ec8022c20 2024-04-27T03:18:30.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.308+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446705e00 con 0x55a4466f92d0 2024-04-27T03:18:30.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94164044762896] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94164044759008] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94164044749776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446705950 con 0x55a4466f92d0 2024-04-27T03:18:30.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467066d0 con 0x55a4466fe330 2024-04-27T03:18:30.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94164044767776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446705950 con 0x55a4466fe330 2024-04-27T03:18:30.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446707430 con 0x55a4466fe330 2024-04-27T03:18:30.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94164044749776] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94164044771104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467073f0 con 0x55a4466fe330 2024-04-27T03:18:30.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446707ec0 con 0x7f5ec8022c20 2024-04-27T03:18:30.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.312+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94164044767776] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'162 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94164044773872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446707ec0 con 0x7f5ec8022c20 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=0 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=1 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=2 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=3 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=4 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=5 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94164044771104] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=6 2024-04-27T03:18:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94164044773872] v27'163 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 20 add_watcher() i=7 2024-04-27T03:18:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446705330 con 0x55a4466f4de0 2024-04-27T03:18:30.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.316+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94164044763952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446708350 con 0x55a4466f4de0 2024-04-27T03:18:30.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.320+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94164044763952] v27'163 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.320+0000 7f5d8dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:30.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.320+0000 7f5ed9d81980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:30.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.320+0000 7f5ed9d81980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:30.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.320+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467138e0 con 0x55a4466f92d0 2024-04-27T03:18:30.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.320+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'49 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.320+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467139e0 con 0x7f5ec8022c20 2024-04-27T03:18:30.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.324+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'26 uv11 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.324+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446714180 con 0x55a4466fe330 2024-04-27T03:18:30.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.324+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'26 uv7 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.324+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446714630 con 0x55a4466f4de0 2024-04-27T03:18:30.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.328+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'164 uv17 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.328+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44670f6b0 con 0x55a4466f92d0 2024-04-27T03:18:30.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.328+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'55 uv19 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.328+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44670f670 con 0x55a4466fe330 2024-04-27T03:18:30.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.328+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'43 uv20 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.328+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44670f640 con 0x55a4466f92d0 2024-04-27T03:18:30.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.332+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'28 uv26 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.332+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44670f810 con 0x55a4466f4de0 2024-04-27T03:18:30.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.332+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'38 uv15 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.332+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715190 con 0x55a4466f92d0 2024-04-27T03:18:30.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.336+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'19 uv3 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.336+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715190 con 0x55a4466f92d0 2024-04-27T03:18:30.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.340+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'25 uv9 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.340+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715190 con 0x55a4466f92d0 2024-04-27T03:18:30.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.340+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'56 uv21 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.340+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715190 con 0x55a4466fe330 2024-04-27T03:18:30.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.344+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'22 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.344+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715190 con 0x55a4466f92d0 2024-04-27T03:18:30.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.344+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'25 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.344+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715290 con 0x55a4466f4de0 2024-04-27T03:18:30.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.348+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'49 uv20 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.348+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f4de0 2024-04-27T03:18:30.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.348+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'165 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.348+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.352+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'47 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.352+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.352+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'57 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.352+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466fe330 2024-04-27T03:18:30.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.356+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'32 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.356+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.356+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'27 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.356+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f4de0 2024-04-27T03:18:30.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.360+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'166 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.360+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.360+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'48 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.360+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.364+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'35 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.364+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466fe330 2024-04-27T03:18:30.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.364+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'27 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.364+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.368+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'37 uv9 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.368+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466fe330 2024-04-27T03:18:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.372+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'21 uv3 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.372+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466fe330 2024-04-27T03:18:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.372+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'44 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.372+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466fe330 2024-04-27T03:18:30.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.372+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'27 uv7 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.372+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.376+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'31 uv11 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.376+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f4de0 2024-04-27T03:18:30.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.380+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'39 uv6 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.380+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f92d0 2024-04-27T03:18:30.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.380+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'49 uv15 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.380+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x55a4466f4de0 2024-04-27T03:18:30.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'50 uv8 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446715720 con 0x7f5ec8022c20 2024-04-27T03:18:30.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'25 uv5 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5d8cfe1640 20 reqs_thread_entry: start 2024-04-27T03:18:30.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5ed9d81980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:30.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5ed9d81980 20 Filter name: none 2024-04-27T03:18:30.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5d067fc640 20 reqs_thread_entry: start 2024-04-27T03:18:30.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=1 2024-04-27T03:18:30.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.384+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:94 6.7 6:e47c9322:gc::gc.29:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(94 gc.29 [call] v27'50 uv50 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(95 gc.29 [call out=15b] v0'0 uv50 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 6.7 6:e47c9322:gc::gc.29:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f92d0 2024-04-27T03:18:30.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(96 gc.29 [call out=24b] v0'0 uv50 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(97 gc.29 [call out=15b] v0'0 uv50 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.388+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.392+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 6.7 6:e47c9322:gc::gc.29:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.392+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(98 gc.29 [call out=15b] v0'0 uv50 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.392+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.392+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 6.7 6:e47c9322:gc::gc.29:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.392+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(99 gc.29 [call] v27'51 uv51 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.392+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=1 2024-04-27T03:18:30.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.392+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:100 6.0 6:013d6c33:gc::gc.30:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(100 gc.30 [call] v27'51 uv51 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:101 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(101 gc.30 [call out=15b] v0'0 uv51 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:102 6.0 6:013d6c33:gc::gc.30:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f4de0 2024-04-27T03:18:30.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(102 gc.30 [call out=24b] v0'0 uv51 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(103 gc.30 [call out=15b] v0'0 uv51 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:104 6.0 6:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(104 gc.30 [call out=15b] v0'0 uv51 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 6.0 6:013d6c33:gc::gc.30:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(105 gc.30 [call] v27'52 uv52 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=1 2024-04-27T03:18:30.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.396+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:106 6.f 6:f24cb4a8:gc::gc.31:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(106 gc.31 [call] v27'26 uv26 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:107 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(107 gc.31 [call out=15b] v0'0 uv26 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:108 6.f 6:f24cb4a8:gc::gc.31:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x7f5ec8022c20 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(108 gc.31 [call out=24b] v0'0 uv26 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:109 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(109 gc.31 [call out=15b] v0'0 uv26 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:110 6.f 6:f24cb4a8:gc::gc.31:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(110 gc.31 [call out=15b] v0'0 uv26 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:111 6.f 6:f24cb4a8:gc::gc.31:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(111 gc.31 [call] v27'27 uv27 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=1 2024-04-27T03:18:30.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.400+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:112 6.19 6:9a6dd8c0:gc::gc.0:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.404+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(112 gc.0 [call] v27'50 uv50 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.404+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:113 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.404+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(113 gc.0 [call out=15b] v0'0 uv50 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.404+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.404+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:114 6.19 6:9a6dd8c0:gc::gc.0:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f92d0 2024-04-27T03:18:30.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.404+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(114 gc.0 [call out=24b] v0'0 uv50 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.404+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:115 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.408+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(115 gc.0 [call out=15b] v0'0 uv50 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.408+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.408+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:116 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.408+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(116 gc.0 [call out=15b] v0'0 uv50 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.408+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.408+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:117 6.19 6:9a6dd8c0:gc::gc.0:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== osd_op_reply(117 gc.0 [call] v27'51 uv51 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=1 2024-04-27T03:18:30.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:118 6.18 6:1e85421f:gc::gc.1:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(118 gc.1 [call] v27'27 uv27 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:119 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(119 gc.1 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:120 6.18 6:1e85421f:gc::gc.1:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x7f5ec8022c20 2024-04-27T03:18:30.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 22 ==== osd_op_reply(120 gc.1 [call out=24b] v0'0 uv27 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:121 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 23 ==== osd_op_reply(121 gc.1 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:122 6.18 6:1e85421f:gc::gc.1:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 24 ==== osd_op_reply(122 gc.1 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.412+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:123 6.18 6:1e85421f:gc::gc.1:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x7f5ec8022c20 2024-04-27T03:18:30.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 25 ==== osd_op_reply(123 gc.1 [call] v27'28 uv28 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=1 2024-04-27T03:18:30.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:124 6.b 6:d242335b:gc::gc.2:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(124 gc.2 [call] v27'27 uv27 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:125 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(125 gc.2 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.416+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:126 6.b 6:d242335b:gc::gc.2:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466fe330 2024-04-27T03:18:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(126 gc.2 [call out=24b] v0'0 uv27 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:127 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(127 gc.2 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:128 6.b 6:d242335b:gc::gc.2:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(128 gc.2 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:129 6.b 6:d242335b:gc::gc.2:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(129 gc.2 [call] v27'28 uv28 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=1 2024-04-27T03:18:30.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.420+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:130 6.4 6:2166d8e1:gc::gc.3:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(130 gc.3 [call] v27'167 uv167 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:131 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(131 gc.3 [call out=15b] v0'0 uv167 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:132 6.4 6:2166d8e1:gc::gc.3:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f4de0 2024-04-27T03:18:30.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(132 gc.3 [call out=24b] v0'0 uv167 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:133 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(133 gc.3 [call out=15b] v0'0 uv167 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:134 6.4 6:2166d8e1:gc::gc.3:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(134 gc.3 [call out=15b] v0'0 uv167 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.424+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:135 6.4 6:2166d8e1:gc::gc.3:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f4de0 2024-04-27T03:18:30.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.428+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(135 gc.3 [call] v27'168 uv168 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.428+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=1 2024-04-27T03:18:30.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.428+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:136 6.3 6:c60da066:gc::gc.4:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.428+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 45 ==== osd_op_reply(136 gc.4 [call] v27'58 uv58 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.428+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:137 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 46 ==== osd_op_reply(137 gc.4 [call out=15b] v0'0 uv58 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:138 6.3 6:c60da066:gc::gc.4:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f92d0 2024-04-27T03:18:30.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 47 ==== osd_op_reply(138 gc.4 [call out=24b] v0'0 uv58 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:139 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 48 ==== osd_op_reply(139 gc.4 [call out=15b] v0'0 uv58 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:140 6.3 6:c60da066:gc::gc.4:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 49 ==== osd_op_reply(140 gc.4 [call out=15b] v0'0 uv58 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.432+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:141 6.3 6:c60da066:gc::gc.4:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.436+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 50 ==== osd_op_reply(141 gc.4 [call] v27'59 uv59 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.436+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=1 2024-04-27T03:18:30.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.436+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:142 6.9 6:954be8d2:gc::gc.5:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(142 gc.5 [call] v27'45 uv45 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:143 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(143 gc.5 [call out=15b] v0'0 uv45 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:144 6.9 6:954be8d2:gc::gc.5:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466fe330 2024-04-27T03:18:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(144 gc.5 [call out=24b] v0'0 uv45 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:145 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(145 gc.5 [call out=15b] v0'0 uv45 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:146 6.9 6:954be8d2:gc::gc.5:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(146 gc.5 [call out=15b] v0'0 uv45 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:147 6.9 6:954be8d2:gc::gc.5:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== osd_op_reply(147 gc.5 [call] v27'46 uv46 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=1 2024-04-27T03:18:30.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.440+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:148 6.e 6:748722ff:gc::gc.6:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 51 ==== osd_op_reply(148 gc.6 [call] v27'29 uv29 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:149 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 52 ==== osd_op_reply(149 gc.6 [call out=15b] v0'0 uv29 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:150 6.e 6:748722ff:gc::gc.6:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f92d0 2024-04-27T03:18:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 53 ==== osd_op_reply(150 gc.6 [call out=24b] v0'0 uv29 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:151 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 54 ==== osd_op_reply(151 gc.6 [call out=15b] v0'0 uv29 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.444+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:152 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 55 ==== osd_op_reply(152 gc.6 [call out=1702b] v0'0 uv29 ondisk = 0) ==== 148+0+1702 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 20 garbage collection: RGWGC::process iterating over entry tag='27a51273-3ccc-49e2-8682-1122abc3d898.4220.16249673198333708825', time=2024-04-27T03:18:29.889059+0000, chain.objs.size()=7 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_1 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:153 3.18s0 3:19fdbda5:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_1:head [call refcount.put in=85b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731ff0 con 0x55a4466fe330 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_2 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:154 3.24s0 3:24370846:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_2:head [call refcount.put in=85b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446734c20 con 0x55a4466fe330 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_3 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:155 3.20s0 3:04fa6fa3:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_3:head [call refcount.put in=85b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446734c20 con 0x7f5ec8022c20 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_4 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:156 3.3cs0 3:3f7c7a13:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_4:head [call refcount.put in=85b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446739bc0 con 0x7f5ec8022c20 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_5 2024-04-27T03:18:30.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:157 3.31s0 3:8cf5830e:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_5:head [call refcount.put in=85b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673c790 con 0x55a4466f4de0 2024-04-27T03:18:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_6 2024-04-27T03:18:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:158 3.23s0 3:c4b0485e:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_6:head [call refcount.put in=85b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673f0f0 con 0x7f5ec8022c20 2024-04-27T03:18:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_7 2024-04-27T03:18:30.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.448+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:159 3.1bs0 3:d89e6060:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_7:head [call refcount.put in=85b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446741ad0 con 0x55a4466fe330 2024-04-27T03:18:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.452+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(153 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_1 [call] v27'4 uv4 ondisk = 0) ==== 230+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.452+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(154 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_2 [call] v27'2 uv2 ondisk = 0) ==== 230+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.452+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== osd_op_reply(159 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_7 [call] v27'2 uv2 ondisk = 0) ==== 230+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.452+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 26 ==== osd_op_reply(156 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_4 [call] v27'2 uv2 ondisk = 0) ==== 230+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.456+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 27 ==== osd_op_reply(155 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_3 [call] v27'2 uv2 ondisk = 0) ==== 230+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.456+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 28 ==== osd_op_reply(158 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_6 [call] v27'2 uv2 ondisk = 0) ==== 230+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.456+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(157 27a51273-3ccc-49e2-8682-1122abc3d898.4220.1__shadow_.evnMkZIWiWAJdmsebdiwUNTMG8jtm_E_5 [call] v27'2 uv2 ondisk = 0) ==== 230+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.456+0000 7f5ed9d81980 5 garbage collection: RGWGC::process removing entries, marker: 2024-04-27T03:18:30.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.456+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:160 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_remove_entries in=47b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673c790 con 0x55a4466f92d0 2024-04-27T03:18:30.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.460+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 56 ==== osd_op_reply(160 gc.6 [call] v27'30 uv30 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.460+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:161 6.e 6:748722ff:gc::gc.6:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673c790 con 0x55a4466f92d0 2024-04-27T03:18:30.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.460+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 57 ==== osd_op_reply(161 gc.6 [call] v27'31 uv31 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.460+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=1 2024-04-27T03:18:30.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.460+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:162 6.1d 6:beaa4be4:gc::gc.7:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673c790 con 0x55a4466f4de0 2024-04-27T03:18:30.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.464+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(162 gc.7 [call] v27'40 uv40 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.464+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:163 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446741560 con 0x55a4466f4de0 2024-04-27T03:18:30.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.464+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 42 ==== osd_op_reply(163 gc.7 [call out=15b] v0'0 uv40 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.464+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.464+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:164 6.1d 6:beaa4be4:gc::gc.7:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f4de0 2024-04-27T03:18:30.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 43 ==== osd_op_reply(164 gc.7 [call out=24b] v0'0 uv40 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:165 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446741560 con 0x55a4466f4de0 2024-04-27T03:18:30.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 44 ==== osd_op_reply(165 gc.7 [call out=15b] v0'0 uv40 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:166 6.1d 6:beaa4be4:gc::gc.7:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446741560 con 0x55a4466f4de0 2024-04-27T03:18:30.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 45 ==== osd_op_reply(166 gc.7 [call out=15b] v0'0 uv40 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.468+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:167 6.1d 6:beaa4be4:gc::gc.7:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446741560 con 0x55a4466f4de0 2024-04-27T03:18:30.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.472+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 46 ==== osd_op_reply(167 gc.7 [call] v27'41 uv41 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.472+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=1 2024-04-27T03:18:30.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.472+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:168 6.12 6:4f78e9a2:gc::gc.8:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.472+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 58 ==== osd_op_reply(168 gc.8 [call] v27'20 uv20 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.472+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:169 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 59 ==== osd_op_reply(169 gc.8 [call out=15b] v0'0 uv20 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:170 6.12 6:4f78e9a2:gc::gc.8:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f92d0 2024-04-27T03:18:30.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 60 ==== osd_op_reply(170 gc.8 [call out=24b] v0'0 uv20 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:171 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 61 ==== osd_op_reply(171 gc.8 [call out=15b] v0'0 uv20 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:172 6.12 6:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 62 ==== osd_op_reply(172 gc.8 [call out=15b] v0'0 uv20 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.476+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:173 6.12 6:4f78e9a2:gc::gc.8:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.480+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 63 ==== osd_op_reply(173 gc.8 [call] v27'21 uv21 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.480+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=1 2024-04-27T03:18:30.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.480+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:174 6.1c 6:3e426232:gc::gc.9:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 64 ==== osd_op_reply(174 gc.9 [call] v27'26 uv26 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:175 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 65 ==== osd_op_reply(175 gc.9 [call out=15b] v0'0 uv26 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:176 6.1c 6:3e426232:gc::gc.9:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f92d0 2024-04-27T03:18:30.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 66 ==== osd_op_reply(176 gc.9 [call out=24b] v0'0 uv26 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:177 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 67 ==== osd_op_reply(177 gc.9 [call out=15b] v0'0 uv26 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:178 6.1c 6:3e426232:gc::gc.9:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 68 ==== osd_op_reply(178 gc.9 [call out=15b] v0'0 uv26 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.484+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:179 6.1c 6:3e426232:gc::gc.9:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.488+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 69 ==== osd_op_reply(179 gc.9 [call] v27'27 uv27 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.488+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=1 2024-04-27T03:18:30.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.488+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:180 6.3 6:c31f41d7:gc::gc.10:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.492+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 70 ==== osd_op_reply(180 gc.10 [call] v27'60 uv60 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.492+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:181 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.492+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 71 ==== osd_op_reply(181 gc.10 [call out=15b] v0'0 uv60 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.492+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.492+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:182 6.3 6:c31f41d7:gc::gc.10:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730b30 con 0x55a4466f92d0 2024-04-27T03:18:30.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.492+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 72 ==== osd_op_reply(182 gc.10 [call out=24b] v0'0 uv60 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.492+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:183 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 73 ==== osd_op_reply(183 gc.10 [call out=15b] v0'0 uv60 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:184 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 74 ==== osd_op_reply(184 gc.10 [call out=15b] v0'0 uv60 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:185 6.3 6:c31f41d7:gc::gc.10:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466f92d0 2024-04-27T03:18:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 75 ==== osd_op_reply(185 gc.10 [call] v27'61 uv61 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=1 2024-04-27T03:18:30.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.496+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:186 6.13 6:c8333aab:gc::gc.11:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446731b40 con 0x55a4466fe330 2024-04-27T03:18:30.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(186 gc.11 [call] v27'23 uv23 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:187 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673f4e0 con 0x55a4466fe330 2024-04-27T03:18:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== osd_op_reply(187 gc.11 [call out=15b] v0'0 uv23 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:188 6.13 6:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730fe0 con 0x55a4466fe330 2024-04-27T03:18:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== osd_op_reply(188 gc.11 [call out=24b] v0'0 uv23 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:189 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== osd_op_reply(189 gc.11 [call out=15b] v0'0 uv23 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:190 6.13 6:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== osd_op_reply(190 gc.11 [call out=15b] v0'0 uv23 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.500+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:191 6.13 6:c8333aab:gc::gc.11:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.504+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== osd_op_reply(191 gc.11 [call] v27'24 uv24 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.504+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=1 2024-04-27T03:18:30.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.504+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:192 6.15 6:ab569f81:gc::gc.12:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.504+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 76 ==== osd_op_reply(192 gc.12 [call] v27'26 uv26 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.504+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:193 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 77 ==== osd_op_reply(193 gc.12 [call out=15b] v0'0 uv26 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:194 6.15 6:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 78 ==== osd_op_reply(194 gc.12 [call out=24b] v0'0 uv26 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:195 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 79 ==== osd_op_reply(195 gc.12 [call out=15b] v0'0 uv26 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:196 6.15 6:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 80 ==== osd_op_reply(196 gc.12 [call out=15b] v0'0 uv26 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.508+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:197 6.15 6:ab569f81:gc::gc.12:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.512+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 81 ==== osd_op_reply(197 gc.12 [call] v27'27 uv27 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.512+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=1 2024-04-27T03:18:30.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.512+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:198 6.0 6:0667bb13:gc::gc.13:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 47 ==== osd_op_reply(198 gc.13 [call] v27'53 uv53 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:199 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 48 ==== osd_op_reply(199 gc.13 [call out=15b] v0'0 uv53 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:200 6.0 6:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 49 ==== osd_op_reply(200 gc.13 [call out=24b] v0'0 uv53 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:201 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 50 ==== osd_op_reply(201 gc.13 [call out=15b] v0'0 uv53 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:202 6.0 6:0667bb13:gc::gc.13:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 51 ==== osd_op_reply(202 gc.13 [call out=15b] v0'0 uv53 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.516+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:203 6.0 6:0667bb13:gc::gc.13:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 52 ==== osd_op_reply(203 gc.13 [call] v27'54 uv54 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=1 2024-04-27T03:18:30.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:204 6.4 6:227f63b5:gc::gc.14:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 53 ==== osd_op_reply(204 gc.14 [call] v27'169 uv169 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:205 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 54 ==== osd_op_reply(205 gc.14 [call out=15b] v0'0 uv169 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.520+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:206 6.4 6:227f63b5:gc::gc.14:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 55 ==== osd_op_reply(206 gc.14 [call out=24b] v0'0 uv169 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:207 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 56 ==== osd_op_reply(207 gc.14 [call out=15b] v0'0 uv169 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:208 6.4 6:227f63b5:gc::gc.14:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 57 ==== osd_op_reply(208 gc.14 [call out=15b] v0'0 uv169 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:209 6.4 6:227f63b5:gc::gc.14:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 58 ==== osd_op_reply(209 gc.14 [call] v27'170 uv170 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=1 2024-04-27T03:18:30.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.524+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:210 6.7 6:e07a6548:gc::gc.15:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.528+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 82 ==== osd_op_reply(210 gc.15 [call] v27'52 uv52 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.528+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:211 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.528+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 83 ==== osd_op_reply(211 gc.15 [call out=15b] v0'0 uv52 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.528+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.528+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:212 6.7 6:e07a6548:gc::gc.15:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.528+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 84 ==== osd_op_reply(212 gc.15 [call out=24b] v0'0 uv52 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:213 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 85 ==== osd_op_reply(213 gc.15 [call out=15b] v0'0 uv52 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:214 6.7 6:e07a6548:gc::gc.15:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 86 ==== osd_op_reply(214 gc.15 [call out=15b] v0'0 uv52 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:215 6.7 6:e07a6548:gc::gc.15:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 87 ==== osd_op_reply(215 gc.15 [call] v27'53 uv53 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=1 2024-04-27T03:18:30.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.532+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:216 6.3 6:c623b173:gc::gc.16:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.536+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 88 ==== osd_op_reply(216 gc.16 [call] v27'62 uv62 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.536+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:217 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 89 ==== osd_op_reply(217 gc.16 [call out=15b] v0'0 uv62 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:218 6.3 6:c623b173:gc::gc.16:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 90 ==== osd_op_reply(218 gc.16 [call out=24b] v0'0 uv62 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:219 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 91 ==== osd_op_reply(219 gc.16 [call out=15b] v0'0 uv62 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.540+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:220 6.3 6:c623b173:gc::gc.16:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.544+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 92 ==== osd_op_reply(220 gc.16 [call out=15b] v0'0 uv62 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.544+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.544+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:221 6.3 6:c623b173:gc::gc.16:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.544+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 93 ==== osd_op_reply(221 gc.16 [call] v27'63 uv63 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.544+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=1 2024-04-27T03:18:30.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.544+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:222 6.16 6:6849fd06:gc::gc.17:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.548+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== osd_op_reply(222 gc.17 [call] v27'33 uv33 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.548+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:223 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.548+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(223 gc.17 [call out=15b] v0'0 uv33 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.548+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.548+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:224 6.16 6:6849fd06:gc::gc.17:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.548+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== osd_op_reply(224 gc.17 [call out=24b] v0'0 uv33 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.548+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:225 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 46 ==== osd_op_reply(225 gc.17 [call out=15b] v0'0 uv33 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:226 6.16 6:6849fd06:gc::gc.17:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 47 ==== osd_op_reply(226 gc.17 [call out=15b] v0'0 uv33 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:227 6.16 6:6849fd06:gc::gc.17:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 48 ==== osd_op_reply(227 gc.17 [call] v27'34 uv34 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=1 2024-04-27T03:18:30.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.552+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:228 6.d 6:b019ea83:gc::gc.18:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.556+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 94 ==== osd_op_reply(228 gc.18 [call] v27'28 uv28 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.556+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:229 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.556+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 95 ==== osd_op_reply(229 gc.18 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.556+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.556+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:230 6.d 6:b019ea83:gc::gc.18:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 96 ==== osd_op_reply(230 gc.18 [call out=24b] v0'0 uv28 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:231 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 97 ==== osd_op_reply(231 gc.18 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:232 6.d 6:b019ea83:gc::gc.18:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 98 ==== osd_op_reply(232 gc.18 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.560+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:233 6.d 6:b019ea83:gc::gc.18:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.564+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 99 ==== osd_op_reply(233 gc.18 [call] v27'29 uv29 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.564+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=1 2024-04-27T03:18:30.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.564+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:234 6.4 6:24c8e1ae:gc::gc.19:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.568+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 59 ==== osd_op_reply(234 gc.19 [call] v27'171 uv171 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.568+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:235 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.568+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 60 ==== osd_op_reply(235 gc.19 [call out=15b] v0'0 uv171 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.568+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.568+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:236 6.4 6:24c8e1ae:gc::gc.19:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.568+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 61 ==== osd_op_reply(236 gc.19 [call out=24b] v0'0 uv171 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.568+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:237 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.572+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 62 ==== osd_op_reply(237 gc.19 [call out=15b] v0'0 uv171 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.572+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.572+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:238 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.572+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 63 ==== osd_op_reply(238 gc.19 [call out=15b] v0'0 uv171 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.572+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.572+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:239 6.4 6:24c8e1ae:gc::gc.19:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.572+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 64 ==== osd_op_reply(239 gc.19 [call] v27'172 uv172 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.576+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=1 2024-04-27T03:18:30.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.576+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:240 6.7 6:e622f76a:gc::gc.20:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.576+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 100 ==== osd_op_reply(240 gc.20 [call] v27'54 uv54 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.576+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:241 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.576+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 101 ==== osd_op_reply(241 gc.20 [call out=15b] v0'0 uv54 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.576+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.576+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:242 6.7 6:e622f76a:gc::gc.20:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 102 ==== osd_op_reply(242 gc.20 [call out=24b] v0'0 uv54 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:243 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 103 ==== osd_op_reply(243 gc.20 [call out=15b] v0'0 uv54 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:244 6.7 6:e622f76a:gc::gc.20:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 104 ==== osd_op_reply(244 gc.20 [call out=15b] v0'0 uv54 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.580+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:245 6.7 6:e622f76a:gc::gc.20:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.584+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 105 ==== osd_op_reply(245 gc.20 [call] v27'55 uv55 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.584+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=1 2024-04-27T03:18:30.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.584+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:246 6.a 6:5127e642:gc::gc.21:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 106 ==== osd_op_reply(246 gc.21 [call] v27'36 uv36 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:247 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 107 ==== osd_op_reply(247 gc.21 [call out=15b] v0'0 uv36 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:248 6.a 6:5127e642:gc::gc.21:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 108 ==== osd_op_reply(248 gc.21 [call out=24b] v0'0 uv36 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:249 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 109 ==== osd_op_reply(249 gc.21 [call out=15b] v0'0 uv36 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.588+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:250 6.a 6:5127e642:gc::gc.21:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.592+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 110 ==== osd_op_reply(250 gc.21 [call out=15b] v0'0 uv36 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.592+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.592+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:251 6.a 6:5127e642:gc::gc.21:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 111 ==== osd_op_reply(251 gc.21 [call] v27'37 uv37 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=1 2024-04-27T03:18:30.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:252 6.11 6:8eb99c53:gc::gc.22:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 49 ==== osd_op_reply(252 gc.22 [call] v27'28 uv28 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:253 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 50 ==== osd_op_reply(253 gc.22 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.596+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:254 6.11 6:8eb99c53:gc::gc.22:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a446730fe0 con 0x55a4466fe330 2024-04-27T03:18:30.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 51 ==== osd_op_reply(254 gc.22 [call out=24b] v0'0 uv28 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:255 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 52 ==== osd_op_reply(255 gc.22 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:256 6.11 6:8eb99c53:gc::gc.22:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 53 ==== osd_op_reply(256 gc.22 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.600+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:257 6.11 6:8eb99c53:gc::gc.22:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.604+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 54 ==== osd_op_reply(257 gc.22 [call] v27'29 uv29 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.604+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=1 2024-04-27T03:18:30.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.604+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:258 6.8 6:17d7e8ed:gc::gc.23:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.604+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 112 ==== osd_op_reply(258 gc.23 [call] v27'38 uv38 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.604+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:259 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 113 ==== osd_op_reply(259 gc.23 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:260 6.8 6:17d7e8ed:gc::gc.23:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 114 ==== osd_op_reply(260 gc.23 [call out=24b] v0'0 uv38 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:261 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 115 ==== osd_op_reply(261 gc.23 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:262 6.8 6:17d7e8ed:gc::gc.23:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 116 ==== osd_op_reply(262 gc.23 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.608+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:263 6.8 6:17d7e8ed:gc::gc.23:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.612+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 117 ==== osd_op_reply(263 gc.23 [call] v27'39 uv39 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.612+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=1 2024-04-27T03:18:30.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.612+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:264 6.1b 6:db9d130d:gc::gc.24:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 55 ==== osd_op_reply(264 gc.24 [call] v27'22 uv22 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:265 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 56 ==== osd_op_reply(265 gc.24 [call out=15b] v0'0 uv22 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:266 6.1b 6:db9d130d:gc::gc.24:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 57 ==== osd_op_reply(266 gc.24 [call out=24b] v0'0 uv22 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:267 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 58 ==== osd_op_reply(267 gc.24 [call out=15b] v0'0 uv22 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:268 6.1b 6:db9d130d:gc::gc.24:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 59 ==== osd_op_reply(268 gc.24 [call out=15b] v0'0 uv22 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.616+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:269 6.1b 6:db9d130d:gc::gc.24:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.620+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 60 ==== osd_op_reply(269 gc.24 [call] v27'23 uv23 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.620+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=1 2024-04-27T03:18:30.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.620+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:270 6.9 6:921ecdb1:gc::gc.25:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.620+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 61 ==== osd_op_reply(270 gc.25 [call] v27'47 uv47 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:271 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 62 ==== osd_op_reply(271 gc.25 [call out=15b] v0'0 uv47 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:272 6.9 6:921ecdb1:gc::gc.25:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 63 ==== osd_op_reply(272 gc.25 [call out=24b] v0'0 uv47 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:273 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 64 ==== osd_op_reply(273 gc.25 [call out=15b] v0'0 uv47 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:274 6.9 6:921ecdb1:gc::gc.25:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 65 ==== osd_op_reply(274 gc.25 [call out=15b] v0'0 uv47 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.624+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:275 6.9 6:921ecdb1:gc::gc.25:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.628+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 66 ==== osd_op_reply(275 gc.25 [call] v27'48 uv48 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.628+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=1 2024-04-27T03:18:30.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.628+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:276 6.1f 6:fde4dd55:gc::gc.26:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.628+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 67 ==== osd_op_reply(276 gc.26 [call] v27'28 uv28 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.628+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:277 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 68 ==== osd_op_reply(277 gc.26 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:278 6.1f 6:fde4dd55:gc::gc.26:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 69 ==== osd_op_reply(278 gc.26 [call out=24b] v0'0 uv28 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:279 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 70 ==== osd_op_reply(279 gc.26 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:280 6.1f 6:fde4dd55:gc::gc.26:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 71 ==== osd_op_reply(280 gc.26 [call out=15b] v0'0 uv28 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.632+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:281 6.1f 6:fde4dd55:gc::gc.26:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466fe330 2024-04-27T03:18:30.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.636+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 72 ==== osd_op_reply(281 gc.26 [call] v27'29 uv29 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.636+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=1 2024-04-27T03:18:30.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.636+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:282 6.1a 6:5909e671:gc::gc.27:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.636+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 118 ==== osd_op_reply(282 gc.27 [call] v27'32 uv32 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.636+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:283 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 119 ==== osd_op_reply(283 gc.27 [call out=15b] v0'0 uv32 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:284 6.1a 6:5909e671:gc::gc.27:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 120 ==== osd_op_reply(284 gc.27 [call out=24b] v0'0 uv32 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:285 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 121 ==== osd_op_reply(285 gc.27 [call out=15b] v0'0 uv32 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.640+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:286 6.1a 6:5909e671:gc::gc.27:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.644+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 122 ==== osd_op_reply(286 gc.27 [call out=15b] v0'0 uv32 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.644+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.644+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:287 6.1a 6:5909e671:gc::gc.27:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f92d0 2024-04-27T03:18:30.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.644+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 123 ==== osd_op_reply(287 gc.27 [call] v27'33 uv33 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.644+0000 7f5ed9d81980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=1 2024-04-27T03:18:30.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.644+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:288 6.1d 6:b88e3271:gc::gc.28:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.648+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 65 ==== osd_op_reply(288 gc.28 [call] v27'42 uv42 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.648+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:289 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.648+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 66 ==== osd_op_reply(289 gc.28 [call out=15b] v0'0 uv42 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.648+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.648+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:290 6.1d 6:b88e3271:gc::gc.28:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 67 ==== osd_op_reply(290 gc.28 [call out=24b] v0'0 uv42 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:291 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 68 ==== osd_op_reply(291 gc.28 [call out=15b] v0'0 uv42 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:18:30.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:292 6.1d 6:b88e3271:gc::gc.28:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 69 ==== osd_op_reply(292 gc.28 [call out=15b] v0'0 uv42 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed9d81980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:18:30.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.652+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:293 6.1d 6:b88e3271:gc::gc.28:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.656+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 70 ==== osd_op_reply(293 gc.28 [call] v27'43 uv43 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.656+0000 7f5d8dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:30.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.656+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:294 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94164044763952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a44673a000 con 0x55a4466f4de0 2024-04-27T03:18:30.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.660+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 71 ==== osd_op_reply(294 data_loggenerations_metadata [watch unwatch cookie 94164044763952] v27'173 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.660+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:295 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94164044752288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a446705330 con 0x55a4466f92d0 2024-04-27T03:18:30.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.660+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 124 ==== osd_op_reply(295 notify.0 [watch unwatch cookie 94164044752288] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.660+0000 7f5ed9d81980 20 remove_watcher() i=0 2024-04-27T03:18:30.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.660+0000 7f5ed9d81980 2 removed watcher, disabling cache 2024-04-27T03:18:30.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.660+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:296 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94164044756704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467025a0 con 0x55a4466f4de0 2024-04-27T03:18:30.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.664+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.1 v2:172.21.15.110:6800/1860462708 72 ==== osd_op_reply(296 notify.1 [watch unwatch cookie 94164044756704] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x55a4466f4de0 2024-04-27T03:18:30.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.664+0000 7f5ed9d81980 20 remove_watcher() i=1 2024-04-27T03:18:30.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.664+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:297 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94164044759008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467025a0 con 0x55a4466f92d0 2024-04-27T03:18:30.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.668+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 125 ==== osd_op_reply(297 notify.2 [watch unwatch cookie 94164044759008] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.668+0000 7f5ed9d81980 20 remove_watcher() i=2 2024-04-27T03:18:30.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.668+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:298 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94164044762896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467025a0 con 0x7f5ec8022c20 2024-04-27T03:18:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.668+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 29 ==== osd_op_reply(298 notify.3 [watch unwatch cookie 94164044762896] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.668+0000 7f5ed9d81980 20 remove_watcher() i=3 2024-04-27T03:18:30.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.668+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:299 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94164044749776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467053c0 con 0x55a4466f92d0 2024-04-27T03:18:30.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.672+0000 7f5ed809d640 1 -- 172.21.15.110:0/3125917484 <== osd.0 v2:172.21.15.77:6800/3496324319 126 ==== osd_op_reply(299 notify.4 [watch unwatch cookie 94164044749776] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ec8043040 con 0x55a4466f92d0 2024-04-27T03:18:30.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.672+0000 7f5ed9d81980 20 remove_watcher() i=4 2024-04-27T03:18:30.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.672+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:300 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94164044767776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467053c0 con 0x55a4466fe330 2024-04-27T03:18:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.672+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 73 ==== osd_op_reply(300 notify.5 [watch unwatch cookie 94164044767776] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.672+0000 7f5ed9d81980 20 remove_watcher() i=5 2024-04-27T03:18:30.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.672+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:301 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94164044771104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467053c0 con 0x55a4466fe330 2024-04-27T03:18:30.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed789c640 1 -- 172.21.15.110:0/3125917484 <== osd.3 v2:172.21.15.110:6816/3916647200 74 ==== osd_op_reply(301 notify.6 [watch unwatch cookie 94164044771104] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ed00d0070 con 0x55a4466fe330 2024-04-27T03:18:30.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed9d81980 20 remove_watcher() i=6 2024-04-27T03:18:30.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:302 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94164044773872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a4467025a0 con 0x7f5ec8022c20 2024-04-27T03:18:30.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 <== osd.2 v2:172.21.15.110:6808/3575017494 30 ==== osd_op_reply(302 notify.7 [watch unwatch cookie 94164044773872] v27'164 uv162 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5ecc03c1c0 con 0x7f5ec8022c20 2024-04-27T03:18:30.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed9d81980 20 remove_watcher() i=7 2024-04-27T03:18:30.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466f92d0 msgr2=0x55a4466fb6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466f92d0 0x55a4466fb6c0 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.676+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466f4de0 msgr2=0x55a4466f72a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466f4de0 0x55a4466f72a0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5ec8022c20 msgr2=0x7f5ec8021370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5ec8022c20 0x7f5ec8021370 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a4466fe330 msgr2=0x55a446700720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a4466fe330 0x55a446700720 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5da00346a0 msgr2=0x7f5da0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5da00346a0 0x7f5da0036b60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0008030 tx=0x7f5ed00db000 comp rx=0 tx=0).stop 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 msgr2=0x55a4466e9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3125917484 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4466e22d0 0x55a4466e9790 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8016f60 tx=0x7f5ec8049ac0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 reap_dead start 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed709b640 1 -- 172.21.15.110:0/3125917484 reap_dead start 2024-04-27T03:18:30.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 shutdown_connections 2024-04-27T03:18:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 >> 172.21.15.110:0/3125917484 conn(0x55a4466de610 msgr2=0x55a4466dbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 shutdown_connections 2024-04-27T03:18:30.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3125917484 wait complete. 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5db0034650 msgr2=0x7f5db0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5db0034650 0x7f5db0036b10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55a4466bfab0 tx=0x7f5ed00033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a44653baa0 msgr2=0x55a446539f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a44653baa0 0x55a446539f40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800aab0 tx=0x7f5ec800af70 comp rx=0 tx=0).stop 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 shutdown_connections 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5db0034650 0x7f5db0036b10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/90298851 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a44653baa0 0x55a446539f40 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 >> 172.21.15.110:0/90298851 conn(0x55a4466c7cf0 msgr2=0x55a4466c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 shutdown_connections 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/90298851 wait complete. 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466c0330 msgr2=0x55a4466c2720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466c0330 0x55a4466c2720 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466bcba0 msgr2=0x55a4466bef90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466bcba0 0x55a4466bef90 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5dc0034650 msgr2=0x7f5dc0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5dc0034650 0x7f5dc0036b10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc000f70 tx=0x7f5ecc0012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5ba0 msgr2=0x55a4466b7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5ba0 0x55a4466b7cd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0000f90 tx=0x7f5ed0019810 comp rx=0 tx=0).stop 2024-04-27T03:18:30.692 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 shutdown_connections 2024-04-27T03:18:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a4466c0330 0x55a4466c2720 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a4466bcba0 0x55a4466bef90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5dc0034650 0x7f5dc0036b10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 --2- 172.21.15.110:0/3880842008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a4465c5ba0 0x55a4466b7cd0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 >> 172.21.15.110:0/3880842008 conn(0x55a4465c0b50 msgr2=0x55a4466ab200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 shutdown_connections 2024-04-27T03:18:30.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.680+0000 7f5ed9d81980 1 -- 172.21.15.110:0/3880842008 wait complete. 2024-04-27T03:18:30.705 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'list', '--include-all'] 2024-04-27T03:18:30.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'list', '--include-all'] 2024-04-27T03:18:30.705 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc list --include-all 2024-04-27T03:18:30.781 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:30.781 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:30.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.772+0000 7f831590f980 1 Processor -- start 2024-04-27T03:18:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.772+0000 7f831590f980 1 -- start start 2024-04-27T03:18:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f831590f980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x562e231959c0 0x562e23197db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f831590f980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x562e2304e0d0 con 0x562e231914b0 2024-04-27T03:18:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f831590f980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x562e23043430 con 0x562e231959c0 2024-04-27T03:18:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f8312c29640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x562e231959c0 0x562e23197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f831342a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x562e231914b0 0x562e23191890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51214/0 (socket says 172.21.15.110:51214) 2024-04-27T03:18:30.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f831342a640 1 -- 172.21.15.110:0/3894768538 learned_addr learned my addr 172.21.15.110:0/3894768538 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:30.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f8312428640 1 -- 172.21.15.110:0/3894768538 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881878106 0 0) 0x562e2304e0d0 con 0x562e231914b0 2024-04-27T03:18:30.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f8312428640 1 -- 172.21.15.110:0/3894768538 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82fc001e20 con 0x562e231914b0 2024-04-27T03:18:30.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f8312c29640 1 -- 172.21.15.110:0/3894768538 >> v1:172.21.15.77:6789/0 conn(0x562e231914b0 legacy=0x562e23191890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f8312c29640 1 -- 172.21.15.110:0/3894768538 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e2303c510 con 0x562e231959c0 2024-04-27T03:18:30.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.776+0000 7f8312c29640 1 --2- 172.21.15.110:0/3894768538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231959c0 0x562e23197db0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f830800a480 tx=0x7f830801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8efee707c61ed08 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f8312428640 1 -- 172.21.15.110:0/3894768538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83080b0070 con 0x562e231959c0 2024-04-27T03:18:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f8312428640 1 -- 172.21.15.110:0/3894768538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83080ab540 con 0x562e231959c0 2024-04-27T03:18:30.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f8312428640 1 -- 172.21.15.110:0/3894768538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83080ab820 con 0x562e231959c0 2024-04-27T03:18:30.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f831590f980 1 -- 172.21.15.110:0/3894768538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231959c0 msgr2=0x562e23197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f831590f980 1 --2- 172.21.15.110:0/3894768538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231959c0 0x562e23197db0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f830800a480 tx=0x7f830801b280 comp rx=0 tx=0).stop 2024-04-27T03:18:30.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f831590f980 1 -- 172.21.15.110:0/3894768538 shutdown_connections 2024-04-27T03:18:30.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f831590f980 1 --2- 172.21.15.110:0/3894768538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231959c0 0x562e23197db0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f831590f980 1 -- 172.21.15.110:0/3894768538 >> 172.21.15.110:0/3894768538 conn(0x562e23104b90 msgr2=0x562e23104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f831590f980 1 -- 172.21.15.110:0/3894768538 shutdown_connections 2024-04-27T03:18:30.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.780+0000 7f831590f980 1 -- 172.21.15.110:0/3894768538 wait complete. 2024-04-27T03:18:30.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 Processor -- start 2024-04-27T03:18:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 -- start start 2024-04-27T03:18:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e2327c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562e230922d0 con 0x562e231914b0 2024-04-27T03:18:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831342a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e2327c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831342a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e2327c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40858/0 (socket says 172.21.15.110:40858) 2024-04-27T03:18:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831342a640 1 -- 172.21.15.110:0/1332991542 learned_addr learned my addr 172.21.15.110:0/1332991542 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831342a640 1 -- 172.21.15.110:0/1332991542 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e23018ae0 con 0x562e231914b0 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831342a640 1 --2- 172.21.15.110:0/1332991542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e2327c3c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f830c006fc0 tx=0x7f830c001af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168117c4107a8364 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f820160a640 1 -- 172.21.15.110:0/1332991542 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f830c004530 con 0x562e231914b0 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f820160a640 1 -- 172.21.15.110:0/1332991542 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f830c0046d0 con 0x562e231914b0 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f820160a640 1 -- 172.21.15.110:0/1332991542 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f830c004a70 con 0x562e231914b0 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 -- 172.21.15.110:0/1332991542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 msgr2=0x562e2327c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 --2- 172.21.15.110:0/1332991542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e2327c3c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f830c006fc0 tx=0x7f830c001af0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 -- 172.21.15.110:0/1332991542 shutdown_connections 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 --2- 172.21.15.110:0/1332991542 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e2327c3c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 -- 172.21.15.110:0/1332991542 >> 172.21.15.110:0/1332991542 conn(0x562e23104b90 msgr2=0x562e23277f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 -- 172.21.15.110:0/1332991542 shutdown_connections 2024-04-27T03:18:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.796+0000 7f831590f980 1 -- 172.21.15.110:0/1332991542 wait complete. 2024-04-27T03:18:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 Processor -- start 2024-04-27T03:18:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 -- start start 2024-04-27T03:18:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e23283430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562e2319ade0 con 0x562e231914b0 2024-04-27T03:18:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831342a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e23283430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831342a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e23283430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40864/0 (socket says 172.21.15.110:40864) 2024-04-27T03:18:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831342a640 1 -- 172.21.15.110:0/2286970280 learned_addr learned my addr 172.21.15.110:0/2286970280 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831342a640 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e23283970 con 0x562e231914b0 2024-04-27T03:18:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831342a640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e23283430 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f830c002260 tx=0x7f830c000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f81ffe07640 1 -- 172.21.15.110:0/2286970280 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f830c013760 con 0x562e231914b0 2024-04-27T03:18:30.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f81ffe07640 1 -- 172.21.15.110:0/2286970280 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f830c013900 con 0x562e231914b0 2024-04-27T03:18:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562e23283c00 con 0x562e231914b0 2024-04-27T03:18:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f81ffe07640 1 -- 172.21.15.110:0/2286970280 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f830c013c00 con 0x562e231914b0 2024-04-27T03:18:30.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e23284090 con 0x562e231914b0 2024-04-27T03:18:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f81ffe07640 1 -- 172.21.15.110:0/2286970280 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f830c010070 con 0x562e231914b0 2024-04-27T03:18:30.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f81ffe07640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81f8034650 0x7f81f8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f81ffe07640 1 -- 172.21.15.110:0/2286970280 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f830c018070 con 0x562e231914b0 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f8312c29640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81f8034650 0x7f81f8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e231959c0 0x562e2328a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e2328a920 con 0x562e231959c0 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f8313c2b640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e231959c0 0x562e2328a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.800+0000 7f8312c29640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81f8034650 0x7f81f8036b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f83080ad000 tx=0x7f83080a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f8313c2b640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e231959c0 0x562e2328a300 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f8313c2b640 1 -- 172.21.15.110:0/2286970280 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f83040083f0 con 0x562e231959c0 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e2328b710 0x562e2328db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e2328e130 con 0x562e2328b710 2024-04-27T03:18:30.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831342a640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e2328b710 0x562e2328db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831342a640 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e2328b710 0x562e2328db00 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831342a640 1 -- 172.21.15.110:0/2286970280 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f830c01b030 con 0x562e2328b710 2024-04-27T03:18:30.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e2328f9e0 con 0x562e231959c0 2024-04-27T03:18:30.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f8313c2b640 1 -- 172.21.15.110:0/2286970280 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f83040083f0 con 0x562e231959c0 2024-04-27T03:18:30.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e2328ec10 con 0x562e2328b710 2024-04-27T03:18:30.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831342a640 1 -- 172.21.15.110:0/2286970280 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f830c01b030 con 0x562e2328b710 2024-04-27T03:18:30.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e2328f000 con 0x562e231959c0 2024-04-27T03:18:30.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.804+0000 7f8313c2b640 1 -- 172.21.15.110:0/2286970280 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f83040083f0 con 0x562e231959c0 2024-04-27T03:18:30.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 Processor -- start 2024-04-27T03:18:30.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 -- start start 2024-04-27T03:18:30.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e2329e320 0x562e2329e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562e23096310 con 0x562e2329e320 2024-04-27T03:18:30.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f8313c2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e2329e320 0x562e2329e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f8313c2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e2329e320 0x562e2329e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40872/0 (socket says 172.21.15.110:40872) 2024-04-27T03:18:30.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f8313c2b640 1 -- 172.21.15.110:0/2984925955 learned_addr learned my addr 172.21.15.110:0/2984925955 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f8313c2b640 1 -- 172.21.15.110:0/2984925955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e2329ec80 con 0x562e2329e320 2024-04-27T03:18:30.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f8313c2b640 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e2329e320 0x562e2329e740 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f830400aab0 tx=0x7f830400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f81fd602640 1 -- 172.21.15.110:0/2984925955 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304007d30 con 0x562e2329e320 2024-04-27T03:18:30.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f81fd602640 1 -- 172.21.15.110:0/2984925955 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f830401d070 con 0x562e2329e320 2024-04-27T03:18:30.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f81fd602640 1 -- 172.21.15.110:0/2984925955 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304018440 con 0x562e2329e320 2024-04-27T03:18:30.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562e2329f720 con 0x562e2329e320 2024-04-27T03:18:30.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e2329fbf0 con 0x562e2329e320 2024-04-27T03:18:30.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f81fd602640 1 -- 172.21.15.110:0/2984925955 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8304013070 con 0x562e2329e320 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f81fd602640 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81ec034650 0x7f81ec036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f81fd602640 1 -- 172.21.15.110:0/2984925955 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f830404b520 con 0x562e2329e320 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831342a640 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81ec034650 0x7f81ec036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 Processor -- start 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 -- start start 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831342a640 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81ec034650 0x7f81ec036b10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x562e2328e730 tx=0x7f830c010360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.808+0000 7f831590f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232ab910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562e2327cb10 con 0x562e232ab4f0 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232ab910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40876/0 (socket says 172.21.15.110:40876) 2024-04-27T03:18:30.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 -- 172.21.15.110:0/1860595083 learned_addr learned my addr 172.21.15.110:0/1860595083 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 -- 172.21.15.110:0/1860595083 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e232abee0 con 0x562e232ab4f0 2024-04-27T03:18:30.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 --2- 172.21.15.110:0/1860595083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232ab910 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8304049930 tx=0x7f8304018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b59e532a84868d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f81f6ffd640 1 -- 172.21.15.110:0/1860595083 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304047070 con 0x562e232ab4f0 2024-04-27T03:18:30.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f81f6ffd640 1 -- 172.21.15.110:0/1860595083 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8304018bc0 con 0x562e232ab4f0 2024-04-27T03:18:30.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f81f6ffd640 1 -- 172.21.15.110:0/1860595083 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304020360 con 0x562e232ab4f0 2024-04-27T03:18:30.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- 172.21.15.110:0/1860595083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 msgr2=0x562e232ab910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 --2- 172.21.15.110:0/1860595083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232ab910 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8304049930 tx=0x7f8304018950 comp rx=0 tx=0).stop 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- 172.21.15.110:0/1860595083 shutdown_connections 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 --2- 172.21.15.110:0/1860595083 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232ab910 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- 172.21.15.110:0/1860595083 >> 172.21.15.110:0/1860595083 conn(0x562e232a7830 msgr2=0x562e232a5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- 172.21.15.110:0/1860595083 shutdown_connections 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- 172.21.15.110:0/1860595083 wait complete. 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 Processor -- start 2024-04-27T03:18:30.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- start start 2024-04-27T03:18:30.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f831590f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562e232ac640 con 0x562e232ab4f0 2024-04-27T03:18:30.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40886/0 (socket says 172.21.15.110:40886) 2024-04-27T03:18:30.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 learned_addr learned my addr 172.21.15.110:0/3979396828 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:30.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e232b2f10 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.812+0000 7f8313c2b640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232b29d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f83040168b0 tx=0x7f83040498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f81f57fa640 1 -- 172.21.15.110:0/3979396828 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304031070 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562e232b31a0 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f81f57fa640 1 -- 172.21.15.110:0/3979396828 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8304020900 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f81f57fa640 1 -- 172.21.15.110:0/3979396828 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8304049a70 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e232b5e80 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f81f57fa640 1 -- 172.21.15.110:0/3979396828 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8304034070 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f81f57fa640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81dc034650 0x7f81dc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f81f57fa640 1 -- 172.21.15.110:0/3979396828 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f830403c380 con 0x562e232ab4f0 2024-04-27T03:18:30.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831342a640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81dc034650 0x7f81dc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e232be050 0x562e232c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c0c40 con 0x562e232be050 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f8312c29640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e232be050 0x562e232c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831342a640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81dc034650 0x7f81dc036b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f830c0108c0 tx=0x7f830c037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f8312c29640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e232be050 0x562e232c04f0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 realm 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c1010 con 0x562e232be050 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c18e0 con 0x562e232be050 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e232c2220 0x562e232c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c4e50 con 0x562e232c2220 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.816+0000 7f8313c2b640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e232c2220 0x562e232c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8313c2b640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e232c2220 0x562e232c46e0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831590f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c56f0 con 0x562e232be050 2024-04-27T03:18:30.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831590f980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:30.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831590f980 20 searching for the correct realm 2024-04-27T03:18:30.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562e232c7460 0x562e232c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c7460 2024-04-27T03:18:30.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831342a640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562e232c7460 0x562e232c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831342a640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562e232c7460 0x562e232c9850 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8313c2b640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8304036070 0x7f830403b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:30.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x7f8304036070 2024-04-27T03:18:30.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8312c29640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8304036070 0x7f830403b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.820+0000 7f8312c29640 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8304036070 0x7f830403b800 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c7460 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x7f8304036070 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.824+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x7f8304036070 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c7460 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c7460 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c7460 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x7f8304036070 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.828+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232be050 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x7f8304036070 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c7460 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x7f8304036070 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c7460 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.832+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x562e232c9e30 con 0x562e232c2220 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:30.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c6070 con 0x562e232be050 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c64c0 con 0x562e232c2220 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232c68e0 con 0x562e232be050 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 zone default found 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 4 Realm: () 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 zonegroup default 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232ca760 con 0x7f8304036070 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232cc170 con 0x562e232c7460 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados->read ofs=0 len=0 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232cc610 con 0x562e232c7460 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 started sync module instance, tier type = 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:30.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.836+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cb5d0 con 0x562e232c2220 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94756158618048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232ccf20 con 0x562e232c2220 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cd770 con 0x562e232be050 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94756158623520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cd930 con 0x562e232be050 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232ce450 con 0x562e232c2220 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94756158618048] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.840+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94756158623520] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94756158627616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232ce410 con 0x562e232c2220 2024-04-27T03:18:30.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cecf0 con 0x7f8304036070 2024-04-27T03:18:30.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94756158629952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232ce410 con 0x7f8304036070 2024-04-27T03:18:30.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cf7a0 con 0x562e232c2220 2024-04-27T03:18:30.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94756158627616] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94756158629952] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94756158631008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cf7a0 con 0x562e232c2220 2024-04-27T03:18:30.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.844+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cfb90 con 0x562e232c7460 2024-04-27T03:18:30.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94756158622320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d0040 con 0x562e232c7460 2024-04-27T03:18:30.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d0a00 con 0x562e232c7460 2024-04-27T03:18:30.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94756158631008] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94756158622320] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94756158634896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d0a00 con 0x562e232c7460 2024-04-27T03:18:30.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d12d0 con 0x7f8304036070 2024-04-27T03:18:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94756158634896] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'165 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94756158637392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d0970 con 0x7f8304036070 2024-04-27T03:18:30.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 20 add_watcher() i=0 2024-04-27T03:18:30.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 20 add_watcher() i=1 2024-04-27T03:18:30.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 20 add_watcher() i=2 2024-04-27T03:18:30.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 20 add_watcher() i=3 2024-04-27T03:18:30.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 20 add_watcher() i=4 2024-04-27T03:18:30.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 20 add_watcher() i=5 2024-04-27T03:18:30.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.848+0000 7f831590f980 20 add_watcher() i=6 2024-04-27T03:18:30.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94756158637392] v27'166 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f831590f980 20 add_watcher() i=7 2024-04-27T03:18:30.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f831590f980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:30.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d43f0 con 0x562e232be050 2024-04-27T03:18:30.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94756158639648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d1910 con 0x562e232be050 2024-04-27T03:18:30.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94756158639648] v27'174 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f81c9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:30.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f831590f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:30.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f831590f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:30.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.852+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232dcc10 con 0x562e232c2220 2024-04-27T03:18:30.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.856+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'52 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.856+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232dcca0 con 0x7f8304036070 2024-04-27T03:18:30.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.856+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'29 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.856+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232dd210 con 0x562e232c7460 2024-04-27T03:18:30.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.860+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'29 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.860+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232dd6c0 con 0x562e232be050 2024-04-27T03:18:30.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.860+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'175 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.860+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232dac50 con 0x562e232c2220 2024-04-27T03:18:30.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.864+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'64 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.864+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232dabf0 con 0x562e232c7460 2024-04-27T03:18:30.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.864+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'49 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.864+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232dabf0 con 0x562e232c2220 2024-04-27T03:18:30.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.868+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'32 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.868+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232daa20 con 0x562e232be050 2024-04-27T03:18:30.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.868+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'44 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.868+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c2220 2024-04-27T03:18:30.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.872+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'22 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.872+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c2220 2024-04-27T03:18:30.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.876+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'28 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.876+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c2220 2024-04-27T03:18:30.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.876+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'65 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.876+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c7460 2024-04-27T03:18:30.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.880+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'25 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.880+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c2220 2024-04-27T03:18:30.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.880+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'28 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.880+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232be050 2024-04-27T03:18:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.884+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'55 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.884+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232be050 2024-04-27T03:18:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.884+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'176 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.884+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c2220 2024-04-27T03:18:30.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.888+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'56 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.888+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c2220 2024-04-27T03:18:30.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.892+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'66 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.892+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de220 con 0x562e232c7460 2024-04-27T03:18:30.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.892+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'35 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.892+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de2b0 con 0x562e232c2220 2024-04-27T03:18:30.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.896+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'30 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.896+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de2b0 con 0x562e232be050 2024-04-27T03:18:30.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.896+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'177 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.896+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de2b0 con 0x562e232c2220 2024-04-27T03:18:30.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.900+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'57 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.900+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de2b0 con 0x562e232c2220 2024-04-27T03:18:30.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.904+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'38 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.904+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de2b0 con 0x562e232c7460 2024-04-27T03:18:30.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.904+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'30 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.904+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232c2220 2024-04-27T03:18:30.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.908+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'40 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.908+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232c7460 2024-04-27T03:18:30.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.912+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'24 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.912+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232c7460 2024-04-27T03:18:30.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.912+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'50 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.912+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232c7460 2024-04-27T03:18:30.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.912+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'30 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.912+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232c2220 2024-04-27T03:18:30.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.916+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'34 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.916+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232be050 2024-04-27T03:18:30.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.920+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'45 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.920+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232c2220 2024-04-27T03:18:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.924+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'58 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.924+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x562e232be050 2024-04-27T03:18:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.924+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'56 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.924+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232de480 con 0x7f8304036070 2024-04-27T03:18:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'28 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f81c8fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f831590f980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f831590f980 20 Filter name: none 2024-04-27T03:18:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f81427fc640 20 reqs_thread_entry: start 2024-04-27T03:18:30.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:94 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232f9ef0 con 0x562e232c2220 2024-04-27T03:18:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(94 gc.0 [call out=15b] v0'0 uv51 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 6.19 6:9a6dd8c0:gc::gc.0:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(95 gc.0 [call out=24b] v0'0 uv51 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.928+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(96 gc.0 [call out=15b] v0'0 uv51 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:97 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x7f8304036070 2024-04-27T03:18:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(97 gc.1 [call out=15b] v0'0 uv28 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 6.18 6:1e85421f:gc::gc.1:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x7f8304036070 2024-04-27T03:18:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(98 gc.1 [call out=24b] v0'0 uv28 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:99 6.18 6:1e85421f:gc::gc.1:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x7f8304036070 2024-04-27T03:18:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(99 gc.1 [call out=15b] v0'0 uv28 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(100 gc.2 [call out=15b] v0'0 uv28 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 6.b 6:d242335b:gc::gc.2:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(101 gc.2 [call out=24b] v0'0 uv28 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 6.b 6:d242335b:gc::gc.2:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(102 gc.2 [call out=15b] v0'0 uv28 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.932+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(103 gc.3 [call out=15b] v0'0 uv168 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:104 6.4 6:2166d8e1:gc::gc.3:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(104 gc.3 [call out=24b] v0'0 uv168 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 6.4 6:2166d8e1:gc::gc.3:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(105 gc.3 [call out=15b] v0'0 uv168 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(106 gc.4 [call out=15b] v0'0 uv59 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:107 6.3 6:c60da066:gc::gc.4:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:30.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(107 gc.4 [call out=24b] v0'0 uv59 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.936+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 6.3 6:c60da066:gc::gc.4:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(108 gc.4 [call out=15b] v0'0 uv59 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:109 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(109 gc.5 [call out=15b] v0'0 uv46 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:110 6.9 6:954be8d2:gc::gc.5:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(110 gc.5 [call out=24b] v0'0 uv46 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:111 6.9 6:954be8d2:gc::gc.5:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(111 gc.5 [call out=15b] v0'0 uv46 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:112 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(112 gc.6 [call out=15b] v0'0 uv31 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:113 6.e 6:748722ff:gc::gc.6:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:30.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(113 gc.6 [call out=24b] v0'0 uv31 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.940+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:114 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(114 gc.6 [call out=15b] v0'0 uv31 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:115 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(115 gc.7 [call out=15b] v0'0 uv41 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:116 6.1d 6:beaa4be4:gc::gc.7:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(116 gc.7 [call out=24b] v0'0 uv41 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:117 6.1d 6:beaa4be4:gc::gc.7:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(117 gc.7 [call out=15b] v0'0 uv41 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:118 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(118 gc.8 [call out=15b] v0'0 uv21 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:119 6.12 6:4f78e9a2:gc::gc.8:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:30.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(119 gc.8 [call out=24b] v0'0 uv21 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.944+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:120 6.12 6:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== osd_op_reply(120 gc.8 [call out=15b] v0'0 uv21 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:121 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 45 ==== osd_op_reply(121 gc.9 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:122 6.1c 6:3e426232:gc::gc.9:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:30.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 46 ==== osd_op_reply(122 gc.9 [call out=24b] v0'0 uv27 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:123 6.1c 6:3e426232:gc::gc.9:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 47 ==== osd_op_reply(123 gc.9 [call out=15b] v0'0 uv27 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.948+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:124 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 48 ==== osd_op_reply(124 gc.10 [call out=15b] v0'0 uv61 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:125 6.3 6:c31f41d7:gc::gc.10:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:30.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 49 ==== osd_op_reply(125 gc.10 [call out=24b] v0'0 uv61 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:126 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 50 ==== osd_op_reply(126 gc.10 [call out=15b] v0'0 uv61 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:127 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(127 gc.11 [call out=15b] v0'0 uv24 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.952+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:128 6.13 6:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(128 gc.11 [call out=24b] v0'0 uv24 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:129 6.13 6:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(129 gc.11 [call out=15b] v0'0 uv24 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:130 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 51 ==== osd_op_reply(130 gc.12 [call out=15b] v0'0 uv27 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:131 6.15 6:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 52 ==== osd_op_reply(131 gc.12 [call out=24b] v0'0 uv27 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.956+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:132 6.15 6:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 53 ==== osd_op_reply(132 gc.12 [call out=15b] v0'0 uv27 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:133 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(133 gc.13 [call out=15b] v0'0 uv54 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:134 6.0 6:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(134 gc.13 [call out=24b] v0'0 uv54 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:135 6.0 6:0667bb13:gc::gc.13:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(135 gc.13 [call out=15b] v0'0 uv54 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:136 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(136 gc.14 [call out=15b] v0'0 uv170 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:137 6.4 6:227f63b5:gc::gc.14:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(137 gc.14 [call out=24b] v0'0 uv170 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:138 6.4 6:227f63b5:gc::gc.14:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(138 gc.14 [call out=15b] v0'0 uv170 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.960+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:139 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 54 ==== osd_op_reply(139 gc.15 [call out=15b] v0'0 uv53 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:140 6.7 6:e07a6548:gc::gc.15:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 55 ==== osd_op_reply(140 gc.15 [call out=24b] v0'0 uv53 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:141 6.7 6:e07a6548:gc::gc.15:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 56 ==== osd_op_reply(141 gc.15 [call out=15b] v0'0 uv53 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:142 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 57 ==== osd_op_reply(142 gc.16 [call out=15b] v0'0 uv63 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.964+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:143 6.3 6:c623b173:gc::gc.16:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 58 ==== osd_op_reply(143 gc.16 [call out=24b] v0'0 uv63 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:144 6.3 6:c623b173:gc::gc.16:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 59 ==== osd_op_reply(144 gc.16 [call out=15b] v0'0 uv63 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:145 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(145 gc.17 [call out=15b] v0'0 uv34 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:146 6.16 6:6849fd06:gc::gc.17:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c7460 2024-04-27T03:18:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(146 gc.17 [call out=24b] v0'0 uv34 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:147 6.16 6:6849fd06:gc::gc.17:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== osd_op_reply(147 gc.17 [call out=15b] v0'0 uv34 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.968+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:148 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.972+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 60 ==== osd_op_reply(148 gc.18 [call out=15b] v0'0 uv29 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.972+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:149 6.d 6:b019ea83:gc::gc.18:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.972+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 61 ==== osd_op_reply(149 gc.18 [call out=24b] v0'0 uv29 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.972+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:150 6.d 6:b019ea83:gc::gc.18:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.972+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 62 ==== osd_op_reply(150 gc.18 [call out=15b] v0'0 uv29 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.972+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:151 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(151 gc.19 [call out=15b] v0'0 uv172 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:152 6.4 6:24c8e1ae:gc::gc.19:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(152 gc.19 [call out=24b] v0'0 uv172 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:153 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 42 ==== osd_op_reply(153 gc.19 [call out=15b] v0'0 uv172 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:154 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 63 ==== osd_op_reply(154 gc.20 [call out=15b] v0'0 uv55 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:155 6.7 6:e622f76a:gc::gc.20:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 64 ==== osd_op_reply(155 gc.20 [call out=24b] v0'0 uv55 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:156 6.7 6:e622f76a:gc::gc.20:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.976+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 65 ==== osd_op_reply(156 gc.20 [call out=15b] v0'0 uv55 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:157 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 66 ==== osd_op_reply(157 gc.21 [call out=15b] v0'0 uv37 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:158 6.a 6:5127e642:gc::gc.21:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 67 ==== osd_op_reply(158 gc.21 [call out=24b] v0'0 uv37 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:159 6.a 6:5127e642:gc::gc.21:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 68 ==== osd_op_reply(159 gc.21 [call out=15b] v0'0 uv37 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:160 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(160 gc.22 [call out=15b] v0'0 uv29 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.980+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:161 6.11 6:8eb99c53:gc::gc.22:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c7460 2024-04-27T03:18:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(161 gc.22 [call out=24b] v0'0 uv29 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:162 6.11 6:8eb99c53:gc::gc.22:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== osd_op_reply(162 gc.22 [call out=15b] v0'0 uv29 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:163 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 69 ==== osd_op_reply(163 gc.23 [call out=15b] v0'0 uv39 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:164 6.8 6:17d7e8ed:gc::gc.23:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 70 ==== osd_op_reply(164 gc.23 [call out=24b] v0'0 uv39 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:165 6.8 6:17d7e8ed:gc::gc.23:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 71 ==== osd_op_reply(165 gc.23 [call out=15b] v0'0 uv39 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:166 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(166 gc.24 [call out=15b] v0'0 uv23 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.984+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:167 6.1b 6:db9d130d:gc::gc.24:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== osd_op_reply(167 gc.24 [call out=24b] v0'0 uv23 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:168 6.1b 6:db9d130d:gc::gc.24:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== osd_op_reply(168 gc.24 [call out=15b] v0'0 uv23 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:169 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== osd_op_reply(169 gc.25 [call out=15b] v0'0 uv48 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:170 6.9 6:921ecdb1:gc::gc.25:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== osd_op_reply(170 gc.25 [call out=24b] v0'0 uv48 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:171 6.9 6:921ecdb1:gc::gc.25:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== osd_op_reply(171 gc.25 [call out=15b] v0'0 uv48 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.988+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:172 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== osd_op_reply(172 gc.26 [call out=15b] v0'0 uv29 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:30.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:173 6.1f 6:fde4dd55:gc::gc.26:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(173 gc.26 [call out=24b] v0'0 uv29 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:174 6.1f 6:fde4dd55:gc::gc.26:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c7460 2024-04-27T03:18:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== osd_op_reply(174 gc.26 [call out=15b] v0'0 uv29 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:31.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:175 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 72 ==== osd_op_reply(175 gc.27 [call out=15b] v0'0 uv33 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:176 6.1a 6:5909e671:gc::gc.27:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:31.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.992+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 73 ==== osd_op_reply(176 gc.27 [call out=24b] v0'0 uv33 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:177 6.1a 6:5909e671:gc::gc.27:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 74 ==== osd_op_reply(177 gc.27 [call out=15b] v0'0 uv33 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:178 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 43 ==== osd_op_reply(178 gc.28 [call out=15b] v0'0 uv43 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:179 6.1d 6:b88e3271:gc::gc.28:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:31.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 44 ==== osd_op_reply(179 gc.28 [call out=24b] v0'0 uv43 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:180 6.1d 6:b88e3271:gc::gc.28:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:31.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 45 ==== osd_op_reply(180 gc.28 [call out=15b] v0'0 uv43 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:30.996+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:181 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:31.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 75 ==== osd_op_reply(181 gc.29 [call out=15b] v0'0 uv51 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:182 6.7 6:e47c9322:gc::gc.29:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d88a0 con 0x562e232c2220 2024-04-27T03:18:31.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 76 ==== osd_op_reply(182 gc.29 [call out=24b] v0'0 uv51 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:183 6.7 6:e47c9322:gc::gc.29:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232c2220 2024-04-27T03:18:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 77 ==== osd_op_reply(183 gc.29 [call out=15b] v0'0 uv51 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:184 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 46 ==== osd_op_reply(184 gc.30 [call out=15b] v0'0 uv52 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.000+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:185 6.0 6:013d6c33:gc::gc.30:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:31.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 47 ==== osd_op_reply(185 gc.30 [call out=24b] v0'0 uv52 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:186 6.0 6:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:31.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 48 ==== osd_op_reply(186 gc.30 [call out=15b] v0'0 uv52 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:187 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x7f8304036070 2024-04-27T03:18:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(187 gc.31 [call out=15b] v0'0 uv27 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:31.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:188 6.f 6:f24cb4a8:gc::gc.31:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x7f8304036070 2024-04-27T03:18:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(188 gc.31 [call out=24b] v0'0 uv27 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:189 6.f 6:f24cb4a8:gc::gc.31:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x7f8304036070 2024-04-27T03:18:31.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.004+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(189 gc.31 [call out=15b] v0'0 uv27 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:31.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.008+0000 7f81c9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:31.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.008+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:190 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94756158639648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d83f0 con 0x562e232be050 2024-04-27T03:18:31.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.008+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 49 ==== osd_op_reply(190 data_loggenerations_metadata [watch unwatch cookie 94756158639648] v27'178 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.008+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:191 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94756158618048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232d0e20 con 0x562e232c2220 2024-04-27T03:18:31.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.012+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 78 ==== osd_op_reply(191 notify.0 [watch unwatch cookie 94756158618048] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.012+0000 7f831590f980 20 remove_watcher() i=0 2024-04-27T03:18:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.012+0000 7f831590f980 2 removed watcher, disabling cache 2024-04-27T03:18:31.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.012+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:192 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94756158623520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cb9c0 con 0x562e232be050 2024-04-27T03:18:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.016+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.1 v2:172.21.15.110:6800/1860462708 50 ==== osd_op_reply(192 notify.1 [watch unwatch cookie 94756158623520] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x562e232be050 2024-04-27T03:18:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.016+0000 7f831590f980 20 remove_watcher() i=1 2024-04-27T03:18:31.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.016+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:193 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94756158627616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cb9c0 con 0x562e232c2220 2024-04-27T03:18:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.016+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 79 ==== osd_op_reply(193 notify.2 [watch unwatch cookie 94756158627616] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.016+0000 7f831590f980 20 remove_watcher() i=2 2024-04-27T03:18:31.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.016+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:194 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94756158629952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cb9c0 con 0x7f8304036070 2024-04-27T03:18:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.020+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(194 notify.3 [watch unwatch cookie 94756158629952] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.020+0000 7f831590f980 20 remove_watcher() i=3 2024-04-27T03:18:31.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.020+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:195 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94756158631008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232ce840 con 0x562e232c2220 2024-04-27T03:18:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.024+0000 7f8313c2b640 1 -- 172.21.15.110:0/3979396828 <== osd.0 v2:172.21.15.77:6800/3496324319 80 ==== osd_op_reply(195 notify.4 [watch unwatch cookie 94756158631008] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8304033030 con 0x562e232c2220 2024-04-27T03:18:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.024+0000 7f831590f980 20 remove_watcher() i=4 2024-04-27T03:18:31.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.024+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:196 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94756158622320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cb9c0 con 0x562e232c7460 2024-04-27T03:18:31.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.024+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 46 ==== osd_op_reply(196 notify.5 [watch unwatch cookie 94756158622320] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.024+0000 7f831590f980 20 remove_watcher() i=5 2024-04-27T03:18:31.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.024+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:197 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94756158634896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cca70 con 0x562e232c7460 2024-04-27T03:18:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831342a640 1 -- 172.21.15.110:0/3979396828 <== osd.3 v2:172.21.15.110:6816/3916647200 47 ==== osd_op_reply(197 notify.6 [watch unwatch cookie 94756158634896] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f830c01b030 con 0x562e232c7460 2024-04-27T03:18:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831590f980 20 remove_watcher() i=6 2024-04-27T03:18:31.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:198 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94756158637392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x562e232cb9c0 con 0x7f8304036070 2024-04-27T03:18:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(198 notify.7 [watch unwatch cookie 94756158637392] v27'167 uv165 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f83080b3040 con 0x7f8304036070 2024-04-27T03:18:31.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831590f980 20 remove_watcher() i=7 2024-04-27T03:18:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e232c2220 msgr2=0x562e232c46e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e232c2220 0x562e232c46e0 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e232be050 msgr2=0x562e232c04f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.028+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e232be050 0x562e232c04f0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8304036070 msgr2=0x7f830403b800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8304036070 0x7f830403b800 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562e232c7460 msgr2=0x562e232c9850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562e232c7460 0x562e232c9850 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81dc034650 msgr2=0x7f81dc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81dc034650 0x7f81dc036b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f830c0108c0 tx=0x7f830c037000 comp rx=0 tx=0).stop 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 msgr2=0x562e232b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f8312c29640 1 -- 172.21.15.110:0/3979396828 reap_dead start 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232b29d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f83040168b0 tx=0x7f83040498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 shutdown_connections 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/3979396828 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e232ab4f0 0x562e232b29d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 >> 172.21.15.110:0/3979396828 conn(0x562e232a7830 msgr2=0x562e232a4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 shutdown_connections 2024-04-27T03:18:31.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/3979396828 wait complete. 2024-04-27T03:18:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81ec034650 msgr2=0x7f81ec036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81ec034650 0x7f81ec036b10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x562e2328e730 tx=0x7f830c010360 comp rx=0 tx=0).stop 2024-04-27T03:18:31.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e2329e320 msgr2=0x562e2329e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e2329e320 0x562e2329e740 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f830400aab0 tx=0x7f830400af70 comp rx=0 tx=0).stop 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 shutdown_connections 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81ec034650 0x7f81ec036b10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2984925955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e2329e320 0x562e2329e740 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 >> 172.21.15.110:0/2984925955 conn(0x562e23293130 msgr2=0x562e23293c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 shutdown_connections 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2984925955 wait complete. 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e2328b710 msgr2=0x562e2328db00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e2328b710 0x562e2328db00 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e231959c0 msgr2=0x562e2328a300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e231959c0 0x562e2328a300 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81f8034650 msgr2=0x7f81f8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81f8034650 0x7f81f8036b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f83080ad000 tx=0x7f83080a5040 comp rx=0 tx=0).stop 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 msgr2=0x562e23283430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e23283430 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f830c002260 tx=0x7f830c000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 shutdown_connections 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562e2328b710 0x562e2328db00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562e231959c0 0x562e2328a300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f81f8034650 0x7f81f8036b10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 --2- 172.21.15.110:0/2286970280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562e231914b0 0x562e23283430 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 >> 172.21.15.110:0/2286970280 conn(0x562e23104b90 msgr2=0x562e23276a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 shutdown_connections 2024-04-27T03:18:31.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.032+0000 7f831590f980 1 -- 172.21.15.110:0/2286970280 wait complete. 2024-04-27T03:18:31.058 INFO:teuthology.orchestra.run.smithi110.stdout:[] 2024-04-27T03:18:31.058 DEBUG:tasks.util.rgw: json result: [] 2024-04-27T03:18:31.058 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo'] 2024-04-27T03:18:31.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo'] 2024-04-27T03:18:31.059 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo 2024-04-27T03:18:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:31.135 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff69f1980 1 Processor -- start 2024-04-27T03:18:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff69f1980 1 -- start start 2024-04-27T03:18:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff69f1980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5591d3a34850 0x5591d3ac2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff69f1980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5591d397f0d0 con 0x5591d3ac3480 2024-04-27T03:18:31.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff69f1980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5591d3974430 con 0x5591d3a34850 2024-04-27T03:18:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff450c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5591d3a34850 0x5591d3ac2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff3d0b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5591d3ac3480 0x5591d3ac5870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51230/0 (socket says 172.21.15.110:51230) 2024-04-27T03:18:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/82933198 learned_addr learned my addr 172.21.15.110:0/82933198 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff450c640 1 -- 172.21.15.110:0/82933198 >> v1:172.21.15.77:6789/0 conn(0x5591d3ac3480 legacy=0x5591d3ac5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.128+0000 7f7ff450c640 1 -- 172.21.15.110:0/82933198 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d396d510 con 0x5591d3a34850 2024-04-27T03:18:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff450c640 1 --2- 172.21.15.110:0/82933198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3ac2f40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00a420 tx=0x7f7fec01afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de64c9dcdc71e57e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff350a640 1 -- 172.21.15.110:0/82933198 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fec0af070 con 0x5591d3a34850 2024-04-27T03:18:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff350a640 1 -- 172.21.15.110:0/82933198 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fec0aa530 con 0x5591d3a34850 2024-04-27T03:18:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff350a640 1 -- 172.21.15.110:0/82933198 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fec0aa850 con 0x5591d3a34850 2024-04-27T03:18:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff69f1980 1 -- 172.21.15.110:0/82933198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 msgr2=0x5591d3ac2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/82933198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3ac2f40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00a420 tx=0x7f7fec01afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff69f1980 1 -- 172.21.15.110:0/82933198 shutdown_connections 2024-04-27T03:18:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/82933198 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3ac2f40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff69f1980 1 -- 172.21.15.110:0/82933198 >> 172.21.15.110:0/82933198 conn(0x5591d3abaa20 msgr2=0x5591d3abce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff69f1980 1 -- 172.21.15.110:0/82933198 shutdown_connections 2024-04-27T03:18:31.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.132+0000 7f7ff69f1980 1 -- 172.21.15.110:0/82933198 wait complete. 2024-04-27T03:18:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff69f1980 1 Processor -- start 2024-04-27T03:18:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff69f1980 1 -- start start 2024-04-27T03:18:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff69f1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff69f1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5591d39c32d0 con 0x5591d3a34850 2024-04-27T03:18:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff450c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff450c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bab4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40906/0 (socket says 172.21.15.110:40906) 2024-04-27T03:18:31.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff450c640 1 -- 172.21.15.110:0/2136760381 learned_addr learned my addr 172.21.15.110:0/2136760381 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff450c640 1 -- 172.21.15.110:0/2136760381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d3949ae0 con 0x5591d3a34850 2024-04-27T03:18:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ff450c640 1 --2- 172.21.15.110:0/2136760381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bab4c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7fec018c00 tx=0x7f7fec0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a6218ccf33a24d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ee260c640 1 -- 172.21.15.110:0/2136760381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fec0af050 con 0x5591d3a34850 2024-04-27T03:18:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ee260c640 1 -- 172.21.15.110:0/2136760381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fec0b0040 con 0x5591d3a34850 2024-04-27T03:18:31.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.148+0000 7f7ee260c640 1 -- 172.21.15.110:0/2136760381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fec0aabb0 con 0x5591d3a34850 2024-04-27T03:18:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2136760381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 msgr2=0x5591d3bab4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2136760381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bab4c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7fec018c00 tx=0x7f7fec0033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2136760381 shutdown_connections 2024-04-27T03:18:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2136760381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bab4c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2136760381 >> 172.21.15.110:0/2136760381 conn(0x5591d3abaa20 msgr2=0x5591d3ba7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2136760381 shutdown_connections 2024-04-27T03:18:31.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2136760381 wait complete. 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 Processor -- start 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- start start 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bb24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5591d3ac7aa0 con 0x5591d3a34850 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff450c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bb24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff450c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bb24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40922/0 (socket says 172.21.15.110:40922) 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff450c640 1 -- 172.21.15.110:0/4014660092 learned_addr learned my addr 172.21.15.110:0/4014660092 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff450c640 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d3bb2a10 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff450c640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bb24d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0b8040 tx=0x7f7fec019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ee0e09640 1 -- 172.21.15.110:0/4014660092 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fec0c5070 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ee0e09640 1 -- 172.21.15.110:0/4014660092 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fec0b6c10 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5591d3bb2ca0 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ee0e09640 1 -- 172.21.15.110:0/4014660092 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fec0c1480 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d3bb58d0 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ee0e09640 1 -- 172.21.15.110:0/4014660092 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7fec0c8070 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ee0e09640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7edc034650 0x7f7edc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.152+0000 7f7ee0e09640 1 -- 172.21.15.110:0/4014660092 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7fec0bb070 con 0x5591d3a34850 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff3d0b640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7edc034650 0x7f7edc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3ac3480 0x5591d3bb93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bb98f0 con 0x5591d3ac3480 2024-04-27T03:18:31.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3ac3480 0x5591d3bb93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff3d0b640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7edc034650 0x7f7edc036b10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8000f70 tx=0x7f7fe80012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3ac3480 0x5591d3bb93b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/4014660092 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3ac3480 2024-04-27T03:18:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bba6e0 0x5591d3bbcad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bbd100 con 0x5591d3bba6e0 2024-04-27T03:18:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff450c640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bba6e0 0x5591d3bbcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff450c640 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bba6e0 0x5591d3bbcad0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff450c640 1 -- 172.21.15.110:0/4014660092 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bba6e0 2024-04-27T03:18:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bbe9b0 con 0x5591d3ac3480 2024-04-27T03:18:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/4014660092 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3ac3480 2024-04-27T03:18:31.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.156+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bbdbe0 con 0x5591d3bba6e0 2024-04-27T03:18:31.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff450c640 1 -- 172.21.15.110:0/4014660092 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bba6e0 2024-04-27T03:18:31.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bbe000 con 0x5591d3ac3480 2024-04-27T03:18:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/4014660092 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3ac3480 2024-04-27T03:18:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff69f1980 1 Processor -- start 2024-04-27T03:18:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff69f1980 1 -- start start 2024-04-27T03:18:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff69f1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bcd2d0 0x5591d3bcd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff69f1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5591d39c7310 con 0x5591d3bcd2d0 2024-04-27T03:18:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff4d0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bcd2d0 0x5591d3bcd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff4d0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bcd2d0 0x5591d3bcd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40926/0 (socket says 172.21.15.110:40926) 2024-04-27T03:18:31.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2736665463 learned_addr learned my addr 172.21.15.110:0/2736665463 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2736665463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d3bcdc30 con 0x5591d3bcd2d0 2024-04-27T03:18:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bcd2d0 0x5591d3bcd6f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400aab0 tx=0x7f7fe400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ed9ffb640 1 -- 172.21.15.110:0/2736665463 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe4007d30 con 0x5591d3bcd2d0 2024-04-27T03:18:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ed9ffb640 1 -- 172.21.15.110:0/2736665463 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe4018440 con 0x5591d3bcd2d0 2024-04-27T03:18:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5591d3bce6d0 con 0x5591d3bcd2d0 2024-04-27T03:18:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ed9ffb640 1 -- 172.21.15.110:0/2736665463 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe4018760 con 0x5591d3bcd2d0 2024-04-27T03:18:31.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.160+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d3bceba0 con 0x5591d3bcd2d0 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ed9ffb640 1 -- 172.21.15.110:0/2736665463 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7fe4013070 con 0x5591d3bcd2d0 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ed9ffb640 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ecc034650 0x7f7ecc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ed9ffb640 1 -- 172.21.15.110:0/2736665463 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7fe404b7e0 con 0x5591d3bcd2d0 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff450c640 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ecc034650 0x7f7ecc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 Processor -- start 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 -- start start 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3bda8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff450c640 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ecc034650 0x7f7ecc036b10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x5591d3bb9ea0 tx=0x7f7fec0e3550 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5591d3babb50 con 0x5591d3bda4a0 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff4d0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3bda8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff4d0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3bda8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40942/0 (socket says 172.21.15.110:40942) 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2956017368 learned_addr learned my addr 172.21.15.110:0/2956017368 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2956017368 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d3bdae90 con 0x5591d3bda4a0 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/2956017368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3bda8c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4015020 tx=0x7f7fe4016e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33db312427bb46e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ed37fe640 1 -- 172.21.15.110:0/2956017368 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe40237f0 con 0x5591d3bda4a0 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ed37fe640 1 -- 172.21.15.110:0/2956017368 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe4016330 con 0x5591d3bda4a0 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ed37fe640 1 -- 172.21.15.110:0/2956017368 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe4023990 con 0x5591d3bda4a0 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2956017368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 msgr2=0x5591d3bda8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2956017368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3bda8c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4015020 tx=0x7f7fe4016e50 comp rx=0 tx=0).stop 2024-04-27T03:18:31.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2956017368 shutdown_connections 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2956017368 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3bda8c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2956017368 >> 172.21.15.110:0/2956017368 conn(0x5591d3bd67e0 msgr2=0x5591d3bd4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2956017368 shutdown_connections 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2956017368 wait complete. 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.164+0000 7f7ff69f1980 1 Processor -- start 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 1 -- start start 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3ac0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5591d3bdb610 con 0x5591d3bda4a0 2024-04-27T03:18:31.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff4d0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3ac0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff4d0d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3ac0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40958/0 (socket says 172.21.15.110:40958) 2024-04-27T03:18:31.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 learned_addr learned my addr 172.21.15.110:0/2629719299 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d3ac0d60 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3ac0820 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4024fd0 tx=0x7f7fe4049ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ed1ffb640 1 -- 172.21.15.110:0/2629719299 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe4018e60 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5591d3abef40 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ed1ffb640 1 -- 172.21.15.110:0/2629719299 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe404b2e0 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ed1ffb640 1 -- 172.21.15.110:0/2629719299 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fe404b5c0 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d3abf3b0 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ed1ffb640 1 -- 172.21.15.110:0/2629719299 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7fe4016040 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ed1ffb640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ec4034600 0x7f7ec4036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ed1ffb640 1 -- 172.21.15.110:0/2629719299 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f7fe4031910 con 0x5591d3bda4a0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff450c640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ec4034600 0x7f7ec4036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3bef190 0x5591d3bf1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf1da0 con 0x5591d3bef190 2024-04-27T03:18:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff3d0b640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3bef190 0x5591d3bf1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff450c640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ec4034600 0x7f7ec4036ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0a3c50 tx=0x7f7fec0c8f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.168+0000 7f7ff3d0b640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3bef190 0x5591d3bf1650 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7fe800e5a0 con 0x5591d3bef190 2024-04-27T03:18:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 realm 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bee5e0 con 0x5591d3bef190 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7fe800e5a0 con 0x5591d3bef190 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf28a0 con 0x5591d3bef190 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7fe800e5a0 con 0x5591d3bef190 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bf3680 0x5591d3bf5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf6120 con 0x5591d3bf3680 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bf3680 0x5591d3bf5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bf3680 0x5591d3bf5a70 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3bf3680 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf6970 con 0x5591d3bef190 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f7fe800e5a0 con 0x5591d3bef190 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 20 searching for the correct realm 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5591d3bf86e0 0x5591d3bfaad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf86e0 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff450c640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5591d3bf86e0 0x5591d3bfaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.172+0000 7f7ff450c640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5591d3bf86e0 0x5591d3bfaad0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf3680 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3bf3680 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff4d0d640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7fe4022c20 0x7f7fe40173a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x7f7fe4022c20 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7fe4022c20 0x7f7fe40173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7fe4022c20 0x7f7fe40173a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe800e5a0 con 0x7f7fe4022c20 2024-04-27T03:18:31.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe800e5a0 con 0x5591d3bef190 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf86e0 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf3680 2024-04-27T03:18:31.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3bf3680 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.176+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf3680 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3bf3680 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x7f7fe4022c20 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x7f7fe4022c20 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x7f7fe4022c20 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f7fe8029040 con 0x7f7fe4022c20 2024-04-27T03:18:31.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf3680 2024-04-27T03:18:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe40083f0 con 0x5591d3bf3680 2024-04-27T03:18:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf86e0 2024-04-27T03:18:31.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.180+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf3680 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf86e0 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf86e0 2024-04-27T03:18:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x7f7fe4022c20 2024-04-27T03:18:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x7f7fe4022c20 2024-04-27T03:18:31.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bef190 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x7f7fe4022c20 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x7f7fe4022c20 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf86e0 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x7f7fe4022c20 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe8029040 con 0x7f7fe4022c20 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf3680 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.184+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfb150 con 0x5591d3bf3680 2024-04-27T03:18:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f7fe4069040 con 0x5591d3bf86e0 2024-04-27T03:18:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f7fe4069040 con 0x5591d3bf3680 2024-04-27T03:18:31.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x7f7fe4069040 con 0x5591d3bf3680 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf71c0 con 0x5591d3bef190 2024-04-27T03:18:31.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f7fe8029040 con 0x5591d3bef190 2024-04-27T03:18:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.188+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf75b0 con 0x5591d3bf3680 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf75b0 con 0x5591d3bef190 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 zone default found 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 4 Realm: () 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:31.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 zonegroup default 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bf6e00 con 0x7f7fe4022c20 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfcf90 con 0x5591d3bf86e0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfd330 con 0x5591d3bf86e0 2024-04-27T03:18:31.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 started sync module instance, tier type = 2024-04-27T03:18:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:31.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfc430 con 0x5591d3bf3680 2024-04-27T03:18:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.192+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94084811180064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfc430 con 0x5591d3bf3680 2024-04-27T03:18:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfdbc0 con 0x5591d3bef190 2024-04-27T03:18:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94084811183824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfdaf0 con 0x5591d3bef190 2024-04-27T03:18:31.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfeac0 con 0x5591d3bf3680 2024-04-27T03:18:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94084811183824] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94084811180064] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94084811177856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfe5d0 con 0x5591d3bf3680 2024-04-27T03:18:31.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.196+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bff360 con 0x7f7fe4022c20 2024-04-27T03:18:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94084811189936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfe5d0 con 0x7f7fe4022c20 2024-04-27T03:18:31.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bffe00 con 0x5591d3bf3680 2024-04-27T03:18:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94084811177856] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94084811189936] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94084811190992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bffdc0 con 0x5591d3bf3680 2024-04-27T03:18:31.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c001f0 con 0x5591d3bf86e0 2024-04-27T03:18:31.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94084811186096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c00a40 con 0x5591d3bf86e0 2024-04-27T03:18:31.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.200+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c01440 con 0x5591d3bf86e0 2024-04-27T03:18:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94084811190992] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94084811186096] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94084811195792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c00f50 con 0x5591d3bf86e0 2024-04-27T03:18:31.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c01e50 con 0x7f7fe4022c20 2024-04-27T03:18:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94084811195792] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'168 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94084811192560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c01e50 con 0x7f7fe4022c20 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 20 add_watcher() i=0 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 20 add_watcher() i=1 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 20 add_watcher() i=2 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 20 add_watcher() i=3 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 20 add_watcher() i=4 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 20 add_watcher() i=5 2024-04-27T03:18:31.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff69f1980 20 add_watcher() i=6 2024-04-27T03:18:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.204+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94084811192560] v27'169 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff69f1980 20 add_watcher() i=7 2024-04-27T03:18:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff69f1980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c05250 con 0x5591d3bef190 2024-04-27T03:18:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94084811213824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c02010 con 0x5591d3bef190 2024-04-27T03:18:31.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94084811213824] v27'179 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7eaa7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:31.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff69f1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:31.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff69f1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:31.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.208+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c04560 con 0x5591d3bf3680 2024-04-27T03:18:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.212+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'53 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.212+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c04650 con 0x7f7fe4022c20 2024-04-27T03:18:31.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.212+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'30 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.212+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0e580 con 0x5591d3bf86e0 2024-04-27T03:18:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.216+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'30 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.216+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0e580 con 0x5591d3bef190 2024-04-27T03:18:31.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.216+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'180 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.216+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0eaf0 con 0x5591d3bf3680 2024-04-27T03:18:31.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.220+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'67 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.220+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0eb80 con 0x5591d3bf86e0 2024-04-27T03:18:31.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.220+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'51 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.220+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0f030 con 0x5591d3bf3680 2024-04-27T03:18:31.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.224+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'33 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.224+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0f030 con 0x5591d3bef190 2024-04-27T03:18:31.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.224+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'46 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.224+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0da90 con 0x5591d3bf3680 2024-04-27T03:18:31.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.228+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'23 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.228+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0da90 con 0x5591d3bf3680 2024-04-27T03:18:31.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.228+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'29 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.228+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0d300 con 0x5591d3bf3680 2024-04-27T03:18:31.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.232+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'68 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.232+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0da90 con 0x5591d3bf86e0 2024-04-27T03:18:31.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.232+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'26 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.232+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0da90 con 0x5591d3bf3680 2024-04-27T03:18:31.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.236+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'29 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.236+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0da90 con 0x5591d3bef190 2024-04-27T03:18:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.236+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'57 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.236+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bef190 2024-04-27T03:18:31.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.240+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'181 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.240+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.240+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'59 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.240+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.244+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'69 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.244+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf86e0 2024-04-27T03:18:31.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.244+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'36 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.244+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.248+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'31 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.248+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bef190 2024-04-27T03:18:31.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.252+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'182 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.252+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.252+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'60 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.252+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.256+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'39 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.256+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf86e0 2024-04-27T03:18:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.256+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'31 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.256+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.260+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'41 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.260+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf86e0 2024-04-27T03:18:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.260+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'25 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.260+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf86e0 2024-04-27T03:18:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.264+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'52 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.264+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf86e0 2024-04-27T03:18:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.264+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'31 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.264+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.268+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'35 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.268+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bef190 2024-04-27T03:18:31.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.272+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'47 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.272+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bf3680 2024-04-27T03:18:31.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.272+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'61 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.272+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x5591d3bef190 2024-04-27T03:18:31.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'58 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c0fb90 con 0x7f7fe4022c20 2024-04-27T03:18:31.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'29 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ea97e2640 20 reqs_thread_entry: start 2024-04-27T03:18:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff69f1980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff69f1980 20 Filter name: none 2024-04-27T03:18:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7e1affd640 20 reqs_thread_entry: start 2024-04-27T03:18:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff69f1980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff69f1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.276+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c2abc0 con 0x5591d3bef190 2024-04-27T03:18:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(94 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv13 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff69f1980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:31.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff69f1980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:31.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff69f1980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:31.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff69f1980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:31.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:95 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c2d090 con 0x7f7fe4022c20 2024-04-27T03:18:31.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(95 foo.buckets [call out=183b] v0'0 uv26 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.289 INFO:teuthology.orchestra.run.smithi110.stderr:could not remove user: unable to remove user, must specify purge data to remove user with buckets 2024-04-27T03:18:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7eaa7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:31.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.280+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94084811213824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c08a50 con 0x5591d3bef190 2024-04-27T03:18:31.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.284+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(96 data_loggenerations_metadata [watch unwatch cookie 94084811213824] v27'183 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.284+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94084811180064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c04c00 con 0x5591d3bf3680 2024-04-27T03:18:31.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.284+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(97 notify.0 [watch unwatch cookie 94084811180064] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.284+0000 7f7ff69f1980 20 remove_watcher() i=0 2024-04-27T03:18:31.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.284+0000 7f7ff69f1980 2 removed watcher, disabling cache 2024-04-27T03:18:31.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.284+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:98 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94084811183824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfc820 con 0x5591d3bef190 2024-04-27T03:18:31.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.288+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(98 notify.1 [watch unwatch cookie 94084811183824] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x5591d3bef190 2024-04-27T03:18:31.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.288+0000 7f7ff69f1980 20 remove_watcher() i=1 2024-04-27T03:18:31.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.288+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94084811177856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfc820 con 0x5591d3bf3680 2024-04-27T03:18:31.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.288+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(99 notify.2 [watch unwatch cookie 94084811177856] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.288+0000 7f7ff69f1980 20 remove_watcher() i=2 2024-04-27T03:18:31.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.288+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:100 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94084811189936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfc820 con 0x7f7fe4022c20 2024-04-27T03:18:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.292+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(100 notify.3 [watch unwatch cookie 94084811189936] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.292+0000 7f7ff69f1980 20 remove_watcher() i=3 2024-04-27T03:18:31.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.292+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94084811190992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfeeb0 con 0x5591d3bf3680 2024-04-27T03:18:31.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.292+0000 7f7ff4d0d640 1 -- 172.21.15.110:0/2629719299 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(101 notify.4 [watch unwatch cookie 94084811190992] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe4064040 con 0x5591d3bf3680 2024-04-27T03:18:31.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.292+0000 7f7ff69f1980 20 remove_watcher() i=4 2024-04-27T03:18:31.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.292+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94084811186096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfc820 con 0x5591d3bf86e0 2024-04-27T03:18:31.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(102 notify.5 [watch unwatch cookie 94084811186096] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 20 remove_watcher() i=5 2024-04-27T03:18:31.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:103 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94084811195792] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3bfc820 con 0x5591d3bf86e0 2024-04-27T03:18:31.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff450c640 1 -- 172.21.15.110:0/2629719299 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(103 notify.6 [watch unwatch cookie 94084811195792] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fec0d1070 con 0x5591d3bf86e0 2024-04-27T03:18:31.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 20 remove_watcher() i=6 2024-04-27T03:18:31.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:104 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94084811192560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x5591d3c00590 con 0x7f7fe4022c20 2024-04-27T03:18:31.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(104 notify.7 [watch unwatch cookie 94084811192560] v27'170 uv168 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7fe803c1c0 con 0x7f7fe4022c20 2024-04-27T03:18:31.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 20 remove_watcher() i=7 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bf3680 msgr2=0x5591d3bf5a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bf3680 0x5591d3bf5a70 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3bef190 msgr2=0x5591d3bf1650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3bef190 0x5591d3bf1650 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7fe4022c20 msgr2=0x7f7fe40173a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7fe4022c20 0x7f7fe40173a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5591d3bf86e0 msgr2=0x5591d3bfaad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5591d3bf86e0 0x5591d3bfaad0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ec4034600 msgr2=0x7f7ec4036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ec4034600 0x7f7ec4036ac0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0a3c50 tx=0x7f7fec0c8f90 comp rx=0 tx=0).stop 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 msgr2=0x5591d3ac0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2629719299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bda4a0 0x5591d3ac0820 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4024fd0 tx=0x7f7fe4049ac0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 reap_dead start 2024-04-27T03:18:31.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff3d0b640 1 -- 172.21.15.110:0/2629719299 reap_dead start 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 shutdown_connections 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 >> 172.21.15.110:0/2629719299 conn(0x5591d3bd67e0 msgr2=0x5591d3bd3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 shutdown_connections 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.296+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2629719299 wait complete. 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ecc034650 msgr2=0x7f7ecc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ecc034650 0x7f7ecc036b10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x5591d3bb9ea0 tx=0x7f7fec0e3550 comp rx=0 tx=0).stop 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bcd2d0 msgr2=0x5591d3bcd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bcd2d0 0x5591d3bcd6f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7fe400aab0 tx=0x7f7fe400af70 comp rx=0 tx=0).stop 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 shutdown_connections 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7ecc034650 0x7f7ecc036b10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/2736665463 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3bcd2d0 0x5591d3bcd6f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 >> 172.21.15.110:0/2736665463 conn(0x5591d3bc20e0 msgr2=0x5591d3bc2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 shutdown_connections 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/2736665463 wait complete. 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bba6e0 msgr2=0x5591d3bbcad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bba6e0 0x5591d3bbcad0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3ac3480 msgr2=0x5591d3bb93b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3ac3480 0x5591d3bb93b0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7edc034650 msgr2=0x7f7edc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7edc034650 0x7f7edc036b10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8000f70 tx=0x7f7fe80012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 msgr2=0x5591d3bb24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bb24d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0b8040 tx=0x7f7fec019810 comp rx=0 tx=0).stop 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 shutdown_connections 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5591d3bba6e0 0x5591d3bbcad0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5591d3ac3480 0x5591d3bb93b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f7edc034650 0x7f7edc036b10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 --2- 172.21.15.110:0/4014660092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5591d3a34850 0x5591d3bb24d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 >> 172.21.15.110:0/4014660092 conn(0x5591d3abaa20 msgr2=0x5591d3ba5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 shutdown_connections 2024-04-27T03:18:31.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.300+0000 7f7ff69f1980 1 -- 172.21.15.110:0/4014660092 wait complete. 2024-04-27T03:18:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 17 2024-04-27T03:18:31.332 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=236 failed=True 2024-04-27T03:18:31.339 INFO:tasks.radosgw_admin:RL: delete_obj myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:31.379 INFO:tasks.radosgw_admin:RL: delete_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:31.394 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:31.439 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=5 bytes_in=0 failed=False 2024-04-27T03:18:31.449 INFO:tasks.radosgw_admin:RL: put_acls myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:31.449 INFO:tasks.util.rgw:rgwadmin: client.0 : ['policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2024-04-27T03:18:31.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'xml', '-n', 'client.0', '--cluster', 'ceph', 'policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2024-04-27T03:18:31.450 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format xml -n client.0 --cluster ceph policy --bucket myfoo --object bb3aec0fdcdbc2974890f805c585d432 2024-04-27T03:18:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:31.489 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.480+0000 7f54c42b0980 1 Processor -- start 2024-04-27T03:18:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.480+0000 7f54c42b0980 1 -- start start 2024-04-27T03:18:31.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.480+0000 7f54c42b0980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556de0b5f160 0x556de0b5f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.480+0000 7f54c42b0980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x556de0a1f0d0 con 0x556de0b5fa80 2024-04-27T03:18:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.480+0000 7f54c42b0980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x556de0a14430 con 0x556de0b5f160 2024-04-27T03:18:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c1dcb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556de0b5f160 0x556de0b5f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c15ca640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x556de0b5fa80 0x556de0b60490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51242/0 (socket says 172.21.15.110:51242) 2024-04-27T03:18:31.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c15ca640 1 -- 172.21.15.110:0/4285554803 learned_addr learned my addr 172.21.15.110:0/4285554803 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c0dc9640 1 -- 172.21.15.110:0/4285554803 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315403842 0 0) 0x556de0a1f0d0 con 0x556de0b5fa80 2024-04-27T03:18:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c0dc9640 1 -- 172.21.15.110:0/4285554803 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54b0001e20 con 0x556de0b5fa80 2024-04-27T03:18:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c1dcb640 1 -- 172.21.15.110:0/4285554803 >> v1:172.21.15.77:6789/0 conn(0x556de0b5fa80 legacy=0x556de0b60490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c1dcb640 1 -- 172.21.15.110:0/4285554803 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de0a0d510 con 0x556de0b5f160 2024-04-27T03:18:31.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/4285554803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0b5f540 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f54b400a420 tx=0x7f54b401afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a30650415b5eab server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c0dc9640 1 -- 172.21.15.110:0/4285554803 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b40af070 con 0x556de0b5f160 2024-04-27T03:18:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c0dc9640 1 -- 172.21.15.110:0/4285554803 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b40aa530 con 0x556de0b5f160 2024-04-27T03:18:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.484+0000 7f54c0dc9640 1 -- 172.21.15.110:0/4285554803 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b40aa850 con 0x556de0b5f160 2024-04-27T03:18:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.488+0000 7f54c42b0980 1 -- 172.21.15.110:0/4285554803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 msgr2=0x556de0b5f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.488+0000 7f54c42b0980 1 --2- 172.21.15.110:0/4285554803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0b5f540 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f54b400a420 tx=0x7f54b401afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.488+0000 7f54c42b0980 1 -- 172.21.15.110:0/4285554803 shutdown_connections 2024-04-27T03:18:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.488+0000 7f54c42b0980 1 --2- 172.21.15.110:0/4285554803 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0b5f540 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.488+0000 7f54c42b0980 1 -- 172.21.15.110:0/4285554803 >> 172.21.15.110:0/4285554803 conn(0x556de0b5aa30 msgr2=0x556de0b5ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.488+0000 7f54c42b0980 1 -- 172.21.15.110:0/4285554803 shutdown_connections 2024-04-27T03:18:31.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.488+0000 7f54c42b0980 1 -- 172.21.15.110:0/4285554803 wait complete. 2024-04-27T03:18:31.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 Processor -- start 2024-04-27T03:18:31.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- start start 2024-04-27T03:18:31.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0ad3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556de0a632d0 con 0x556de0b5f160 2024-04-27T03:18:31.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c1dcb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0ad3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c1dcb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0ad3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40980/0 (socket says 172.21.15.110:40980) 2024-04-27T03:18:31.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c1dcb640 1 -- 172.21.15.110:0/3517628217 learned_addr learned my addr 172.21.15.110:0/3517628217 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c1dcb640 1 -- 172.21.15.110:0/3517628217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de09e9ae0 con 0x556de0b5f160 2024-04-27T03:18:31.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/3517628217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0ad3f70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f54b400a420 tx=0x7f54b40033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d0d6b2043c7ff9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f53afe07640 1 -- 172.21.15.110:0/3517628217 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b40af050 con 0x556de0b5f160 2024-04-27T03:18:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f53afe07640 1 -- 172.21.15.110:0/3517628217 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b40b0040 con 0x556de0b5f160 2024-04-27T03:18:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f53afe07640 1 -- 172.21.15.110:0/3517628217 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b40aab00 con 0x556de0b5f160 2024-04-27T03:18:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- 172.21.15.110:0/3517628217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 msgr2=0x556de0ad3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 --2- 172.21.15.110:0/3517628217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0ad3f70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f54b400a420 tx=0x7f54b40033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- 172.21.15.110:0/3517628217 shutdown_connections 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 --2- 172.21.15.110:0/3517628217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0ad3f70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- 172.21.15.110:0/3517628217 >> 172.21.15.110:0/3517628217 conn(0x556de0b5aa30 msgr2=0x556de0c46470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- 172.21.15.110:0/3517628217 shutdown_connections 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- 172.21.15.110:0/3517628217 wait complete. 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 Processor -- start 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- start start 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0c52890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.504+0000 7f54c42b0980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556de0b69200 con 0x556de0b5f160 2024-04-27T03:18:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c1dcb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0c52890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c1dcb640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0c52890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40992/0 (socket says 172.21.15.110:40992) 2024-04-27T03:18:31.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c1dcb640 1 -- 172.21.15.110:0/2151369080 learned_addr learned my addr 172.21.15.110:0/2151369080 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c1dcb640 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de0c52dd0 con 0x556de0b5f160 2024-04-27T03:18:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0c52890 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f54b400a420 tx=0x7f54b4019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f53ae604640 1 -- 172.21.15.110:0/2151369080 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b40c5070 con 0x556de0b5f160 2024-04-27T03:18:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f53ae604640 1 -- 172.21.15.110:0/2151369080 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b40b6ba0 con 0x556de0b5f160 2024-04-27T03:18:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556de0c53060 con 0x556de0b5f160 2024-04-27T03:18:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f53ae604640 1 -- 172.21.15.110:0/2151369080 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54b40c1480 con 0x556de0b5f160 2024-04-27T03:18:31.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de0c55c90 con 0x556de0b5f160 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f53ae604640 1 -- 172.21.15.110:0/2151369080 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f54b40c16f0 con 0x556de0b5f160 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f53ae604640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a80346a0 0x7f53a8036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f53ae604640 1 -- 172.21.15.110:0/2151369080 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f54b40bb070 con 0x556de0b5f160 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c15ca640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a80346a0 0x7f53a8036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0b5fa80 0x556de0c598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c59e20 con 0x556de0b5fa80 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c25cc640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0b5fa80 0x556de0c598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c15ca640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a80346a0 0x7f53a8036b60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f54ac001dc0 tx=0x7f54ac001360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.508+0000 7f54c25cc640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0b5fa80 0x556de0c598e0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c25cc640 1 -- 172.21.15.110:0/2151369080 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54bc0083f0 con 0x556de0b5fa80 2024-04-27T03:18:31.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c5abd0 0x556de0c5cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c5d600 con 0x556de0c5abd0 2024-04-27T03:18:31.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c5abd0 0x556de0c5cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c5abd0 0x556de0c5cfc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c1dcb640 1 -- 172.21.15.110:0/2151369080 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c5abd0 2024-04-27T03:18:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c5eeb0 con 0x556de0b5fa80 2024-04-27T03:18:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c25cc640 1 -- 172.21.15.110:0/2151369080 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f54bc0083f0 con 0x556de0b5fa80 2024-04-27T03:18:31.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c5e0e0 con 0x556de0c5abd0 2024-04-27T03:18:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c1dcb640 1 -- 172.21.15.110:0/2151369080 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c5abd0 2024-04-27T03:18:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c5e4d0 con 0x556de0b5fa80 2024-04-27T03:18:31.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c25cc640 1 -- 172.21.15.110:0/2151369080 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f54bc0083f0 con 0x556de0b5fa80 2024-04-27T03:18:31.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 Processor -- start 2024-04-27T03:18:31.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 -- start start 2024-04-27T03:18:31.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c6d7f0 0x556de0c6dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.512+0000 7f54c42b0980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556de0a67310 con 0x556de0c6d7f0 2024-04-27T03:18:31.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c6d7f0 0x556de0c6dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c6d7f0 0x556de0c6dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40996/0 (socket says 172.21.15.110:40996) 2024-04-27T03:18:31.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 -- 172.21.15.110:0/1170637299 learned_addr learned my addr 172.21.15.110:0/1170637299 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 -- 172.21.15.110:0/1170637299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de0c6e150 con 0x556de0c6d7f0 2024-04-27T03:18:31.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c6d7f0 0x556de0c6dc10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00aab0 tx=0x7f54bc00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f53a77fe640 1 -- 172.21.15.110:0/1170637299 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc007d30 con 0x556de0c6d7f0 2024-04-27T03:18:31.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f53a77fe640 1 -- 172.21.15.110:0/1170637299 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc01d070 con 0x556de0c6d7f0 2024-04-27T03:18:31.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556de0c6ebf0 con 0x556de0c6d7f0 2024-04-27T03:18:31.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f53a77fe640 1 -- 172.21.15.110:0/1170637299 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc018440 con 0x556de0c6d7f0 2024-04-27T03:18:31.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de0c6f0c0 con 0x556de0c6d7f0 2024-04-27T03:18:31.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f53a77fe640 1 -- 172.21.15.110:0/1170637299 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f54bc013070 con 0x556de0c6d7f0 2024-04-27T03:18:31.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f53a77fe640 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f53a77fe640 1 -- 172.21.15.110:0/1170637299 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f54bc04b520 con 0x556de0c6d7f0 2024-04-27T03:18:31.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c42b0980 1 Processor -- start 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c42b0980 1 -- start start 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x556de0c5dc00 tx=0x7f54b40e3430 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c42b0980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c4c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c42b0980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556de0ad4600 con 0x556de0c4bd70 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c4c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c4c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41012/0 (socket says 172.21.15.110:41012) 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.516+0000 7f54c25cc640 1 -- 172.21.15.110:0/1945499793 learned_addr learned my addr 172.21.15.110:0/1945499793 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c25cc640 1 -- 172.21.15.110:0/1945499793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de0c4a5c0 con 0x556de0c4bd70 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c25cc640 1 --2- 172.21.15.110:0/1945499793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c4c190 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f54bc049930 tx=0x7f54bc018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4abc8d781710ae5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f53a57fa640 1 -- 172.21.15.110:0/1945499793 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc047070 con 0x556de0c4bd70 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f53a57fa640 1 -- 172.21.15.110:0/1945499793 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc018bc0 con 0x556de0c4bd70 2024-04-27T03:18:31.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f53a57fa640 1 -- 172.21.15.110:0/1945499793 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc020360 con 0x556de0c4bd70 2024-04-27T03:18:31.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- 172.21.15.110:0/1945499793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 msgr2=0x556de0c4c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 --2- 172.21.15.110:0/1945499793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c4c190 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f54bc049930 tx=0x7f54bc018950 comp rx=0 tx=0).stop 2024-04-27T03:18:31.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- 172.21.15.110:0/1945499793 shutdown_connections 2024-04-27T03:18:31.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 --2- 172.21.15.110:0/1945499793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c4c190 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- 172.21.15.110:0/1945499793 >> 172.21.15.110:0/1945499793 conn(0x556de0c76d00 msgr2=0x556de0c74d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- 172.21.15.110:0/1945499793 shutdown_connections 2024-04-27T03:18:31.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- 172.21.15.110:0/1945499793 wait complete. 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 Processor -- start 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- start start 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c84040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556de0c4aa90 con 0x556de0c4bd70 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c25cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c84040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c25cc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c84040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41022/0 (socket says 172.21.15.110:41022) 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 learned_addr learned my addr 172.21.15.110:0/878159346 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de0c84580 con 0x556de0c4bd70 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c25cc640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c84040 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f54bc0168b0 tx=0x7f54bc0498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f53937fe640 1 -- 172.21.15.110:0/878159346 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc031070 con 0x556de0c4bd70 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556de0c84810 con 0x556de0c4bd70 2024-04-27T03:18:31.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f53937fe640 1 -- 172.21.15.110:0/878159346 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc020900 con 0x556de0c4bd70 2024-04-27T03:18:31.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f53937fe640 1 -- 172.21.15.110:0/878159346 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54bc049a70 con 0x556de0c4bd70 2024-04-27T03:18:31.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.520+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de0c874f0 con 0x556de0c4bd70 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f53937fe640 1 -- 172.21.15.110:0/878159346 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f54bc034070 con 0x556de0c4bd70 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f53937fe640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f53937fe640 1 -- 172.21.15.110:0/878159346 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f54bc037c90 con 0x556de0c4bd70 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0c8f6f0 0x556de0c91bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c92300 con 0x556de0c8f6f0 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c15ca640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0c8f6f0 0x556de0c91bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54b40d1f30 tx=0x7f54b40e6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c15ca640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0c8f6f0 0x556de0c91bb0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54ac00e660 con 0x556de0c8f6f0 2024-04-27T03:18:31.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 realm 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c8eb40 con 0x556de0c8f6f0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54ac00e660 con 0x556de0c8f6f0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c92dd0 con 0x556de0c8f6f0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54ac00e660 con 0x556de0c8f6f0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c93bb0 0x556de0c95fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c96620 con 0x556de0c93bb0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.524+0000 7f54c25cc640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c93bb0 0x556de0c95fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c25cc640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c93bb0 0x556de0c95fa0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c42b0980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c96e70 con 0x556de0c8f6f0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f54ac00e660 con 0x556de0c8f6f0 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c42b0980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c42b0980 20 searching for the correct realm 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556de0c98be0 0x556de0c9afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c98be0 2024-04-27T03:18:31.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556de0c98be0 0x556de0c9afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c1dcb640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556de0c98be0 0x556de0c9afd0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c25cc640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x7f54bc036070 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c15ca640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c15ca640 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc02d440 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac00e660 con 0x7f54bc036070 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.528+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac00e660 con 0x556de0c8f6f0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c98be0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x7f54bc036070 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x7f54bc036070 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x7f54bc036070 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f54ac029040 con 0x7f54bc036070 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.532+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c98be0 2024-04-27T03:18:31.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c98be0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c98be0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x7f54bc036070 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x7f54bc036070 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c8f6f0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x7f54bc036070 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.536+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x7f54bc036070 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c98be0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x7f54bc036070 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54ac029040 con 0x7f54bc036070 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c98be0 2024-04-27T03:18:31.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9b650 con 0x556de0c93bb0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.540+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c977c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f54ac029040 con 0x556de0c8f6f0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c97400 con 0x556de0c93bb0 2024-04-27T03:18:31.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c98120 con 0x556de0c8f6f0 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 zone default found 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 4 Realm: () 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 zonegroup default 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9bb10 con 0x7f54bc036070 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9d930 con 0x556de0c98be0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9dcd0 con 0x556de0c98be0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 started sync module instance, tier type = 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:31.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.544+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9cdd0 con 0x556de0c93bb0 2024-04-27T03:18:31.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93930411119040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9e610 con 0x556de0c93bb0 2024-04-27T03:18:31.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9ee60 con 0x556de0c8f6f0 2024-04-27T03:18:31.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93930411124240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9ef60 con 0x556de0c8f6f0 2024-04-27T03:18:31.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9fa80 con 0x556de0c93bb0 2024-04-27T03:18:31.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93930411119040] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.548+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 93930411124240] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93930411125296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9fa40 con 0x556de0c93bb0 2024-04-27T03:18:31.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca0320 con 0x7f54bc036070 2024-04-27T03:18:31.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93930411130480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9fa40 con 0x7f54bc036070 2024-04-27T03:18:31.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca0db0 con 0x556de0c93bb0 2024-04-27T03:18:31.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 93930411130480] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.552+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93930411125296] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93930411128144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca0db0 con 0x556de0c93bb0 2024-04-27T03:18:31.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca11a0 con 0x556de0c98be0 2024-04-27T03:18:31.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93930411131536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca19f0 con 0x556de0c98be0 2024-04-27T03:18:31.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca23f0 con 0x556de0c98be0 2024-04-27T03:18:31.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93930411128144] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93930411131536] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93930411136320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca1f00 con 0x556de0c98be0 2024-04-27T03:18:31.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.556+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca2e00 con 0x7f54bc036070 2024-04-27T03:18:31.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93930411136320] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'171 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93930411133152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca2e00 con 0x7f54bc036070 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=0 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=1 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=2 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=3 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=4 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=5 2024-04-27T03:18:31.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=6 2024-04-27T03:18:31.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 93930411133152] v27'172 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 20 add_watcher() i=7 2024-04-27T03:18:31.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:31.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca6080 con 0x556de0c8f6f0 2024-04-27T03:18:31.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.560+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93930411141824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0ca3390 con 0x556de0c8f6f0 2024-04-27T03:18:31.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93930411141824] v27'184 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f5373fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:31.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:31.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:31.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 5 note: GC not initialized 2024-04-27T03:18:31.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f5372ffd640 20 reqs_thread_entry: start 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 Filter name: none 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f5370ff9640 20 reqs_thread_entry: start 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0cc9dd0 con 0x7f54bc036070 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv33 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:31.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 : miss 2024-04-27T03:18:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:63 8.14 8:2ed3d538:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.5:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0ccb990 con 0x556de0c93bb0 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(63 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv1 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 info.flags=0x17 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 to cache LRU end 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 get_obj_state: octx=0x7f54ac00e5e0 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x556de0cce058 s->prefetch_data=0 2024-04-27T03:18:31.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 3.1cs0 3:3b792c84:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x556de0ccee80 con 0x556de0c8f6f0 2024-04-27T03:18:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432 [getxattrs out=848b,stat out=16b] v0'0 uv2 ondisk = 0) ==== 262+0+864 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 10 manifest: total_size = 5 2024-04-27T03:18:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.564+0000 7f54c42b0980 20 get_obj_state: setting s->obj_tag to _Br6fSRabapzaIhzRPtRoxslv5d6rPiG 2024-04-27T03:18:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.568+0000 7f5373fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.568+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93930411141824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0cc76a0 con 0x556de0c8f6f0 2024-04-27T03:18:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.568+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(65 data_loggenerations_metadata [watch unwatch cookie 93930411141824] v27'185 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.568+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93930411119040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0cc76a0 con 0x556de0c93bb0 2024-04-27T03:18:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.572+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.0 [watch unwatch cookie 93930411119040] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.572+0000 7f54c42b0980 20 remove_watcher() i=0 2024-04-27T03:18:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.572+0000 7f54c42b0980 2 removed watcher, disabling cache 2024-04-27T03:18:31.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.572+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93930411124240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9d1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.572+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(67 notify.1 [watch unwatch cookie 93930411124240] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x556de0c8f6f0 2024-04-27T03:18:31.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.572+0000 7f54c42b0980 20 remove_watcher() i=1 2024-04-27T03:18:31.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.572+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93930411125296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0cc76a0 con 0x556de0c93bb0 2024-04-27T03:18:31.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.576+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.2 [watch unwatch cookie 93930411125296] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.576+0000 7f54c42b0980 20 remove_watcher() i=2 2024-04-27T03:18:31.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.576+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93930411130480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9d1c0 con 0x7f54bc036070 2024-04-27T03:18:31.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.576+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [watch unwatch cookie 93930411130480] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.576+0000 7f54c42b0980 20 remove_watcher() i=3 2024-04-27T03:18:31.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.576+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93930411128144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0cc76a0 con 0x556de0c93bb0 2024-04-27T03:18:31.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c25cc640 1 -- 172.21.15.110:0/878159346 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 notify.4 [watch unwatch cookie 93930411128144] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54bc033030 con 0x556de0c93bb0 2024-04-27T03:18:31.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c42b0980 20 remove_watcher() i=4 2024-04-27T03:18:31.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93930411131536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9fe70 con 0x556de0c98be0 2024-04-27T03:18:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(71 notify.5 [watch unwatch cookie 93930411131536] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c42b0980 20 remove_watcher() i=5 2024-04-27T03:18:31.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93930411136320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0c9d1c0 con 0x556de0c98be0 2024-04-27T03:18:31.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c1dcb640 1 -- 172.21.15.110:0/878159346 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(72 notify.6 [watch unwatch cookie 93930411136320] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54b40d0070 con 0x556de0c98be0 2024-04-27T03:18:31.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c42b0980 20 remove_watcher() i=6 2024-04-27T03:18:31.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93930411133152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x556de0cc76a0 con 0x7f54bc036070 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(73 notify.7 [watch unwatch cookie 93930411133152] v27'173 uv171 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f54ac03c1c0 con 0x7f54bc036070 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.580+0000 7f54c42b0980 20 remove_watcher() i=7 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c93bb0 msgr2=0x556de0c95fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c93bb0 0x556de0c95fa0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0c8f6f0 msgr2=0x556de0c91bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0c8f6f0 0x556de0c91bb0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 msgr2=0x7f54bc02d440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f54bc036070 0x7f54bc02d440 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556de0c98be0 msgr2=0x556de0c9afd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556de0c98be0 0x556de0c9afd0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 msgr2=0x7f538c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f538c034650 0x7f538c036b10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54b40d1f30 tx=0x7f54b40e6000 comp rx=0 tx=0).stop 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 msgr2=0x556de0c84040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c15ca640 1 -- 172.21.15.110:0/878159346 reap_dead start 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c84040 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f54bc0168b0 tx=0x7f54bc0498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 shutdown_connections 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/878159346 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c4bd70 0x556de0c84040 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 >> 172.21.15.110:0/878159346 conn(0x556de0c76d00 msgr2=0x556de0c74360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 shutdown_connections 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/878159346 wait complete. 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 msgr2=0x7f539c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x556de0c5dc00 tx=0x7f54b40e3430 comp rx=0 tx=0).stop 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c6d7f0 msgr2=0x556de0c6dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c6d7f0 0x556de0c6dc10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00aab0 tx=0x7f54bc00af70 comp rx=0 tx=0).stop 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 shutdown_connections 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f539c034650 0x7f539c036b10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/1170637299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0c6d7f0 0x556de0c6dc10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 >> 172.21.15.110:0/1170637299 conn(0x556de0c62600 msgr2=0x556de0c63110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 shutdown_connections 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/1170637299 wait complete. 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c5abd0 msgr2=0x556de0c5cfc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c5abd0 0x556de0c5cfc0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0b5fa80 msgr2=0x556de0c598e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0b5fa80 0x556de0c598e0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a80346a0 msgr2=0x7f53a8036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a80346a0 0x7f53a8036b60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f54ac001dc0 tx=0x7f54ac001360 comp rx=0 tx=0).stop 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 msgr2=0x556de0c52890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0c52890 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f54b400a420 tx=0x7f54b4019810 comp rx=0 tx=0).stop 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 shutdown_connections 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556de0c5abd0 0x556de0c5cfc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556de0b5fa80 0x556de0c598e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53a80346a0 0x7f53a8036b60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 --2- 172.21.15.110:0/2151369080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556de0b5f160 0x556de0c52890 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 >> 172.21.15.110:0/2151369080 conn(0x556de0b5aa30 msgr2=0x556de0c44f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 shutdown_connections 2024-04-27T03:18:31.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.584+0000 7f54c42b0980 1 -- 172.21.15.110:0/2151369080 wait complete. 2024-04-27T03:18:31.604 INFO:teuthology.orchestra.run.smithi110.stdout:fooFoofooFooFULL_CONTROL 2024-04-27T03:18:31.605 DEBUG:tasks.util.rgw: raw result: fooFoofooFooFULL_CONTROL 2024-04-27T03:18:31.607 INFO:tasks.radosgw_admin:RL: get_acls myfoo foo bytes_out=0 bytes_in=417 failed=False 2024-04-27T03:18:31.613 INFO:tasks.radosgw_admin:RL: put_acls myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:31.613 INFO:tasks.util.rgw:rgwadmin: client.0 : ['policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2024-04-27T03:18:31.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'xml', '-n', 'client.0', '--cluster', 'ceph', 'policy', '--bucket', 'myfoo', '--object', 'bb3aec0fdcdbc2974890f805c585d432'] 2024-04-27T03:18:31.614 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format xml -n client.0 --cluster ceph policy --bucket myfoo --object bb3aec0fdcdbc2974890f805c585d432 2024-04-27T03:18:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:31.680 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4d816980 1 Processor -- start 2024-04-27T03:18:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4d816980 1 -- start start 2024-04-27T03:18:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4d816980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55fd2776d4c0 0x55fd2776d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4d816980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55fd2762a0d0 con 0x55fd2776dde0 2024-04-27T03:18:31.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4d816980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55fd2761f430 con 0x55fd2776d4c0 2024-04-27T03:18:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4ab30640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55fd2776dde0 0x55fd2776e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51248/0 (socket says 172.21.15.110:51248) 2024-04-27T03:18:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4b331640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55fd2776d4c0 0x55fd2776d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2659239121 learned_addr learned my addr 172.21.15.110:0/2659239121 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4b331640 1 -- 172.21.15.110:0/2659239121 >> v1:172.21.15.77:6789/0 conn(0x55fd2776dde0 legacy=0x55fd2776e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.672+0000 7f4d4b331640 1 -- 172.21.15.110:0/2659239121 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd27618510 con 0x55fd2776d4c0 2024-04-27T03:18:31.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4b331640 1 --2- 172.21.15.110:0/2659239121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776d4c0 0x55fd2776d8a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400a420 tx=0x7f4d4401afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ea684c57219e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4a32f640 1 -- 172.21.15.110:0/2659239121 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d440af070 con 0x55fd2776d4c0 2024-04-27T03:18:31.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4a32f640 1 -- 172.21.15.110:0/2659239121 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d440aa530 con 0x55fd2776d4c0 2024-04-27T03:18:31.685 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4a32f640 1 -- 172.21.15.110:0/2659239121 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d440aa850 con 0x55fd2776d4c0 2024-04-27T03:18:31.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4d816980 1 -- 172.21.15.110:0/2659239121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776d4c0 msgr2=0x55fd2776d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2659239121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776d4c0 0x55fd2776d8a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400a420 tx=0x7f4d4401afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4d816980 1 -- 172.21.15.110:0/2659239121 shutdown_connections 2024-04-27T03:18:31.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2659239121 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776d4c0 0x55fd2776d8a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4d816980 1 -- 172.21.15.110:0/2659239121 >> 172.21.15.110:0/2659239121 conn(0x55fd276e0ba0 msgr2=0x55fd276e0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4d816980 1 -- 172.21.15.110:0/2659239121 shutdown_connections 2024-04-27T03:18:31.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.676+0000 7f4d4d816980 1 -- 172.21.15.110:0/2659239121 wait complete. 2024-04-27T03:18:31.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4d816980 1 Processor -- start 2024-04-27T03:18:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4d816980 1 -- start start 2024-04-27T03:18:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4d816980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776dde0 0x55fd27858f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4d816980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fd2766e2d0 con 0x55fd2776dde0 2024-04-27T03:18:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4b331640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776dde0 0x55fd27858f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4b331640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776dde0 0x55fd27858f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41044/0 (socket says 172.21.15.110:41044) 2024-04-27T03:18:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4b331640 1 -- 172.21.15.110:0/1181171810 learned_addr learned my addr 172.21.15.110:0/1181171810 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.692+0000 7f4d4b331640 1 -- 172.21.15.110:0/1181171810 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd275f4ae0 con 0x55fd2776dde0 2024-04-27T03:18:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4b331640 1 --2- 172.21.15.110:0/1181171810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776dde0 0x55fd27858f40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4d440ad040 tx=0x7f4d440033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5753cd9f209ec56 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4c3960a640 1 -- 172.21.15.110:0/1181171810 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d440af050 con 0x55fd2776dde0 2024-04-27T03:18:31.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4c3960a640 1 -- 172.21.15.110:0/1181171810 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d440b0040 con 0x55fd2776dde0 2024-04-27T03:18:31.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4c3960a640 1 -- 172.21.15.110:0/1181171810 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d440aab30 con 0x55fd2776dde0 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- 172.21.15.110:0/1181171810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776dde0 msgr2=0x55fd27858f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 --2- 172.21.15.110:0/1181171810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776dde0 0x55fd27858f40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4d440ad040 tx=0x7f4d440033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- 172.21.15.110:0/1181171810 shutdown_connections 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 --2- 172.21.15.110:0/1181171810 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2776dde0 0x55fd27858f40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- 172.21.15.110:0/1181171810 >> 172.21.15.110:0/1181171810 conn(0x55fd276e0ba0 msgr2=0x55fd27854a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- 172.21.15.110:0/1181171810 shutdown_connections 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- 172.21.15.110:0/1181171810 wait complete. 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 Processor -- start 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- start start 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278600b0 0x55fd27860490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fd27777510 con 0x55fd278600b0 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4b331640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278600b0 0x55fd27860490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4b331640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278600b0 0x55fd27860490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41048/0 (socket says 172.21.15.110:41048) 2024-04-27T03:18:31.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4b331640 1 -- 172.21.15.110:0/3283735800 learned_addr learned my addr 172.21.15.110:0/3283735800 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4b331640 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd278609d0 con 0x55fd278600b0 2024-04-27T03:18:31.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4b331640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278600b0 0x55fd27860490 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4d44002240 tx=0x7f4d44017d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4c37e07640 1 -- 172.21.15.110:0/3283735800 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d440b6be0 con 0x55fd278600b0 2024-04-27T03:18:31.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fd27860c60 con 0x55fd278600b0 2024-04-27T03:18:31.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4c37e07640 1 -- 172.21.15.110:0/3283735800 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d440b6d80 con 0x55fd278600b0 2024-04-27T03:18:31.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd27863640 con 0x55fd278600b0 2024-04-27T03:18:31.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.696+0000 7f4c37e07640 1 -- 172.21.15.110:0/3283735800 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d440c05a0 con 0x55fd278600b0 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4c37e07640 1 -- 172.21.15.110:0/3283735800 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4d440c0740 con 0x55fd278600b0 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4c37e07640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c300346a0 0x7f4c30036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4c37e07640 1 -- 172.21.15.110:0/3283735800 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f4d44002040 con 0x55fd278600b0 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4ab30640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c300346a0 0x7f4c30036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd27865280 0x55fd27867670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd27867ce0 con 0x55fd27865280 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd27865280 0x55fd27867670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4ab30640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c300346a0 0x7f4c30036b60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4d34000f70 tx=0x7f4d340012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd27865280 0x55fd27867670 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4bb32640 1 -- 172.21.15.110:0/3283735800 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4d3c0083f0 con 0x55fd27865280 2024-04-27T03:18:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd27868b00 0x55fd2786aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd2786b530 con 0x55fd27868b00 2024-04-27T03:18:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4b331640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd27868b00 0x55fd2786aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4b331640 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd27868b00 0x55fd2786aef0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4b331640 1 -- 172.21.15.110:0/3283735800 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd27868b00 2024-04-27T03:18:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.700+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd2786cde0 con 0x55fd27865280 2024-04-27T03:18:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4bb32640 1 -- 172.21.15.110:0/3283735800 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f4d3c0083f0 con 0x55fd27865280 2024-04-27T03:18:31.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd2786c010 con 0x55fd27868b00 2024-04-27T03:18:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4b331640 1 -- 172.21.15.110:0/3283735800 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd27868b00 2024-04-27T03:18:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd2786c400 con 0x55fd27865280 2024-04-27T03:18:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4bb32640 1 -- 172.21.15.110:0/3283735800 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f4d3c0083f0 con 0x55fd27865280 2024-04-27T03:18:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 Processor -- start 2024-04-27T03:18:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 -- start start 2024-04-27T03:18:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2787b6f0 0x55fd2787bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.713 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fd27672310 con 0x55fd2787b6f0 2024-04-27T03:18:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4bb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2787b6f0 0x55fd2787bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4bb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2787b6f0 0x55fd2787bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41054/0 (socket says 172.21.15.110:41054) 2024-04-27T03:18:31.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2399861547 learned_addr learned my addr 172.21.15.110:0/2399861547 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2399861547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd2787c050 con 0x55fd2787b6f0 2024-04-27T03:18:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2787b6f0 0x55fd2787bb10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00aab0 tx=0x7f4d3c00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4c35602640 1 -- 172.21.15.110:0/2399861547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d3c007d30 con 0x55fd2787b6f0 2024-04-27T03:18:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fd2787caf0 con 0x55fd2787b6f0 2024-04-27T03:18:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4c35602640 1 -- 172.21.15.110:0/2399861547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3c018440 con 0x55fd2787b6f0 2024-04-27T03:18:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4c35602640 1 -- 172.21.15.110:0/2399861547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d3c0185f0 con 0x55fd2787b6f0 2024-04-27T03:18:31.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.704+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd2787cfc0 con 0x55fd2787b6f0 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4c35602640 1 -- 172.21.15.110:0/2399861547 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4d3c013070 con 0x55fd2787b6f0 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4c35602640 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c24034650 0x7f4c24036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4c35602640 1 -- 172.21.15.110:0/2399861547 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f4d3c04b7d0 con 0x55fd2787b6f0 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4b331640 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c24034650 0x7f4c24036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 Processor -- start 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 -- start start 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd27888cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4b331640 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c24034650 0x7f4c24036b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55fd2786bb30 tx=0x7f4d440d3040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fd278595d0 con 0x55fd278888a0 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4bb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd27888cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4bb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd27888cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41070/0 (socket says 172.21.15.110:41070) 2024-04-27T03:18:31.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2432556387 learned_addr learned my addr 172.21.15.110:0/2432556387 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2432556387 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd27889290 con 0x55fd278888a0 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/2432556387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd27888cc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c049b20 tx=0x7f4d3c016c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efff504385e38653 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4c2effd640 1 -- 172.21.15.110:0/2432556387 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d3c0237f0 con 0x55fd278888a0 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4c2effd640 1 -- 172.21.15.110:0/2432556387 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3c016330 con 0x55fd278888a0 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4c2effd640 1 -- 172.21.15.110:0/2432556387 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d3c023990 con 0x55fd278888a0 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 -- 172.21.15.110:0/2432556387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 msgr2=0x55fd27888cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2432556387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd27888cc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c049b20 tx=0x7f4d3c016c80 comp rx=0 tx=0).stop 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 -- 172.21.15.110:0/2432556387 shutdown_connections 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2432556387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd27888cc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 -- 172.21.15.110:0/2432556387 >> 172.21.15.110:0/2432556387 conn(0x55fd27884c30 msgr2=0x55fd27882c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 -- 172.21.15.110:0/2432556387 shutdown_connections 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.708+0000 7f4d4d816980 1 -- 172.21.15.110:0/2432556387 wait complete. 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 Processor -- start 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 -- start start 2024-04-27T03:18:31.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd2788fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fd27889a00 con 0x55fd278888a0 2024-04-27T03:18:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4bb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd2788fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4bb32640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd2788fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41082/0 (socket says 172.21.15.110:41082) 2024-04-27T03:18:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 learned_addr learned my addr 172.21.15.110:0/2625601787 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd27890310 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd2788fdd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c016fd0 tx=0x7f4d3c049ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4c2d7fa640 1 -- 172.21.15.110:0/2625601787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d3c032070 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fd278905a0 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4c2d7fa640 1 -- 172.21.15.110:0/2625601787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3c04b2e0 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4c2d7fa640 1 -- 172.21.15.110:0/2625601787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d3c04b5c0 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd27893280 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4c2d7fa640 1 -- 172.21.15.110:0/2625601787 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4d3c035070 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4c2d7fa640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c140347f0 0x7f4c14036cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4c2d7fa640 1 -- 172.21.15.110:0/2625601787 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f4d3c038c90 con 0x55fd278888a0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4b331640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c140347f0 0x7f4c14036cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd2789b480 0x55fd2789d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd2789e090 con 0x55fd2789b480 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.712+0000 7f4d4ab30640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd2789b480 0x55fd2789d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4b331640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c140347f0 0x7f4c14036cb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4d440e2440 tx=0x7f4d440da000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4ab30640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd2789b480 0x55fd2789d940 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4d3400e5a0 con 0x55fd2789b480 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 realm 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd2789a8d0 con 0x55fd2789b480 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4d3400e5a0 con 0x55fd2789b480 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd2789eb90 con 0x55fd2789b480 2024-04-27T03:18:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4d3400e5a0 con 0x55fd2789b480 2024-04-27T03:18:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd2789f970 0x55fd278a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a2410 con 0x55fd2789f970 2024-04-27T03:18:31.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd2789f970 0x55fd278a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd2789f970 0x55fd278a1d60 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a2c60 con 0x55fd2789b480 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f4d3400e5a0 con 0x55fd2789b480 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 20 searching for the correct realm 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fd278a49d0 0x55fd278a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd278a49d0 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4b331640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fd278a49d0 0x55fd278a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.716+0000 7f4d4b331640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fd278a49d0 0x55fd278a6dc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4bb32640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4d3c037070 0x7f4d3c018790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x7f4d3c037070 2024-04-27T03:18:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4d3c037070 0x7f4d3c018790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4d3c037070 0x7f4d3c018790 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3400e5a0 con 0x7f4d3c037070 2024-04-27T03:18:31.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3400e5a0 con 0x55fd2789b480 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd278a49d0 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.720+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x7f4d3c037070 2024-04-27T03:18:31.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x7f4d3c037070 2024-04-27T03:18:31.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x7f4d3c037070 2024-04-27T03:18:31.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f4d34029040 con 0x7f4d3c037070 2024-04-27T03:18:31.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd278a49d0 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.724+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd278a49d0 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd278a49d0 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x7f4d3c037070 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x7f4d3c037070 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789b480 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x7f4d3c037070 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x7f4d3c037070 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd278a49d0 2024-04-27T03:18:31.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x7f4d3c037070 2024-04-27T03:18:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d34029040 con 0x7f4d3c037070 2024-04-27T03:18:31.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.728+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd278a49d0 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7440 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a3610 con 0x55fd2789b480 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4d34029040 con 0x55fd2789b480 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a3eb0 con 0x55fd2789f970 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a3250 con 0x55fd2789b480 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 zone default found 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 4 Realm: () 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 zonegroup default 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a7910 con 0x7f4d3c037070 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.732+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a9730 con 0x55fd278a49d0 2024-04-27T03:18:31.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a9b00 con 0x55fd278a49d0 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 20 started sync module instance, tier type = 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:31.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a8bd0 con 0x55fd2789f970 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94545778479344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278a8bd0 con 0x55fd2789f970 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.736+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278aa5c0 con 0x55fd2789b480 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94545778483408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278aa4f0 con 0x55fd2789b480 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ab510 con 0x55fd2789f970 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94545778479344] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94545778483408] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94545778485728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ab4d0 con 0x55fd2789f970 2024-04-27T03:18:31.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.740+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278abdb0 con 0x7f4d3c037070 2024-04-27T03:18:31.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94545778489600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ab4d0 con 0x7f4d3c037070 2024-04-27T03:18:31.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ac860 con 0x55fd2789f970 2024-04-27T03:18:31.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94545778485728] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94545778489600] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94545778490656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ac860 con 0x55fd2789f970 2024-04-27T03:18:31.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278acc50 con 0x55fd278a49d0 2024-04-27T03:18:31.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94545778487296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ad4a0 con 0x55fd278a49d0 2024-04-27T03:18:31.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.744+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ade60 con 0x55fd278a49d0 2024-04-27T03:18:31.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94545778490656] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94545778487296] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94545778497968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ae280 con 0x55fd278a49d0 2024-04-27T03:18:31.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278aead0 con 0x7f4d3c037070 2024-04-27T03:18:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94545778497968] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'174 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94545778500224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278aeef0 con 0x7f4d3c037070 2024-04-27T03:18:31.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=0 2024-04-27T03:18:31.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=1 2024-04-27T03:18:31.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=2 2024-04-27T03:18:31.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=3 2024-04-27T03:18:31.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=4 2024-04-27T03:18:31.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=5 2024-04-27T03:18:31.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=6 2024-04-27T03:18:31.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94545778500224] v27'175 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 20 add_watcher() i=7 2024-04-27T03:18:31.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.748+0000 7f4d4d816980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:31.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278b1f20 con 0x55fd2789b480 2024-04-27T03:18:31.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94545778501280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278af460 con 0x55fd2789b480 2024-04-27T03:18:31.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94545778501280] v27'186 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4c01fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:31.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:31.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:31.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 5 note: GC not initialized 2024-04-27T03:18:31.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4c00fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:31.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:31.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 20 Filter name: none 2024-04-27T03:18:31.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4b7a7fc640 20 reqs_thread_entry: start 2024-04-27T03:18:31.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278d5c40 con 0x7f4d3c037070 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(62 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv33 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 : miss 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.752+0000 7f4d4d816980 20 rados->read ofs=0 len=0 2024-04-27T03:18:31.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:63 8.14 8:2ed3d538:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.5:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278d7800 con 0x55fd2789f970 2024-04-27T03:18:31.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(63 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv1 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:31.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 info.flags=0x17 2024-04-27T03:18:31.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 to cache LRU end 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 20 get_obj_state: octx=0x7f4d3400e520 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x55fd278d9ec8 s->prefetch_data=0 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 3.1cs0 3:3b792c84:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55fd278dacf0 con 0x55fd2789b480 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432 [getxattrs out=910b,stat out=16b] v0'0 uv3 ondisk = 0) ==== 262+0+926 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 10 manifest: total_size = 5 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 20 get_obj_state: setting s->obj_tag to _dskPOg4xx-r9FrhG4hncHE-fznh_Fxj 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4c01fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.756+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94545778501280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278d3510 con 0x55fd2789b480 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(65 data_loggenerations_metadata [watch unwatch cookie 94545778501280] v27'187 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94545778479344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278d3510 con 0x55fd2789f970 2024-04-27T03:18:31.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.0 [watch unwatch cookie 94545778479344] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4d816980 20 remove_watcher() i=0 2024-04-27T03:18:31.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4d816980 2 removed watcher, disabling cache 2024-04-27T03:18:31.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94545778483408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278d3510 con 0x55fd2789b480 2024-04-27T03:18:31.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(67 notify.1 [watch unwatch cookie 94545778483408] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x55fd2789b480 2024-04-27T03:18:31.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4d816980 20 remove_watcher() i=1 2024-04-27T03:18:31.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.760+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94545778485728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278aa0d0 con 0x55fd2789f970 2024-04-27T03:18:31.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.764+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.2 [watch unwatch cookie 94545778485728] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.764+0000 7f4d4d816980 20 remove_watcher() i=2 2024-04-27T03:18:31.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.764+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94545778489600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278d3510 con 0x7f4d3c037070 2024-04-27T03:18:31.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.764+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [watch unwatch cookie 94545778489600] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.764+0000 7f4d4d816980 20 remove_watcher() i=3 2024-04-27T03:18:31.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.764+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94545778490656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278aa0d0 con 0x55fd2789f970 2024-04-27T03:18:31.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.768+0000 7f4d4bb32640 1 -- 172.21.15.110:0/2625601787 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 notify.4 [watch unwatch cookie 94545778490656] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3c053040 con 0x55fd2789f970 2024-04-27T03:18:31.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.768+0000 7f4d4d816980 20 remove_watcher() i=4 2024-04-27T03:18:31.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.768+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94545778487296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278d3510 con 0x55fd278a49d0 2024-04-27T03:18:31.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.768+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(71 notify.5 [watch unwatch cookie 94545778487296] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.768+0000 7f4d4d816980 20 remove_watcher() i=5 2024-04-27T03:18:31.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.768+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94545778497968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278aa0d0 con 0x55fd278a49d0 2024-04-27T03:18:31.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4b331640 1 -- 172.21.15.110:0/2625601787 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(72 notify.6 [watch unwatch cookie 94545778497968] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d440b2070 con 0x55fd278a49d0 2024-04-27T03:18:31.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 20 remove_watcher() i=6 2024-04-27T03:18:31.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94545778500224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55fd278ab000 con 0x7f4d3c037070 2024-04-27T03:18:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(73 notify.7 [watch unwatch cookie 94545778500224] v27'176 uv174 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4d3403c1c0 con 0x7f4d3c037070 2024-04-27T03:18:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 20 remove_watcher() i=7 2024-04-27T03:18:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd2789f970 msgr2=0x55fd278a1d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd2789f970 0x55fd278a1d60 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd2789b480 msgr2=0x55fd2789d940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd2789b480 0x55fd2789d940 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4d3c037070 msgr2=0x7f4d3c018790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4d3c037070 0x7f4d3c018790 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fd278a49d0 msgr2=0x55fd278a6dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fd278a49d0 0x55fd278a6dc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c140347f0 msgr2=0x7f4c14036cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c140347f0 0x7f4c14036cb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4d440e2440 tx=0x7f4d440da000 comp rx=0 tx=0).stop 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 msgr2=0x55fd2788fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4ab30640 1 -- 172.21.15.110:0/2625601787 reap_dead start 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd2788fdd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c016fd0 tx=0x7f4d3c049ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 shutdown_connections 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2625601787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278888a0 0x55fd2788fdd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 >> 172.21.15.110:0/2625601787 conn(0x55fd27884c30 msgr2=0x55fd2788a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 shutdown_connections 2024-04-27T03:18:31.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2625601787 wait complete. 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c24034650 msgr2=0x7f4c24036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c24034650 0x7f4c24036b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55fd2786bb30 tx=0x7f4d440d3040 comp rx=0 tx=0).stop 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2787b6f0 msgr2=0x55fd2787bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2787b6f0 0x55fd2787bb10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00aab0 tx=0x7f4d3c00af70 comp rx=0 tx=0).stop 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 shutdown_connections 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c24034650 0x7f4c24036b10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/2399861547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd2787b6f0 0x55fd2787bb10 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 >> 172.21.15.110:0/2399861547 conn(0x55fd27870500 msgr2=0x55fd27871010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 shutdown_connections 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/2399861547 wait complete. 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd27868b00 msgr2=0x55fd2786aef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd27868b00 0x55fd2786aef0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd27865280 msgr2=0x55fd27867670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd27865280 0x55fd27867670 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c300346a0 msgr2=0x7f4c30036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c300346a0 0x7f4c30036b60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4d34000f70 tx=0x7f4d340012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278600b0 msgr2=0x55fd27860490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278600b0 0x55fd27860490 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4d44002240 tx=0x7f4d44017d70 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 shutdown_connections 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fd27868b00 0x55fd2786aef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fd27865280 0x55fd27867670 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4c300346a0 0x7f4c30036b60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 --2- 172.21.15.110:0/3283735800 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fd278600b0 0x55fd27860490 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 >> 172.21.15.110:0/3283735800 conn(0x55fd276e0ba0 msgr2=0x55fd27853540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 shutdown_connections 2024-04-27T03:18:31.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.772+0000 7f4d4d816980 1 -- 172.21.15.110:0/3283735800 wait complete. 2024-04-27T03:18:31.793 INFO:teuthology.orchestra.run.smithi110.stdout:fooFoohttp://acs.amazonaws.com/groups/global/AllUsersREADfooFooFULL_CONTROL 2024-04-27T03:18:31.793 DEBUG:tasks.util.rgw: raw result: fooFoohttp://acs.amazonaws.com/groups/global/AllUsersREADfooFooFULL_CONTROL 2024-04-27T03:18:31.796 INFO:tasks.radosgw_admin:RL: get_acls myfoo foo bytes_out=0 bytes_in=609 failed=False 2024-04-27T03:18:31.820 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:31.931 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=5 bytes_in=0 failed=False 2024-04-27T03:18:31.931 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=4 bytes_in=0 failed=False 2024-04-27T03:18:31.931 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=3 bytes_in=0 failed=False 2024-04-27T03:18:31.931 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=6 bytes_in=0 failed=False 2024-04-27T03:18:31.932 INFO:tasks.util.rgw:rgwadmin: client.0 : ['bucket', 'rm', '--bucket', 'myfoo', '--purge-objects'] 2024-04-27T03:18:31.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'bucket', 'rm', '--bucket', 'myfoo', '--purge-objects'] 2024-04-27T03:18:31.932 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph bucket rm --bucket myfoo --purge-objects 2024-04-27T03:18:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d418a980 1 Processor -- start 2024-04-27T03:18:31.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d418a980 1 -- start start 2024-04-27T03:18:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d418a980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a583ea4160 0x55a583ea4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d418a980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a583d640d0 con 0x55a583ea4a80 2024-04-27T03:18:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d418a980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a583d59430 con 0x55a583ea4160 2024-04-27T03:18:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d1ca5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a583ea4160 0x55a583ea4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d14a4640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55a583ea4a80 0x55a583ea5490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51252/0 (socket says 172.21.15.110:51252) 2024-04-27T03:18:31.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d14a4640 1 -- 172.21.15.110:0/2699434795 learned_addr learned my addr 172.21.15.110:0/2699434795 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d1ca5640 1 -- 172.21.15.110:0/2699434795 >> v1:172.21.15.77:6789/0 conn(0x55a583ea4a80 legacy=0x55a583ea5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.964+0000 7f84d1ca5640 1 -- 172.21.15.110:0/2699434795 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a583d52510 con 0x55a583ea4160 2024-04-27T03:18:31.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/2699434795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583ea4540 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f84c400a480 tx=0x7f84c401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7dfdd3eb4f1019b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d0ca3640 1 -- 172.21.15.110:0/2699434795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c40b0070 con 0x55a583ea4160 2024-04-27T03:18:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d0ca3640 1 -- 172.21.15.110:0/2699434795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c40ab540 con 0x55a583ea4160 2024-04-27T03:18:31.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d0ca3640 1 -- 172.21.15.110:0/2699434795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c40ab840 con 0x55a583ea4160 2024-04-27T03:18:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d418a980 1 -- 172.21.15.110:0/2699434795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 msgr2=0x55a583ea4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d418a980 1 --2- 172.21.15.110:0/2699434795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583ea4540 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f84c400a480 tx=0x7f84c401b280 comp rx=0 tx=0).stop 2024-04-27T03:18:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d418a980 1 -- 172.21.15.110:0/2699434795 shutdown_connections 2024-04-27T03:18:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d418a980 1 --2- 172.21.15.110:0/2699434795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583ea4540 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d418a980 1 -- 172.21.15.110:0/2699434795 >> 172.21.15.110:0/2699434795 conn(0x55a583e9fa30 msgr2=0x55a583ea1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d418a980 1 -- 172.21.15.110:0/2699434795 shutdown_connections 2024-04-27T03:18:31.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.968+0000 7f84d418a980 1 -- 172.21.15.110:0/2699434795 wait complete. 2024-04-27T03:18:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d418a980 1 Processor -- start 2024-04-27T03:18:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d418a980 1 -- start start 2024-04-27T03:18:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d418a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f8fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d418a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a583da82d0 con 0x55a583ea4160 2024-04-27T03:18:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d1ca5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f8fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d1ca5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f8fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41102/0 (socket says 172.21.15.110:41102) 2024-04-27T03:18:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d1ca5640 1 -- 172.21.15.110:0/3241085031 learned_addr learned my addr 172.21.15.110:0/3241085031 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d1ca5640 1 -- 172.21.15.110:0/3241085031 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a583d2eae0 con 0x55a583ea4160 2024-04-27T03:18:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/3241085031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f8fa80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f84c40ae040 tx=0x7f84c4003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287164fcb7751345 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f83bfe07640 1 -- 172.21.15.110:0/3241085031 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c40b0040 con 0x55a583ea4160 2024-04-27T03:18:31.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f83bfe07640 1 -- 172.21.15.110:0/3241085031 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c40ab9d0 con 0x55a583ea4160 2024-04-27T03:18:31.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.984+0000 7f83bfe07640 1 -- 172.21.15.110:0/3241085031 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c40abcf0 con 0x55a583ea4160 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- 172.21.15.110:0/3241085031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 msgr2=0x55a583f8fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 --2- 172.21.15.110:0/3241085031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f8fa80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f84c40ae040 tx=0x7f84c4003030 comp rx=0 tx=0).stop 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- 172.21.15.110:0/3241085031 shutdown_connections 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 --2- 172.21.15.110:0/3241085031 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f8fa80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- 172.21.15.110:0/3241085031 >> 172.21.15.110:0/3241085031 conn(0x55a583e9fa30 msgr2=0x55a583f8b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- 172.21.15.110:0/3241085031 shutdown_connections 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- 172.21.15.110:0/3241085031 wait complete. 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 Processor -- start 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- start start 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f97aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a583eae200 con 0x55a583ea4160 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d1ca5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f97aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d1ca5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f97aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41114/0 (socket says 172.21.15.110:41114) 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d1ca5640 1 -- 172.21.15.110:0/3910604231 learned_addr learned my addr 172.21.15.110:0/3910604231 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d1ca5640 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a583f97fe0 con 0x55a583ea4160 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f97aa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f84c4018eb0 tx=0x7f84c4018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f83be604640 1 -- 172.21.15.110:0/3910604231 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c40b0070 con 0x55a583ea4160 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a583f98270 con 0x55a583ea4160 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f83be604640 1 -- 172.21.15.110:0/3910604231 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c40b68f0 con 0x55a583ea4160 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f83be604640 1 -- 172.21.15.110:0/3910604231 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84c40b6bd0 con 0x55a583ea4160 2024-04-27T03:18:31.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.988+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a583f9aea0 con 0x55a583ea4160 2024-04-27T03:18:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f83be604640 1 -- 172.21.15.110:0/3910604231 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f84c40bf3f0 con 0x55a583ea4160 2024-04-27T03:18:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f83be604640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83b8034650 0x7f83b8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f83be604640 1 -- 172.21.15.110:0/3910604231 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f84c40bc070 con 0x55a583ea4160 2024-04-27T03:18:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d14a4640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83b8034650 0x7f83b8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583ea4a80 0x55a583f9e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:31.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583f9ef50 con 0x55a583ea4a80 2024-04-27T03:18:32.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d24a6640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583ea4a80 0x55a583f9e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d14a4640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83b8034650 0x7f83b8036b10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f84c8000f70 tx=0x7f84c80012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d24a6640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583ea4a80 0x55a583f9e980 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d24a6640 1 -- 172.21.15.110:0/3910604231 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f84cc00a4d0 con 0x55a583ea4a80 2024-04-27T03:18:32.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583f9fd90 0x55a583fa2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fa27b0 con 0x55a583f9fd90 2024-04-27T03:18:32.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583f9fd90 0x55a583fa2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583f9fd90 0x55a583fa2180 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d1ca5640 1 -- 172.21.15.110:0/3910604231 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583f9fd90 2024-04-27T03:18:32.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fa4060 con 0x55a583ea4a80 2024-04-27T03:18:32.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d24a6640 1 -- 172.21.15.110:0/3910604231 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f84cc00a4d0 con 0x55a583ea4a80 2024-04-27T03:18:32.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.992+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fa3290 con 0x55a583f9fd90 2024-04-27T03:18:32.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d1ca5640 1 -- 172.21.15.110:0/3910604231 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583f9fd90 2024-04-27T03:18:32.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fa3680 con 0x55a583ea4a80 2024-04-27T03:18:32.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d24a6640 1 -- 172.21.15.110:0/3910604231 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f84cc00a4d0 con 0x55a583ea4a80 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d418a980 1 Processor -- start 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d418a980 1 -- start start 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d418a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583f94c00 0x55a583f930a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d418a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a583dac310 con 0x55a583f94c00 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d24a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583f94c00 0x55a583f930a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d24a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583f94c00 0x55a583f930a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41116/0 (socket says 172.21.15.110:41116) 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d24a6640 1 -- 172.21.15.110:0/3597462853 learned_addr learned my addr 172.21.15.110:0/3597462853 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d24a6640 1 -- 172.21.15.110:0/3597462853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a583f935e0 con 0x55a583f94c00 2024-04-27T03:18:32.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d24a6640 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583f94c00 0x55a583f930a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00df50 tx=0x7f84cc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f83b77fe640 1 -- 172.21.15.110:0/3597462853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84cc012840 con 0x55a583f94c00 2024-04-27T03:18:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f83b77fe640 1 -- 172.21.15.110:0/3597462853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84cc0129e0 con 0x55a583f94c00 2024-04-27T03:18:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a583f93870 con 0x55a583f94c00 2024-04-27T03:18:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f83b77fe640 1 -- 172.21.15.110:0/3597462853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84cc012d00 con 0x55a583f94c00 2024-04-27T03:18:32.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a583f93d40 con 0x55a583f94c00 2024-04-27T03:18:32.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:31.996+0000 7f83b77fe640 1 -- 172.21.15.110:0/3597462853 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f84cc014410 con 0x55a583f94c00 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f83b77fe640 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83ac034650 0x7f83ac036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f83b77fe640 1 -- 172.21.15.110:0/3597462853 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f84cc00e040 con 0x55a583f94c00 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83ac034650 0x7f83ac036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 Processor -- start 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- start start 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83ac034650 0x7f83ac036b10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55a583fa2db0 tx=0x7f84c40e2be0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a583f90110 con 0x55a583fc1d80 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41128/0 (socket says 172.21.15.110:41128) 2024-04-27T03:18:32.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 -- 172.21.15.110:0/3895685473 learned_addr learned my addr 172.21.15.110:0/3895685473 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 -- 172.21.15.110:0/3895685473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a583fc2770 con 0x55a583fc1d80 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 --2- 172.21.15.110:0/3895685473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc21a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f84cc04a640 tx=0x7f84cc04c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fc986bbd06cf9e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f83b57fa640 1 -- 172.21.15.110:0/3895685473 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84cc03c070 con 0x55a583fc1d80 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f83b57fa640 1 -- 172.21.15.110:0/3895685473 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84cc014c20 con 0x55a583fc1d80 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f83b57fa640 1 -- 172.21.15.110:0/3895685473 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84cc011c50 con 0x55a583fc1d80 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- 172.21.15.110:0/3895685473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 msgr2=0x55a583fc21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 --2- 172.21.15.110:0/3895685473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc21a0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f84cc04a640 tx=0x7f84cc04c1c0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- 172.21.15.110:0/3895685473 shutdown_connections 2024-04-27T03:18:32.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 --2- 172.21.15.110:0/3895685473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc21a0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- 172.21.15.110:0/3895685473 >> 172.21.15.110:0/3895685473 conn(0x55a583fbe0c0 msgr2=0x55a583fbc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- 172.21.15.110:0/3895685473 shutdown_connections 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- 172.21.15.110:0/3895685473 wait complete. 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 Processor -- start 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- start start 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d418a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a583fc2ed0 con 0x55a583fc1d80 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41136/0 (socket says 172.21.15.110:41136) 2024-04-27T03:18:32.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 learned_addr learned my addr 172.21.15.110:0/1567361234 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.000+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a583fc9750 con 0x55a583fc1d80 2024-04-27T03:18:32.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d24a6640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc9210 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f84cc01e4b0 tx=0x7f84cc04a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f83a37fe640 1 -- 172.21.15.110:0/1567361234 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84cc014830 con 0x55a583fc1d80 2024-04-27T03:18:32.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f83a37fe640 1 -- 172.21.15.110:0/1567361234 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84cc01eb60 con 0x55a583fc1d80 2024-04-27T03:18:32.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a583fc99e0 con 0x55a583fc1d80 2024-04-27T03:18:32.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f83a37fe640 1 -- 172.21.15.110:0/1567361234 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f84cc01ee40 con 0x55a583fc1d80 2024-04-27T03:18:32.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a583fcc6c0 con 0x55a583fc1d80 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f83a37fe640 1 -- 172.21.15.110:0/1567361234 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f84cc04aa50 con 0x55a583fc1d80 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f83a37fe640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f839c034600 0x7f839c036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f83a37fe640 1 -- 172.21.15.110:0/1567361234 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f84cc033c70 con 0x55a583fc1d80 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f839c034600 0x7f839c036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583fd4870 0x55a583fd6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fd7480 con 0x55a583fd4870 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d14a4640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583fd4870 0x55a583fd6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f839c034600 0x7f839c036ac0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f84c4006b40 tx=0x7f84c40dd000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d14a4640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583fd4870 0x55a583fd6d30 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f84c800e5a0 con 0x55a583fd4870 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 realm 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fd3d10 con 0x55a583fd4870 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f84c800e5a0 con 0x55a583fd4870 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fd7f80 con 0x55a583fd4870 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f84c800e5a0 con 0x55a583fd4870 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583fd8d60 0x55a583fdb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.004+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fdb800 con 0x55a583fd8d60 2024-04-27T03:18:32.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d24a6640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583fd8d60 0x55a583fdb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d24a6640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583fd8d60 0x55a583fdb150 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f84cc038020 con 0x55a583fd8d60 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d418a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fdc050 con 0x55a583fd4870 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f84c800e5a0 con 0x55a583fd4870 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d418a980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d418a980 20 searching for the correct realm 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a583fdddc0 0x55a583fe01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fdddc0 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a583fdddc0 0x55a583fe01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d1ca5640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a583fdddc0 0x55a583fe01b0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc038020 con 0x55a583fd8d60 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d24a6640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f84cc014030 0x7f84cc013890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x7f84cc014030 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.008+0000 7f84d14a4640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f84cc014030 0x7f84cc013890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f84cc014030 0x7f84cc013890 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c800e5a0 con 0x7f84cc014030 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c800e5a0 con 0x55a583fd4870 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fdddc0 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x7f84cc014030 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x7f84cc014030 2024-04-27T03:18:32.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.012+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x7f84cc014030 2024-04-27T03:18:32.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f84c8029040 con 0x7f84cc014030 2024-04-27T03:18:32.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fdddc0 2024-04-27T03:18:32.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fdddc0 2024-04-27T03:18:32.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.016+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fdddc0 2024-04-27T03:18:32.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x7f84cc014030 2024-04-27T03:18:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x7f84cc014030 2024-04-27T03:18:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd4870 2024-04-27T03:18:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x7f84cc014030 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x7f84cc014030 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fdddc0 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x7f84cc014030 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c8029040 con 0x7f84cc014030 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fdddc0 2024-04-27T03:18:32.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.020+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe0830 con 0x55a583fd8d60 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fdca00 con 0x55a583fd4870 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f84c8029040 con 0x55a583fd4870 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fdc640 con 0x55a583fd8d60 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fdd360 con 0x55a583fd4870 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 zone default found 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 4 Realm: () 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 zonegroup default 2024-04-27T03:18:32.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fdced0 con 0x7f84cc014030 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe2ba0 con 0x55a583fdddc0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe2f70 con 0x55a583fdddc0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 started sync module instance, tier type = 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:32.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.024+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe2040 con 0x55a583fd8d60 2024-04-27T03:18:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.028+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.028+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94169372435504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe3880 con 0x55a583fd8d60 2024-04-27T03:18:32.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.028+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe40d0 con 0x55a583fd4870 2024-04-27T03:18:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.028+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.028+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94169372440704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe4190 con 0x55a583fd4870 2024-04-27T03:18:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.028+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe4cb0 con 0x55a583fd8d60 2024-04-27T03:18:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.028+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94169372435504] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94169372440704] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94169372444544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe4c70 con 0x55a583fd8d60 2024-04-27T03:18:32.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe5550 con 0x7f84cc014030 2024-04-27T03:18:32.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94169372446880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe4c70 con 0x7f84cc014030 2024-04-27T03:18:32.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe5fe0 con 0x55a583fd8d60 2024-04-27T03:18:32.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.032+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94169372446880] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94169372444544] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94169372439504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe5fe0 con 0x55a583fd8d60 2024-04-27T03:18:32.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe63d0 con 0x55a583fdddc0 2024-04-27T03:18:32.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94169372447936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe6c20 con 0x55a583fdddc0 2024-04-27T03:18:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe7620 con 0x55a583fdddc0 2024-04-27T03:18:32.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94169372439504] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94169372447936] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94169372452720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe7130 con 0x55a583fdddc0 2024-04-27T03:18:32.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.036+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe8030 con 0x7f84cc014030 2024-04-27T03:18:32.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'177 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94169372457856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe8450 con 0x7f84cc014030 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=0 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=1 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=2 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=3 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=4 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=5 2024-04-27T03:18:32.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94169372452720] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=6 2024-04-27T03:18:32.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94169372457856] v27'178 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 20 add_watcher() i=7 2024-04-27T03:18:32.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:32.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe8450 con 0x55a583fd4870 2024-04-27T03:18:32.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.040+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94169372458912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583fe83c0 con 0x55a583fd4870 2024-04-27T03:18:32.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94169372458912] v27'188 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f83837fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f84d418a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:32.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f84d418a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:32.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff3e70 con 0x55a583fd8d60 2024-04-27T03:18:32.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'54 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff3f10 con 0x7f84cc014030 2024-04-27T03:18:32.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.044+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'31 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.048+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff4580 con 0x55a583fdddc0 2024-04-27T03:18:32.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.048+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'31 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.048+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff4980 con 0x55a583fd4870 2024-04-27T03:18:32.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.052+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'189 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.052+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff4ef0 con 0x55a583fd8d60 2024-04-27T03:18:32.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.052+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'70 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.052+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff4ef0 con 0x55a583fdddc0 2024-04-27T03:18:32.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.052+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'53 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.052+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff52f0 con 0x55a583fd8d60 2024-04-27T03:18:32.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.056+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'34 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.056+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff52f0 con 0x55a583fd4870 2024-04-27T03:18:32.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.056+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'48 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.056+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.060+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'24 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.060+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.064+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'30 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.064+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.064+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'71 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.064+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fdddc0 2024-04-27T03:18:32.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.068+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'27 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.068+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.068+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'30 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.068+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd4870 2024-04-27T03:18:32.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.072+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'59 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.072+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd4870 2024-04-27T03:18:32.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.072+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'190 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.072+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.076+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'62 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.076+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.076+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'72 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.076+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fdddc0 2024-04-27T03:18:32.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.080+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'37 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.080+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.080+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'32 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.080+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd4870 2024-04-27T03:18:32.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.084+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'191 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.084+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.084+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'63 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.084+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.088+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'40 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.088+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fdddc0 2024-04-27T03:18:32.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.088+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'32 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.088+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.092+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'42 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.092+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fdddc0 2024-04-27T03:18:32.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.092+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'26 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.092+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fdddc0 2024-04-27T03:18:32.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.096+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'54 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.096+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fdddc0 2024-04-27T03:18:32.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.096+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'32 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.096+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd8d60 2024-04-27T03:18:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.100+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'36 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.100+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff57a0 con 0x55a583fd4870 2024-04-27T03:18:32.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.104+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'49 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.104+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff5e00 con 0x55a583fd8d60 2024-04-27T03:18:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.104+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'64 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.104+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff5e00 con 0x55a583fd4870 2024-04-27T03:18:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'60 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff5e00 con 0x7f84cc014030 2024-04-27T03:18:32.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'30 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f83827fc640 20 reqs_thread_entry: start 2024-04-27T03:18:32.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:32.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 20 Filter name: none 2024-04-27T03:18:32.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f82f3fff640 20 reqs_thread_entry: start 2024-04-27T03:18:32.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:18:32.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584010e90 con 0x7f84cc014030 2024-04-27T03:18:32.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(94 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv33 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:18:32.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:18:32.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:18:32.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 : miss 2024-04-27T03:18:32.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.108+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 8.14 8:2ed3d538:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.5:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584012840 con 0x55a583fd8d60 2024-04-27T03:18:32.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(95 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv1 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:18:32.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 info.flags=0x17 2024-04-27T03:18:32.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 to cache LRU end 2024-04-27T03:18:32.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:32.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:18:32.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 2024-04-27T03:18:32.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 4.20 4:07e5129a:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff20 con 0x55a583fd8d60 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(96 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:97 4.3a 4:5d49dbd8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff60 con 0x55a583fdddc0 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(97 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:98 4.4 4:23b46250:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff60 con 0x55a583fd4870 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(98 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2 [call out=426b] v0'0 uv3 ondisk = 0) ==== 194+0+426 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 10 cls_bucket_list_unordered: got 9c8454ddf7aa50116496bac348d7550d 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:99 4.2e 4:7755dc4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff60 con 0x55a583fdddc0 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(99 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3 [call out=85b] v0'0 uv9 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.112+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 4.25 4:a5b599e6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff60 con 0x55a583fd8d60 2024-04-27T03:18:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(100 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 4.23 4:c71d5ee3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff60 con 0x55a583fdddc0 2024-04-27T03:18:32.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(101 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 4.21 4:8648ba88:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff20 con 0x55a583fdddc0 2024-04-27T03:18:32.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(102 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6 [call out=425b] v0'0 uv3 ondisk = 0) ==== 194+0+425 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 10 cls_bucket_list_unordered: got b1b9a972ccd8c962a473909b97007eb4 2024-04-27T03:18:32.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 4.24 4:253d3e93:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408ff20 con 0x55a583fd4870 2024-04-27T03:18:32.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(103 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:104 4.3e 4:7e2ce779:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840906a0 con 0x55a583fd8d60 2024-04-27T03:18:32.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(104 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8 [call out=425b] v0'0 uv3 ondisk = 0) ==== 194+0+425 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 10 cls_bucket_list_unordered: got c785e1ed2950e3e36b1e2ca01f299a54 2024-04-27T03:18:32.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:105 4.3d 4:bd8f1db3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840906a0 con 0x55a583fd8d60 2024-04-27T03:18:32.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(105 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.116+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:106 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840906a0 con 0x55a583fd4870 2024-04-27T03:18:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(106 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [call out=698b] v0'0 uv9 ondisk = 0) ==== 195+0+698 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 10 cls_bucket_list_unordered: got 24d27c169c2c881eb09a065116f2aa5c 2024-04-27T03:18:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 10 cls_bucket_list_unordered: got bb3aec0fdcdbc2974890f805c585d432 2024-04-27T03:18:32.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:9c8454ddf7aa50116496bac348d7550d state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:107 3.34s0 3:2d288bd5:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_9c8454ddf7aa50116496bac348d7550d:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408daa0 con 0x55a583fd8d60 2024-04-27T03:18:32.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(107 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_9c8454ddf7aa50116496bac348d7550d [getxattrs out=879b,stat out=16b] v0'0 uv3 ondisk = 0) ==== 262+0+895 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 10 manifest: total_size = 6 2024-04-27T03:18:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11168846131453053148 2024-04-27T03:18:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:9c8454ddf7aa50116496bac348d7550d state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 20 prepare_atomic_modification: state is not atomic. state=0x55a5840130c8 2024-04-27T03:18:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2 2024-04-27T03:18:32.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:108 4.4 4:23b46250:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584017a80 con 0x55a583fd4870 2024-04-27T03:18:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(108 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2 [stat,call,call] v27'4 uv4 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.120+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:109 3.34s0 3:2d288bd5:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_9c8454ddf7aa50116496bac348d7550d:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584015a60 con 0x55a583fd8d60 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.124+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(109 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_9c8454ddf7aa50116496bac348d7550d [call] v27'4 uv4 ondisk = 0) ==== 220+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.124+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:110 4.4 4:23b46250:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584018510 con 0x55a583fd4870 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.124+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:b1b9a972ccd8c962a473909b97007eb4 state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.124+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:111 3.14s0 3:296849e6:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_b1b9a972ccd8c962a473909b97007eb4:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584019120 con 0x55a583fd8d60 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(111 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_b1b9a972ccd8c962a473909b97007eb4 [getxattrs out=877b,stat out=16b] v0'0 uv1 ondisk = 0) ==== 262+0+893 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d418a980 10 manifest: total_size = 3 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d418a980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.4018145399102900065 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:b1b9a972ccd8c962a473909b97007eb4 state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d418a980 20 prepare_atomic_modification: state is not atomic. state=0x55a5840130c8 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d418a980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6 2024-04-27T03:18:32.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:112 4.21 4:8648ba88:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584018510 con 0x55a583fdddc0 2024-04-27T03:18:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(110 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2 [stat,call,call] v27'5 uv5 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f83b57fa640 20 handle_completion(): completion ok for obj=9c8454ddf7aa50116496bac348d7550d 2024-04-27T03:18:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(112 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6 [stat,call,call] v27'4 uv4 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.128+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:113 3.14s0 3:296849e6:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_b1b9a972ccd8c962a473909b97007eb4:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584018060 con 0x55a583fd8d60 2024-04-27T03:18:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(113 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_b1b9a972ccd8c962a473909b97007eb4 [call] v27'2 uv2 ondisk = 0) ==== 220+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:114 4.21 4:8648ba88:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584018510 con 0x55a583fdddc0 2024-04-27T03:18:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:c785e1ed2950e3e36b1e2ca01f299a54 state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:115 3.35s0 3:af6bcc10:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_c785e1ed2950e3e36b1e2ca01f299a54:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584018510 con 0x55a583fd8d60 2024-04-27T03:18:32.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(115 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_c785e1ed2950e3e36b1e2ca01f299a54 [getxattrs out=877b,stat out=16b] v0'0 uv1 ondisk = 0) ==== 262+0+893 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 10 manifest: total_size = 4 2024-04-27T03:18:32.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.8068974608337234949 2024-04-27T03:18:32.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:c785e1ed2950e3e36b1e2ca01f299a54 state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 20 prepare_atomic_modification: state is not atomic. state=0x55a5840130c8 2024-04-27T03:18:32.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8 2024-04-27T03:18:32.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:116 4.3e 4:7e2ce779:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58401dde0 con 0x55a583fd8d60 2024-04-27T03:18:32.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(114 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6 [stat,call,call] v27'5 uv5 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f83b4ff9640 20 handle_completion(): completion ok for obj=b1b9a972ccd8c962a473909b97007eb4 2024-04-27T03:18:32.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(116 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8 [stat,call,call] v27'4 uv4 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.132+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:117 3.35s0 3:af6bcc10:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_c785e1ed2950e3e36b1e2ca01f299a54:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584015a60 con 0x55a583fd8d60 2024-04-27T03:18:32.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== osd_op_reply(117 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_c785e1ed2950e3e36b1e2ca01f299a54 [call] v27'2 uv2 ondisk = 0) ==== 220+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:118 4.3e 4:7e2ce779:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584019da0 con 0x55a583fd8d60 2024-04-27T03:18:32.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:24d27c169c2c881eb09a065116f2aa5c state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:119 3.1bs0 3:d9542cec:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_24d27c169c2c881eb09a065116f2aa5c:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584019da0 con 0x55a583fdddc0 2024-04-27T03:18:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(119 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_24d27c169c2c881eb09a065116f2aa5c [getxattrs out=879b,stat out=16b] v0'0 uv3 ondisk = 0) ==== 262+0+895 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 10 manifest: total_size = 5 2024-04-27T03:18:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.16155580919102344950 2024-04-27T03:18:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:24d27c169c2c881eb09a065116f2aa5c state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 20 prepare_atomic_modification: state is not atomic. state=0x55a5840130c8 2024-04-27T03:18:32.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 2024-04-27T03:18:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:120 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58401d9a0 con 0x55a583fd4870 2024-04-27T03:18:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 45 ==== osd_op_reply(118 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8 [stat,call,call] v27'5 uv5 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.136+0000 7f83b57fa640 20 handle_completion(): completion ok for obj=c785e1ed2950e3e36b1e2ca01f299a54 2024-04-27T03:18:32.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(120 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [stat,call,call] v27'10 uv10 ondisk = 0) ==== 279+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:121 3.1bs0 3:d9542cec:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_24d27c169c2c881eb09a065116f2aa5c:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58401c4e0 con 0x55a583fdddc0 2024-04-27T03:18:32.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(121 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_24d27c169c2c881eb09a065116f2aa5c [call] v27'4 uv4 ondisk = 0) ==== 220+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:122 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58401d9a0 con 0x55a583fd4870 2024-04-27T03:18:32.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:123 3.1cs0 3:3b792c84:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584014f00 con 0x55a583fd4870 2024-04-27T03:18:32.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(123 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432 [getxattrs out=910b,stat out=16b] v0'0 uv3 ondisk = 0) ==== 262+0+926 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 10 manifest: total_size = 5 2024-04-27T03:18:32.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 20 get_obj_state: setting s->obj_tag to _dskPOg4xx-r9FrhG4hncHE-fznh_Fxj 2024-04-27T03:18:32.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 20 get_obj_state: octx=0x7f84c800e520 obj=myfoo:bb3aec0fdcdbc2974890f805c585d432 state=0x55a5840130c8 s->prefetch_data=0 2024-04-27T03:18:32.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 20 prepare_atomic_modification: state is not atomic. state=0x55a5840130c8 2024-04-27T03:18:32.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 2024-04-27T03:18:32.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.140+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:124 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58401d9a0 con 0x55a583fd4870 2024-04-27T03:18:32.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.144+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(122 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [stat,call,call] v27'11 uv11 ondisk = 0) ==== 279+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.144+0000 7f83b4ff9640 20 handle_completion(): completion ok for obj=24d27c169c2c881eb09a065116f2aa5c 2024-04-27T03:18:32.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.144+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(124 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [stat,call,call] v27'12 uv12 ondisk = 0) ==== 279+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.144+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:125 3.1cs0 3:3b792c84:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58401c990 con 0x55a583fd4870 2024-04-27T03:18:32.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.144+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(125 27a51273-3ccc-49e2-8682-1122abc3d898.4220.5_bb3aec0fdcdbc2974890f805c585d432 [call] v27'4 uv4 ondisk = 0) ==== 220+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.144+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:126 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58401d8c0 con 0x55a583fd4870 2024-04-27T03:18:32.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 20 list_objects_ordered: starting attempt 1 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:127 4.20 4:07e5129a:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408dd90 con 0x55a583fd8d60 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:128 4.3a 4:5d49dbd8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58408dd90 con 0x55a583fdddc0 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:129 4.4 4:23b46250:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584012ee0 con 0x55a583fd4870 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:130 4.2e 4:7755dc4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840243c0 con 0x55a583fdddc0 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:131 4.25 4:a5b599e6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584012ee0 con 0x55a583fd8d60 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:132 4.23 4:c71d5ee3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584012ee0 con 0x55a583fdddc0 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:133 4.21 4:8648ba88:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bbc0 con 0x55a583fdddc0 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:134 4.24 4:253d3e93:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58402ea40 con 0x55a583fd4870 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:135 4.3e 4:7e2ce779:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc00 con 0x55a583fd8d60 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:136 4.3d 4:bd8f1db3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840339b0 con 0x55a583fd8d60 2024-04-27T03:18:32.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:137 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584036390 con 0x55a583fd4870 2024-04-27T03:18:32.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 46 ==== osd_op_reply(127 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(129 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 47 ==== osd_op_reply(131 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(134 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(128 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(132 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(130 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3 [call out=85b] v0'0 uv9 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== osd_op_reply(133 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 48 ==== osd_op_reply(136 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(126 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [stat,call,call] v27'13 uv13 ondisk = 0) ==== 279+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 49 ==== osd_op_reply(135 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f83b57fa640 20 handle_completion(): completion ok for obj=bb3aec0fdcdbc2974890f805c585d432 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(137 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [call out=124b] v0'0 uv13 ondisk = 0) ==== 195+0+124 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.5])"; uploads.size()=0; is_truncated=0 2024-04-27T03:18:32.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:138 4.20 4:07e5129a:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840339b0 con 0x55a583fd8d60 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:139 4.3a 4:5d49dbd8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584036390 con 0x55a583fdddc0 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:140 4.4 4:23b46250:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584013370 con 0x55a583fd4870 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:141 4.2e 4:7755dc4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584036390 con 0x55a583fdddc0 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:142 4.25 4:a5b599e6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584036390 con 0x55a583fd8d60 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:143 4.23 4:c71d5ee3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840339b0 con 0x55a583fdddc0 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:144 4.21 4:8648ba88:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584036390 con 0x55a583fdddc0 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:145 4.24 4:253d3e93:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840339b0 con 0x55a583fd4870 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 42 ==== osd_op_reply(140 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:146 4.3e 4:7e2ce779:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584036390 con 0x55a583fd8d60 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:147 4.3d 4:bd8f1db3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a5840339b0 con 0x55a583fd8d60 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(139 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:148 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584036390 con 0x55a583fd4870 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(141 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3 [call out=85b] v0'0 uv9 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 50 ==== osd_op_reply(142 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 43 ==== osd_op_reply(145 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== osd_op_reply(143 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(144 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.148+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 51 ==== osd_op_reply(138 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 44 ==== osd_op_reply(148 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [call out=124b] v0'0 uv13 ondisk = 0) ==== 195+0+124 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 52 ==== osd_op_reply(146 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8 [call out=124b] v0'0 uv5 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 53 ==== osd_op_reply(147 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:149 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x7f84cc014030 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(149 foo.buckets [call] v27'37 uv37 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d418a980 10 cache get: name=default.rgw.meta+root+myfoo : hit (requested=0x11, cached=0x11) 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:150 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x7f84cc014030 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(150 myfoo [call,call,delete] v27'38 uv38 ondisk = 0) ==== 233+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d418a980 10 removing default.rgw.meta+root+myfoo from cache 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d418a980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:151 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94169372736608 in=161b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a584013130 con 0x55a583fdddc0 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== watch-notify(notify (1) cookie 94169372452720 notify 115964117010 ret 0) ==== 191+0+0 (crc 0 0 0) 0x55a583ff3860 con 0x55a583fdddc0 2024-04-27T03:18:32.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f83b57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117010 cookie 94169372452720 notifier 4639 bl.length()=149 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== osd_op_reply(151 notify.6 [notify cookie 94169372736608 out=8b] v0'0 uv177 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.152+0000 7f83b57fa640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:152 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f8394003d70 con 0x55a583fdddc0 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== osd_op_reply(152 notify.6 [notify-ack] v0'0 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== watch-notify(notify_complete (2) cookie 94169372736608 notify 115964117010 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a583ff3ed0 con 0x55a583fdddc0 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:153 8.14 8:2ed3d538:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.5:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd8d60 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 54 ==== osd_op_reply(153 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 [call,call,delete] v27'2 uv2 ondisk = 0) ==== 290+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d418a980 10 removing default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 from cache 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d418a980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.5, ofs0, ns] 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:154 7.15 7:a93a5511:::notify.2:head [notify cookie 94169372678336 in=218b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd8d60 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 55 ==== watch-notify(notify (1) cookie 94169372444544 notify 115964117029 ret 0) ==== 248+0+0 (crc 0 0 0) 0x55a583ff30e0 con 0x55a583fd8d60 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f83b4ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117029 cookie 94169372444544 notifier 4639 bl.length()=206 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f83b4ff9640 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:155 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f82fc003330 con 0x55a583fd8d60 2024-04-27T03:18:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.156+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 56 ==== osd_op_reply(154 notify.2 [notify cookie 94169372678336 out=8b] v0'0 uv177 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 57 ==== osd_op_reply(155 notify.2 [notify-ack] v0'0 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 58 ==== watch-notify(notify_complete (2) cookie 94169372678336 notify 115964117029 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a583fc99e0 con 0x55a583fd8d60 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:156 4.20 4:07e5129a:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd8d60 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:157 4.3a 4:5d49dbd8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff1f10 con 0x55a583fdddc0 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:158 4.4 4:23b46250:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff2760 con 0x55a583fd4870 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:159 4.2e 4:7755dc4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff2b00 con 0x55a583fdddc0 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:160 4.25 4:a5b599e6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584029b60 con 0x55a583fd8d60 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:161 4.23 4:c71d5ee3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fdddc0 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:162 4.21 4:8648ba88:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584011c30 con 0x55a583fdddc0 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:163 4.24 4:253d3e93:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402a380 con 0x55a583fd4870 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:164 4.3e 4:7e2ce779:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a584029b60 con 0x55a583fd8d60 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:165 4.3d 4:bd8f1db3:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a583ff1f10 con 0x55a583fd8d60 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:166 4.2 4:409912db:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402b570 con 0x55a583fd4870 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== osd_op_reply(162 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.6 [delete] v27'6 uv6 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== osd_op_reply(159 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.3 [delete] v27'10 uv10 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 59 ==== osd_op_reply(160 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.4 [delete] v27'4 uv4 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.160+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 60 ==== osd_op_reply(156 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.0 [delete] v27'4 uv4 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 61 ==== osd_op_reply(164 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.8 [delete] v27'6 uv6 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 62 ==== osd_op_reply(165 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.9 [delete] v27'10 uv10 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 45 ==== osd_op_reply(163 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.7 [delete] v27'6 uv6 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 46 ==== osd_op_reply(158 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.2 [delete] v27'6 uv6 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 47 ==== osd_op_reply(166 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.10 [delete] v27'14 uv14 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(157 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.1 [delete] v27'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== osd_op_reply(161 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.5.5 [delete] v27'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 10 cache get: name=default.rgw.log++pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 : miss 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:167 6.1f 6:fa7bba35:::pubsub..bucket.myfoo%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.5:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fdddc0 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 46 ==== osd_op_reply(167 pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 208+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 10 cache put: name=default.rgw.log++pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 info.flags=0x0 2024-04-27T03:18:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 10 adding default.rgw.log++pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 to cache LRU end 2024-04-27T03:18:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:168 6.1f 6:fa7bba35:::pubsub..bucket.myfoo%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fdddc0 2024-04-27T03:18:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 47 ==== osd_op_reply(168 pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.5 [delete] v27'33 uv29 ondisk = -2 ((2) No such file or directory)) ==== 208+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.164+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:169 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x7f84cc014030 2024-04-27T03:18:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.168+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(169 foo.buckets [call] v27'39 uv39 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.168+0000 7f83837fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.168+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:170 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94169372458912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd4870 2024-04-27T03:18:32.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.172+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 48 ==== osd_op_reply(170 data_loggenerations_metadata [watch unwatch cookie 94169372458912] v27'192 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.172+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:171 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94169372435504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd8d60 2024-04-27T03:18:32.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.172+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 63 ==== osd_op_reply(171 notify.0 [watch unwatch cookie 94169372435504] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.172+0000 7f84d418a980 20 remove_watcher() i=0 2024-04-27T03:18:32.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.172+0000 7f84d418a980 2 removed watcher, disabling cache 2024-04-27T03:18:32.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.172+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:172 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94169372440704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd4870 2024-04-27T03:18:32.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.176+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.1 v2:172.21.15.110:6800/1860462708 49 ==== osd_op_reply(172 notify.1 [watch unwatch cookie 94169372440704] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x55a583fd4870 2024-04-27T03:18:32.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.176+0000 7f84d418a980 20 remove_watcher() i=1 2024-04-27T03:18:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.176+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:173 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94169372444544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd8d60 2024-04-27T03:18:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.176+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 64 ==== osd_op_reply(173 notify.2 [watch unwatch cookie 94169372444544] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.176+0000 7f84d418a980 20 remove_watcher() i=2 2024-04-27T03:18:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.176+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:174 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94169372446880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x7f84cc014030 2024-04-27T03:18:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.180+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(174 notify.3 [watch unwatch cookie 94169372446880] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.180+0000 7f84d418a980 20 remove_watcher() i=3 2024-04-27T03:18:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.180+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:175 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94169372439504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fd8d60 2024-04-27T03:18:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.180+0000 7f84d24a6640 1 -- 172.21.15.110:0/1567361234 <== osd.0 v2:172.21.15.77:6800/3496324319 65 ==== osd_op_reply(175 notify.4 [watch unwatch cookie 94169372439504] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84cc05b040 con 0x55a583fd8d60 2024-04-27T03:18:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.180+0000 7f84d418a980 20 remove_watcher() i=4 2024-04-27T03:18:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.180+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:176 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94169372447936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fdddc0 2024-04-27T03:18:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 48 ==== osd_op_reply(176 notify.5 [watch unwatch cookie 94169372447936] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 20 remove_watcher() i=5 2024-04-27T03:18:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:177 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94169372452720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x55a583fdddc0 2024-04-27T03:18:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d1ca5640 1 -- 172.21.15.110:0/1567361234 <== osd.3 v2:172.21.15.110:6816/3916647200 49 ==== osd_op_reply(177 notify.6 [watch unwatch cookie 94169372452720] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c40b3040 con 0x55a583fdddc0 2024-04-27T03:18:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 20 remove_watcher() i=6 2024-04-27T03:18:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:178 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94169372457856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55a58402bc60 con 0x7f84cc014030 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(178 notify.7 [watch unwatch cookie 94169372457856] v27'179 uv177 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f84c803c1c0 con 0x7f84cc014030 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 20 remove_watcher() i=7 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583fd8d60 msgr2=0x55a583fdb150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583fd8d60 0x55a583fdb150 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583fd4870 msgr2=0x55a583fd6d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583fd4870 0x55a583fd6d30 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f84cc014030 msgr2=0x7f84cc013890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f84cc014030 0x7f84cc013890 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a583fdddc0 msgr2=0x55a583fe01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a583fdddc0 0x55a583fe01b0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f839c034600 msgr2=0x7f839c036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f839c034600 0x7f839c036ac0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f84c4006b40 tx=0x7f84c40dd000 comp rx=0 tx=0).stop 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d14a4640 1 -- 172.21.15.110:0/1567361234 reap_dead start 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 msgr2=0x55a583fc9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.184+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc9210 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f84cc01e4b0 tx=0x7f84cc04a5d0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 shutdown_connections 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/1567361234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583fc1d80 0x55a583fc9210 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 >> 172.21.15.110:0/1567361234 conn(0x55a583fbe0c0 msgr2=0x55a583fbb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 shutdown_connections 2024-04-27T03:18:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/1567361234 wait complete. 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83ac034650 msgr2=0x7f83ac036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83ac034650 0x7f83ac036b10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x55a583fa2db0 tx=0x7f84c40e2be0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583f94c00 msgr2=0x55a583f930a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583f94c00 0x55a583f930a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00df50 tx=0x7f84cc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 shutdown_connections 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83ac034650 0x7f83ac036b10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3597462853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583f94c00 0x55a583f930a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 >> 172.21.15.110:0/3597462853 conn(0x55a583fa7780 msgr2=0x55a583fa8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 shutdown_connections 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3597462853 wait complete. 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583f9fd90 msgr2=0x55a583fa2180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583f9fd90 0x55a583fa2180 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583ea4a80 msgr2=0x55a583f9e980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583ea4a80 0x55a583f9e980 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83b8034650 msgr2=0x7f83b8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83b8034650 0x7f83b8036b10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f84c8000f70 tx=0x7f84c80012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 msgr2=0x55a583f97aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f97aa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f84c4018eb0 tx=0x7f84c4018020 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 shutdown_connections 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a583f9fd90 0x55a583fa2180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a583ea4a80 0x55a583f9e980 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f83b8034650 0x7f83b8036b10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 --2- 172.21.15.110:0/3910604231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a583ea4160 0x55a583f97aa0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 >> 172.21.15.110:0/3910604231 conn(0x55a583e9fa30 msgr2=0x55a583f8a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 shutdown_connections 2024-04-27T03:18:32.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.188+0000 7f84d418a980 1 -- 172.21.15.110:0/3910604231 wait complete. 2024-04-27T03:18:32.213 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'foo', '--caps', 'user=read'] 2024-04-27T03:18:32.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'foo', '--caps', 'user=read'] 2024-04-27T03:18:32.213 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid foo --caps user=read 2024-04-27T03:18:32.297 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:32.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.288+0000 7f5e199e1980 1 Processor -- start 2024-04-27T03:18:32.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.288+0000 7f5e199e1980 1 -- start start 2024-04-27T03:18:32.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.288+0000 7f5e199e1980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f51a972a90 0x55f51a974e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.288+0000 7f5e199e1980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55f51a82b0d0 con 0x55f51a8dffd0 2024-04-27T03:18:32.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e199e1980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55f51a820430 con 0x55f51a972a90 2024-04-27T03:18:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e16cfb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f51a972a90 0x55f51a974e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e16cfb640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f51a972a90 0x55f51a974e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41142/0 (socket says 172.21.15.110:41142) 2024-04-27T03:18:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e174fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55f51a8dffd0 0x55f51a971370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51268/0 (socket says 172.21.15.110:51268) 2024-04-27T03:18:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e16cfb640 1 -- 172.21.15.110:0/2627659398 learned_addr learned my addr 172.21.15.110:0/2627659398 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e16cfb640 1 -- 172.21.15.110:0/2627659398 >> v1:172.21.15.77:6789/0 conn(0x55f51a8dffd0 legacy=0x55f51a971370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e16cfb640 1 -- 172.21.15.110:0/2627659398 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51a819510 con 0x55f51a972a90 2024-04-27T03:18:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e16cfb640 1 --2- 172.21.15.110:0/2627659398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a972a90 0x55f51a974e80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00a5c0 tx=0x7f5e0c01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef94c7b963397002 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e164fa640 1 -- 172.21.15.110:0/2627659398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e0c0a6070 con 0x55f51a972a90 2024-04-27T03:18:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e164fa640 1 -- 172.21.15.110:0/2627659398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e0c0af070 con 0x55f51a972a90 2024-04-27T03:18:32.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.292+0000 7f5e164fa640 1 -- 172.21.15.110:0/2627659398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e0c0aa5c0 con 0x55f51a972a90 2024-04-27T03:18:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.296+0000 7f5e199e1980 1 -- 172.21.15.110:0/2627659398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a972a90 msgr2=0x55f51a974e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.296+0000 7f5e199e1980 1 --2- 172.21.15.110:0/2627659398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a972a90 0x55f51a974e80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00a5c0 tx=0x7f5e0c01b340 comp rx=0 tx=0).stop 2024-04-27T03:18:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.296+0000 7f5e199e1980 1 -- 172.21.15.110:0/2627659398 shutdown_connections 2024-04-27T03:18:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.296+0000 7f5e199e1980 1 --2- 172.21.15.110:0/2627659398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a972a90 0x55f51a974e80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.296+0000 7f5e199e1980 1 -- 172.21.15.110:0/2627659398 >> 172.21.15.110:0/2627659398 conn(0x55f51a966a50 msgr2=0x55f51a968e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.296+0000 7f5e199e1980 1 -- 172.21.15.110:0/2627659398 shutdown_connections 2024-04-27T03:18:32.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.296+0000 7f5e199e1980 1 -- 172.21.15.110:0/2627659398 wait complete. 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 Processor -- start 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- start start 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa59620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f51a86f2d0 con 0x55f51a8dffd0 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa59620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa59620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41146/0 (socket says 172.21.15.110:41146) 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 -- 172.21.15.110:0/3292922241 learned_addr learned my addr 172.21.15.110:0/3292922241 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 -- 172.21.15.110:0/3292922241 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51a7f5ae0 con 0x55f51a8dffd0 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 --2- 172.21.15.110:0/3292922241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa59620 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5e080015f0 tx=0x7f5e08001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b754e4fb6fc536a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5d0560a640 1 -- 172.21.15.110:0/3292922241 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e08013420 con 0x55f51a8dffd0 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5d0560a640 1 -- 172.21.15.110:0/3292922241 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e080135e0 con 0x55f51a8dffd0 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5d0560a640 1 -- 172.21.15.110:0/3292922241 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e08013940 con 0x55f51a8dffd0 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- 172.21.15.110:0/3292922241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 msgr2=0x55f51aa59620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3292922241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa59620 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f5e080015f0 tx=0x7f5e08001ab0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- 172.21.15.110:0/3292922241 shutdown_connections 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3292922241 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa59620 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- 172.21.15.110:0/3292922241 >> 172.21.15.110:0/3292922241 conn(0x55f51a966a50 msgr2=0x55f51aa551b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- 172.21.15.110:0/3292922241 shutdown_connections 2024-04-27T03:18:32.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- 172.21.15.110:0/3292922241 wait complete. 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 Processor -- start 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- start start 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa60660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e199e1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f51a977e20 con 0x55f51a8dffd0 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa60660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa60660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41154/0 (socket says 172.21.15.110:41154) 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.312+0000 7f5e174fc640 1 -- 172.21.15.110:0/634627265 learned_addr learned my addr 172.21.15.110:0/634627265 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e174fc640 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51aa60ba0 con 0x55f51a8dffd0 2024-04-27T03:18:32.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e174fc640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa60660 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5e08006640 tx=0x7f5e08000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5d03e07640 1 -- 172.21.15.110:0/634627265 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e08022020 con 0x55f51a8dffd0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f51aa60e30 con 0x55f51a8dffd0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5d03e07640 1 -- 172.21.15.110:0/634627265 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e08006320 con 0x55f51a8dffd0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5d03e07640 1 -- 172.21.15.110:0/634627265 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e08004ab0 con 0x55f51a8dffd0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f51aa63ab0 con 0x55f51a8dffd0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5d03e07640 1 -- 172.21.15.110:0/634627265 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5e08025020 con 0x55f51a8dffd0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5d03e07640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cfc034650 0x7f5cfc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5d03e07640 1 -- 172.21.15.110:0/634627265 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5e08018030 con 0x55f51a8dffd0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e16cfb640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cfc034650 0x7f5cfc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51a972a90 0x55f51aa67590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa67be0 con 0x55f51a972a90 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51a972a90 0x55f51aa67590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e16cfb640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cfc034650 0x7f5cfc036b10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c0ae000 tx=0x7f5e0c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51a972a90 0x55f51aa67590 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e17cfd640 1 -- 172.21.15.110:0/634627265 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5e100083f0 con 0x55f51a972a90 2024-04-27T03:18:32.327 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa689d0 0x55f51aa6adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa6b3f0 con 0x55f51aa689d0 2024-04-27T03:18:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.316+0000 7f5e174fc640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa689d0 0x55f51aa6adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e174fc640 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa689d0 0x55f51aa6adc0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e174fc640 1 -- 172.21.15.110:0/634627265 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aa689d0 2024-04-27T03:18:32.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa6cca0 con 0x55f51a972a90 2024-04-27T03:18:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e17cfd640 1 -- 172.21.15.110:0/634627265 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f5e100083f0 con 0x55f51a972a90 2024-04-27T03:18:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa6bed0 con 0x55f51aa689d0 2024-04-27T03:18:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e174fc640 1 -- 172.21.15.110:0/634627265 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aa689d0 2024-04-27T03:18:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa6c2c0 con 0x55f51a972a90 2024-04-27T03:18:32.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e17cfd640 1 -- 172.21.15.110:0/634627265 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f5e100083f0 con 0x55f51a972a90 2024-04-27T03:18:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e199e1980 1 Processor -- start 2024-04-27T03:18:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e199e1980 1 -- start start 2024-04-27T03:18:32.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e199e1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa7b5c0 0x55f51aa7b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e199e1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f51a873310 con 0x55f51aa7b5c0 2024-04-27T03:18:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e17cfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa7b5c0 0x55f51aa7b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e17cfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa7b5c0 0x55f51aa7b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41168/0 (socket says 172.21.15.110:41168) 2024-04-27T03:18:32.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.320+0000 7f5e17cfd640 1 -- 172.21.15.110:0/999875313 learned_addr learned my addr 172.21.15.110:0/999875313 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e17cfd640 1 -- 172.21.15.110:0/999875313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51aa7bf20 con 0x55f51aa7b5c0 2024-04-27T03:18:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa7b5c0 0x55f51aa7b9e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000aab0 tx=0x7f5e1000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5d01602640 1 -- 172.21.15.110:0/999875313 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e10007d30 con 0x55f51aa7b5c0 2024-04-27T03:18:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5d01602640 1 -- 172.21.15.110:0/999875313 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e1001d070 con 0x55f51aa7b5c0 2024-04-27T03:18:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f51aa7c9c0 con 0x55f51aa7b5c0 2024-04-27T03:18:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5d01602640 1 -- 172.21.15.110:0/999875313 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e10018440 con 0x55f51aa7b5c0 2024-04-27T03:18:32.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f51aa7ce90 con 0x55f51aa7b5c0 2024-04-27T03:18:32.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5d01602640 1 -- 172.21.15.110:0/999875313 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5e10013070 con 0x55f51aa7b5c0 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5d01602640 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cf00346a0 0x7f5cf0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5d01602640 1 -- 172.21.15.110:0/999875313 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5e1004b5e0 con 0x55f51aa7b5c0 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e174fc640 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cf00346a0 0x7f5cf0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e199e1980 1 Processor -- start 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e199e1980 1 -- start start 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e174fc640 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cf00346a0 0x7f5cf0036b60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55f51aa6b9f0 tx=0x7f5e08048350 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e199e1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa88b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e199e1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f51aa59d70 con 0x55f51aa88760 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e17cfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa88b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e17cfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa88b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41174/0 (socket says 172.21.15.110:41174) 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.324+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3222498427 learned_addr learned my addr 172.21.15.110:0/3222498427 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3222498427 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51aa89150 con 0x55f51aa88760 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/3222498427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa88b80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5e10049930 tx=0x7f5e10018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa36c021582880fa server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5cfaffd640 1 -- 172.21.15.110:0/3222498427 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e10047070 con 0x55f51aa88760 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5cfaffd640 1 -- 172.21.15.110:0/3222498427 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e10018bc0 con 0x55f51aa88760 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5cfaffd640 1 -- 172.21.15.110:0/3222498427 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e10020360 con 0x55f51aa88760 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- 172.21.15.110:0/3222498427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 msgr2=0x55f51aa88b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3222498427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa88b80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5e10049930 tx=0x7f5e10018950 comp rx=0 tx=0).stop 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- 172.21.15.110:0/3222498427 shutdown_connections 2024-04-27T03:18:32.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3222498427 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa88b80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- 172.21.15.110:0/3222498427 >> 172.21.15.110:0/3222498427 conn(0x55f51aa84aa0 msgr2=0x55f51aa82b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- 172.21.15.110:0/3222498427 shutdown_connections 2024-04-27T03:18:32.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- 172.21.15.110:0/3222498427 wait complete. 2024-04-27T03:18:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 Processor -- start 2024-04-27T03:18:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- start start 2024-04-27T03:18:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa8fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f51aa898b0 con 0x55f51aa88760 2024-04-27T03:18:32.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e17cfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa8fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e17cfd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa8fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41186/0 (socket says 172.21.15.110:41186) 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 learned_addr learned my addr 172.21.15.110:0/3543926853 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51aa90130 con 0x55f51aa88760 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa8fbf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5e100168b0 tx=0x7f5e100498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5cf97fa640 1 -- 172.21.15.110:0/3543926853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e10031070 con 0x55f51aa88760 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5cf97fa640 1 -- 172.21.15.110:0/3543926853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e10020900 con 0x55f51aa88760 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f51aa903c0 con 0x55f51aa88760 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5cf97fa640 1 -- 172.21.15.110:0/3543926853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e10049a70 con 0x55f51aa88760 2024-04-27T03:18:32.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.328+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f51aa930a0 con 0x55f51aa88760 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5cf97fa640 1 -- 172.21.15.110:0/3543926853 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5e10034070 con 0x55f51aa88760 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5cf97fa640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ce0034650 0x7f5ce0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5cf97fa640 1 -- 172.21.15.110:0/3543926853 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f5e10037c90 con 0x55f51aa88760 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e174fc640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ce0034650 0x7f5ce0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51aa9b2a0 0x55f51aa9d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa9deb0 con 0x55f51aa9b2a0 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e16cfb640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51aa9b2a0 0x55f51aa9d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e174fc640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ce0034650 0x7f5ce0036b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5e08012fd0 tx=0x7f5e0803d000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e16cfb640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51aa9b2a0 0x55f51aa9d760 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 realm 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa9e2e0 con 0x55f51aa9b2a0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aa9ebb0 con 0x55f51aa9b2a0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa9f5d0 0x55f51aaa1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa2170 con 0x55f51aa9f5d0 2024-04-27T03:18:32.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa9f5d0 0x55f51aaa1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.332+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa9f5d0 0x55f51aaa1a00 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e199e1980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa29c0 con 0x55f51aa9b2a0 2024-04-27T03:18:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e199e1980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e199e1980 20 searching for the correct realm 2024-04-27T03:18:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f51aaa4730 0x55f51aaa6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aaa4730 2024-04-27T03:18:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e174fc640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f51aaa4730 0x55f51aaa6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e174fc640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f51aaa4730 0x55f51aaa6b20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e17cfd640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5e10036070 0x7f5e10017a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x7f5e10036070 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e16cfb640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5e10036070 0x7f5e10017a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.336+0000 7f5e16cfb640 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5e10036070 0x7f5e10017a40 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aaa4730 2024-04-27T03:18:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x7f5e10036070 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x7f5e10036070 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.340+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aaa4730 2024-04-27T03:18:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aaa4730 2024-04-27T03:18:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aaa4730 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x7f5e10036070 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9b2a0 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.344+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x7f5e10036070 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aaa4730 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x7f5e10036070 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aaa4730 2024-04-27T03:18:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.348+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:32.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa3370 con 0x55f51aa9b2a0 2024-04-27T03:18:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa3c10 con 0x55f51aa9f5d0 2024-04-27T03:18:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa2fb0 con 0x55f51aa9b2a0 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 zone default found 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 4 Realm: () 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 zonegroup default 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa7a80 con 0x7f5e10036070 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa9d30 con 0x55f51aaa4730 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaaa0d0 con 0x55f51aaa4730 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 started sync module instance, tier type = 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:32.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.352+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaa9170 con 0x55f51aa9f5d0 2024-04-27T03:18:32.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.356+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.356+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94511202735504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaaa920 con 0x55f51aa9f5d0 2024-04-27T03:18:32.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.356+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaab170 con 0x55f51aa9b2a0 2024-04-27T03:18:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.356+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.356+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94511202740512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaab270 con 0x55f51aa9b2a0 2024-04-27T03:18:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.356+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaabd70 con 0x55f51aa9f5d0 2024-04-27T03:18:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.356+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94511202735504] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94511202740512] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94511202741568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaaa470 con 0x55f51aa9f5d0 2024-04-27T03:18:32.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaac220 con 0x7f5e10036070 2024-04-27T03:18:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94511202745648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaac600 con 0x7f5e10036070 2024-04-27T03:18:32.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaad0f0 con 0x55f51aa9f5d0 2024-04-27T03:18:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94511202745648] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.360+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94511202741568] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94511202749376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaac150 con 0x55f51aa9f5d0 2024-04-27T03:18:32.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaad990 con 0x55f51aaa4730 2024-04-27T03:18:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94511202751712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaac150 con 0x55f51aaa4730 2024-04-27T03:18:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaae800 con 0x55f51aaa4730 2024-04-27T03:18:32.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94511202749376] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94511202751712] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94511202753840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaae350 con 0x55f51aaa4730 2024-04-27T03:18:32.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.364+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaaed10 con 0x7f5e10036070 2024-04-27T03:18:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94511202753840] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'180 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94511202744416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaaed10 con 0x7f5e10036070 2024-04-27T03:18:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=0 2024-04-27T03:18:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=1 2024-04-27T03:18:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=2 2024-04-27T03:18:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=3 2024-04-27T03:18:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=4 2024-04-27T03:18:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=5 2024-04-27T03:18:32.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=6 2024-04-27T03:18:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94511202744416] v27'181 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 20 add_watcher() i=7 2024-04-27T03:18:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aab2110 con 0x55f51aa9b2a0 2024-04-27T03:18:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.368+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94511202757168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aaaf630 con 0x55f51aa9b2a0 2024-04-27T03:18:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94511202757168] v27'193 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5ccdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 5 note: GC not initialized 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5cccfe1640 20 reqs_thread_entry: start 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 20 Filter name: none 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5c467fc640 20 reqs_thread_entry: start 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad5930 con 0x55f51aa9b2a0 2024-04-27T03:18:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv13 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:32.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~403 in=403b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aadbad0 con 0x55f51aa9b2a0 2024-04-27T03:18:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [delete,create,call,call,writefull 0~403] v27'14 uv14 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:32.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94511202940256 in=591b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad73b0 con 0x55f51aa9f5d0 2024-04-27T03:18:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94511202749376 notify 115964117030 ret 0) ==== 621+0+0 (crc 0 0 0) 0x55f51aa930a0 con 0x55f51aa9f5d0 2024-04-27T03:18:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5cfaffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117030 cookie 94511202749376 notifier 4646 bl.length()=579 2024-04-27T03:18:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5cfaffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5cfaffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5cfaffd640 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f5cdc003eb0 con 0x55f51aa9f5d0 2024-04-27T03:18:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.372+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94511202940256 out=8b] v0'0 uv180 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.376+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.376+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94511202940256 notify 115964117030 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f5e10063020 con 0x55f51aa9f5d0 2024-04-27T03:18:32.384 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[{"type":"user","perm":"read"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.376+0000 7f5ccdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.376+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94511202757168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x55f51aa9b2a0 2024-04-27T03:18:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.376+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94511202757168] v27'194 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.376+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94511202735504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x55f51aa9f5d0 2024-04-27T03:18:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.380+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94511202735504] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.380+0000 7f5e199e1980 20 remove_watcher() i=0 2024-04-27T03:18:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.380+0000 7f5e199e1980 2 removed watcher, disabling cache 2024-04-27T03:18:32.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.380+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94511202740512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x55f51aa9b2a0 2024-04-27T03:18:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.380+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94511202740512] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x55f51aa9b2a0 2024-04-27T03:18:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.380+0000 7f5e199e1980 20 remove_watcher() i=1 2024-04-27T03:18:32.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.380+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94511202741568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x55f51aa9f5d0 2024-04-27T03:18:32.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.384+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94511202741568] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.384+0000 7f5e199e1980 20 remove_watcher() i=2 2024-04-27T03:18:32.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.384+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94511202745648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x7f5e10036070 2024-04-27T03:18:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.384+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94511202745648] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.384+0000 7f5e199e1980 20 remove_watcher() i=3 2024-04-27T03:18:32.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.384+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94511202749376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x55f51aa9f5d0 2024-04-27T03:18:32.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e17cfd640 1 -- 172.21.15.110:0/3543926853 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94511202749376] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e10033030 con 0x55f51aa9f5d0 2024-04-27T03:18:32.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e199e1980 20 remove_watcher() i=4 2024-04-27T03:18:32.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94511202751712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x55f51aaa4730 2024-04-27T03:18:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94511202751712] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e199e1980 20 remove_watcher() i=5 2024-04-27T03:18:32.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94511202753840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x55f51aaa4730 2024-04-27T03:18:32.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e174fc640 1 -- 172.21.15.110:0/3543926853 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94511202753840] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0801b070 con 0x55f51aaa4730 2024-04-27T03:18:32.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e199e1980 20 remove_watcher() i=6 2024-04-27T03:18:32.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.388+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94511202744416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55f51aad3660 con 0x7f5e10036070 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94511202744416] v27'182 uv180 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5e0c0aaa20 con 0x7f5e10036070 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 20 remove_watcher() i=7 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa9f5d0 msgr2=0x55f51aaa1a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa9f5d0 0x55f51aaa1a00 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51aa9b2a0 msgr2=0x55f51aa9d760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51aa9b2a0 0x55f51aa9d760 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5e10036070 msgr2=0x7f5e10017a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5e10036070 0x7f5e10017a40 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f51aaa4730 msgr2=0x55f51aaa6b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f51aaa4730 0x55f51aaa6b20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ce0034650 msgr2=0x7f5ce0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ce0034650 0x7f5ce0036b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5e08012fd0 tx=0x7f5e0803d000 comp rx=0 tx=0).stop 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e16cfb640 1 -- 172.21.15.110:0/3543926853 reap_dead start 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 msgr2=0x55f51aa8fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa8fbf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5e100168b0 tx=0x7f5e100498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 shutdown_connections 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/3543926853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa88760 0x55f51aa8fbf0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 >> 172.21.15.110:0/3543926853 conn(0x55f51aa84aa0 msgr2=0x55f51aa82100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 shutdown_connections 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/3543926853 wait complete. 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cf00346a0 msgr2=0x7f5cf0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cf00346a0 0x7f5cf0036b60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x55f51aa6b9f0 tx=0x7f5e08048350 comp rx=0 tx=0).stop 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa7b5c0 msgr2=0x55f51aa7b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa7b5c0 0x55f51aa7b9e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000aab0 tx=0x7f5e1000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 shutdown_connections 2024-04-27T03:18:32.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cf00346a0 0x7f5cf0036b60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/999875313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51aa7b5c0 0x55f51aa7b9e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 >> 172.21.15.110:0/999875313 conn(0x55f51aa703d0 msgr2=0x55f51aa70ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 shutdown_connections 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/999875313 wait complete. 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa689d0 msgr2=0x55f51aa6adc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa689d0 0x55f51aa6adc0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51a972a90 msgr2=0x55f51aa67590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51a972a90 0x55f51aa67590 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cfc034650 msgr2=0x7f5cfc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cfc034650 0x7f5cfc036b10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c0ae000 tx=0x7f5e0c0a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 msgr2=0x55f51aa60660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa60660 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5e08006640 tx=0x7f5e08000f50 comp rx=0 tx=0).stop 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 shutdown_connections 2024-04-27T03:18:32.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f51aa689d0 0x55f51aa6adc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f51a972a90 0x55f51aa67590 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5cfc034650 0x7f5cfc036b10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 --2- 172.21.15.110:0/634627265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f51a8dffd0 0x55f51aa60660 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 >> 172.21.15.110:0/634627265 conn(0x55f51a966a50 msgr2=0x55f51aa53cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 shutdown_connections 2024-04-27T03:18:32.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.392+0000 7f5e199e1980 1 -- 172.21.15.110:0/634627265 wait complete. 2024-04-27T03:18:32.412 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [{'type': 'user', 'perm': 'read'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:32.412 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'rm', '--uid', 'foo', '--caps', 'user=read'] 2024-04-27T03:18:32.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'rm', '--uid', 'foo', '--caps', 'user=read'] 2024-04-27T03:18:32.412 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps rm --uid foo --caps user=read 2024-04-27T03:18:32.493 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:32.493 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8ebad8d980 1 Processor -- start 2024-04-27T03:18:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8ebad8d980 1 -- start start 2024-04-27T03:18:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8ebad8d980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55bd894654c0 0x55bd894658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8ebad8d980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55bd893220d0 con 0x55bd89465de0 2024-04-27T03:18:32.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8ebad8d980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55bd89317430 con 0x55bd894654c0 2024-04-27T03:18:32.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8eb88a8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55bd894654c0 0x55bd894658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8eb80a7640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55bd89465de0 0x55bd894667a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51284/0 (socket says 172.21.15.110:51284) 2024-04-27T03:18:32.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8eb88a8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55bd894654c0 0x55bd894658a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41200/0 (socket says 172.21.15.110:41200) 2024-04-27T03:18:32.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.484+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2971053011 learned_addr learned my addr 172.21.15.110:0/2971053011 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:18:32.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2971053011 >> v1:172.21.15.77:6789/0 conn(0x55bd89465de0 legacy=0x55bd894667a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2971053011 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd89310510 con 0x55bd894654c0 2024-04-27T03:18:32.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/2971053011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd894658a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800a420 tx=0x7f8ea801afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8942f8c5f2ab9393 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8eb78a6640 1 -- 172.21.15.110:0/2971053011 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ea80af070 con 0x55bd894654c0 2024-04-27T03:18:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8eb78a6640 1 -- 172.21.15.110:0/2971053011 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea80aa530 con 0x55bd894654c0 2024-04-27T03:18:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8eb78a6640 1 -- 172.21.15.110:0/2971053011 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ea80aa850 con 0x55bd894654c0 2024-04-27T03:18:32.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2971053011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 msgr2=0x55bd894658a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2971053011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd894658a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800a420 tx=0x7f8ea801afd0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2971053011 shutdown_connections 2024-04-27T03:18:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2971053011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd894658a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2971053011 >> 172.21.15.110:0/2971053011 conn(0x55bd893d8ba0 msgr2=0x55bd893d8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2971053011 shutdown_connections 2024-04-27T03:18:32.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.488+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2971053011 wait complete. 2024-04-27T03:18:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.504+0000 7f8ebad8d980 1 Processor -- start 2024-04-27T03:18:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.504+0000 7f8ebad8d980 1 -- start start 2024-04-27T03:18:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.504+0000 7f8ebad8d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89550d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.504+0000 7f8ebad8d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd893662d0 con 0x55bd894654c0 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89550d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89550d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41206/0 (socket says 172.21.15.110:41206) 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 -- 172.21.15.110:0/694754036 learned_addr learned my addr 172.21.15.110:0/694754036 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 -- 172.21.15.110:0/694754036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd892ecae0 con 0x55bd894654c0 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/694754036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89550d20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800a420 tx=0x7f8ea80033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f4981ff8be92 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8da6e0d640 1 -- 172.21.15.110:0/694754036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ea80af050 con 0x55bd894654c0 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8da6e0d640 1 -- 172.21.15.110:0/694754036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea80b0040 con 0x55bd894654c0 2024-04-27T03:18:32.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8da6e0d640 1 -- 172.21.15.110:0/694754036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ea80aac60 con 0x55bd894654c0 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 -- 172.21.15.110:0/694754036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 msgr2=0x55bd89550d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/694754036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89550d20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800a420 tx=0x7f8ea80033b0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 -- 172.21.15.110:0/694754036 shutdown_connections 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/694754036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89550d20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 -- 172.21.15.110:0/694754036 >> 172.21.15.110:0/694754036 conn(0x55bd893d8ba0 msgr2=0x55bd8954c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 -- 172.21.15.110:0/694754036 shutdown_connections 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 -- 172.21.15.110:0/694754036 wait complete. 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 Processor -- start 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 -- start start 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89557ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8ebad8d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd8946f510 con 0x55bd894654c0 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89557ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89557ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41214/0 (socket says 172.21.15.110:41214) 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 -- 172.21.15.110:0/896857505 learned_addr learned my addr 172.21.15.110:0/896857505 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd895581e0 con 0x55bd894654c0 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.508+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89557ca0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8ea801b010 tx=0x7f8ea8019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8da560a640 1 -- 172.21.15.110:0/896857505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ea80c5030 con 0x55bd894654c0 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bd89558470 con 0x55bd894654c0 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8da560a640 1 -- 172.21.15.110:0/896857505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea80c15b0 con 0x55bd894654c0 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd8955b050 con 0x55bd894654c0 2024-04-27T03:18:32.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8da560a640 1 -- 172.21.15.110:0/896857505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ea80c1890 con 0x55bd894654c0 2024-04-27T03:18:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8da560a640 1 -- 172.21.15.110:0/896857505 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8ea80c8070 con 0x55bd894654c0 2024-04-27T03:18:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8da560a640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8da00346a0 0x7f8da0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8da560a640 1 -- 172.21.15.110:0/896857505 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f8ea80c6070 con 0x55bd894654c0 2024-04-27T03:18:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8eb80a7640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8da00346a0 0x7f8da0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89465de0 0x55bd8955eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd8955f1e0 con 0x55bd89465de0 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89465de0 0x55bd8955eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8eb80a7640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8da00346a0 0x7f8da0036b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0000f70 tx=0x7f8ea00012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89465de0 0x55bd8955eca0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8eb90a9640 1 -- 172.21.15.110:0/896857505 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8eb00083f0 con 0x55bd89465de0 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd8955ff40 0x55bd89562330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd89562940 con 0x55bd8955ff40 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd8955ff40 0x55bd89562330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.512+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd8955ff40 0x55bd89562330 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb88a8640 1 -- 172.21.15.110:0/896857505 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8955ff40 2024-04-27T03:18:32.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895641a0 con 0x55bd89465de0 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb90a9640 1 -- 172.21.15.110:0/896857505 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f8eb00083f0 con 0x55bd89465de0 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895633a0 con 0x55bd8955ff40 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb88a8640 1 -- 172.21.15.110:0/896857505 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8955ff40 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd89563790 con 0x55bd89465de0 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb90a9640 1 -- 172.21.15.110:0/896857505 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f8eb00083f0 con 0x55bd89465de0 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8ebad8d980 1 Processor -- start 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8ebad8d980 1 -- start start 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8ebad8d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd89572ae0 0x55bd89572f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8ebad8d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd8936a310 con 0x55bd89572ae0 2024-04-27T03:18:32.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb90a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd89572ae0 0x55bd89572f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb90a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd89572ae0 0x55bd89572f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41226/0 (socket says 172.21.15.110:41226) 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb90a9640 1 -- 172.21.15.110:0/3114031922 learned_addr learned my addr 172.21.15.110:0/3114031922 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb90a9640 1 -- 172.21.15.110:0/3114031922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd89573440 con 0x55bd89572ae0 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.516+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd89572ae0 0x55bd89572f00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000aab0 tx=0x7f8eb000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d9e7fc640 1 -- 172.21.15.110:0/3114031922 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eb0007d30 con 0x55bd89572ae0 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d9e7fc640 1 -- 172.21.15.110:0/3114031922 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb001d070 con 0x55bd89572ae0 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bd89573ee0 con 0x55bd89572ae0 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d9e7fc640 1 -- 172.21.15.110:0/3114031922 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eb0018440 con 0x55bd89572ae0 2024-04-27T03:18:32.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd895743b0 con 0x55bd89572ae0 2024-04-27T03:18:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d9e7fc640 1 -- 172.21.15.110:0/3114031922 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8eb0013070 con 0x55bd89572ae0 2024-04-27T03:18:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d9e7fc640 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d90034650 0x7f8d90036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d9e7fc640 1 -- 172.21.15.110:0/3114031922 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f8eb004b520 con 0x55bd89572ae0 2024-04-27T03:18:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d90034650 0x7f8d90036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8ebad8d980 1 Processor -- start 2024-04-27T03:18:32.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8ebad8d980 1 -- start start 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d90034650 0x7f8d90036b10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55bd89562ec0 tx=0x7f8ea80e3560 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8ebad8d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895800b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8ebad8d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd895513b0 con 0x55bd8957fc90 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8eb90a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895800b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8eb90a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895800b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41234/0 (socket says 172.21.15.110:41234) 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8eb90a9640 1 -- 172.21.15.110:0/1034255216 learned_addr learned my addr 172.21.15.110:0/1034255216 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8eb90a9640 1 -- 172.21.15.110:0/1034255216 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd89580680 con 0x55bd8957fc90 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/1034255216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895800b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0049930 tx=0x7f8eb0018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd8ed670e30b2fd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d8ffff640 1 -- 172.21.15.110:0/1034255216 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eb0047070 con 0x55bd8957fc90 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d8ffff640 1 -- 172.21.15.110:0/1034255216 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb0018bc0 con 0x55bd8957fc90 2024-04-27T03:18:32.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.520+0000 7f8d8ffff640 1 -- 172.21.15.110:0/1034255216 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eb0020360 con 0x55bd8957fc90 2024-04-27T03:18:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/1034255216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 msgr2=0x55bd895800b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/1034255216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895800b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0049930 tx=0x7f8eb0018950 comp rx=0 tx=0).stop 2024-04-27T03:18:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/1034255216 shutdown_connections 2024-04-27T03:18:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/1034255216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895800b0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/1034255216 >> 172.21.15.110:0/1034255216 conn(0x55bd8957c020 msgr2=0x55bd8957a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/1034255216 shutdown_connections 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/1034255216 wait complete. 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 Processor -- start 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- start start 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895871e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55bd89580df0 con 0x55bd8957fc90 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8eb90a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8eb90a9640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895871e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41246/0 (socket says 172.21.15.110:41246) 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 learned_addr learned my addr 172.21.15.110:0/2170768681 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd89587720 con 0x55bd8957fc90 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895871e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8eb00168b0 tx=0x7f8eb00498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8d8e7fc640 1 -- 172.21.15.110:0/2170768681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eb0031070 con 0x55bd8957fc90 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8d8e7fc640 1 -- 172.21.15.110:0/2170768681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb0020900 con 0x55bd8957fc90 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bd895879b0 con 0x55bd8957fc90 2024-04-27T03:18:32.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8d8e7fc640 1 -- 172.21.15.110:0/2170768681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8eb0049a70 con 0x55bd8957fc90 2024-04-27T03:18:32.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd8958a690 con 0x55bd8957fc90 2024-04-27T03:18:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8d8e7fc640 1 -- 172.21.15.110:0/2170768681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8eb0034070 con 0x55bd8957fc90 2024-04-27T03:18:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8d8e7fc640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d84034650 0x7f8d84036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8d8e7fc640 1 -- 172.21.15.110:0/2170768681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7f8eb0037c90 con 0x55bd8957fc90 2024-04-27T03:18:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d84034650 0x7f8d84036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89592840 0x55bd89594d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd89595450 con 0x55bd89592840 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.524+0000 7f8eb80a7640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89592840 0x55bd89594d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d84034650 0x7f8d84036b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80c8ad0 tx=0x7f8ea80bf980 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb80a7640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89592840 0x55bd89594d00 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8ea000e5a0 con 0x55bd89592840 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 realm 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd89591c90 con 0x55bd89592840 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8ea000e5a0 con 0x55bd89592840 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd89595f50 con 0x55bd89592840 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8ea000e5a0 con 0x55bd89592840 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd89596d30 0x55bd89599120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895997d0 con 0x55bd89596d30 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd89596d30 0x55bd89599120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd89596d30 0x55bd89599120 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959a020 con 0x55bd89592840 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f8ea000e5a0 con 0x55bd89592840 2024-04-27T03:18:32.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 20 searching for the correct realm 2024-04-27T03:18:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bd8959bd90 0x55bd8959e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd8959bd90 2024-04-27T03:18:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bd8959bd90 0x55bd8959e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.528+0000 7f8eb88a8640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bd8959bd90 0x55bd8959e180 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb90a9640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8eb0036070 0x7f8eb003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x7f8eb0036070 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8eb0036070 0x7f8eb003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8eb0036070 0x7f8eb003b620 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea000e5a0 con 0x7f8eb0036070 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea000e5a0 con 0x55bd89592840 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd8959bd90 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.532+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x7f8eb0036070 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x7f8eb0036070 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x7f8eb0036070 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f8ea0029040 con 0x7f8eb0036070 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd8959bd90 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.536+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd8959bd90 2024-04-27T03:18:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd8959bd90 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x7f8eb0036070 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x7f8eb0036070 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89592840 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x7f8eb0036070 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x7f8eb0036070 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd8959bd90 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x7f8eb0036070 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea0029040 con 0x7f8eb0036070 2024-04-27T03:18:32.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.540+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd8959bd90 2024-04-27T03:18:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959e800 con 0x55bd89596d30 2024-04-27T03:18:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:32.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959a9a0 con 0x55bd89592840 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8ea0029040 con 0x55bd89592840 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959a5b0 con 0x55bd89596d30 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959b290 con 0x55bd89592840 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 zone default found 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 4 Realm: () 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 zonegroup default 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959eca0 con 0x7f8eb0036070 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x7f8eb0036070 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a0ac0 con 0x55bd8959bd90 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.544+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a0e60 con 0x55bd8959bd90 2024-04-27T03:18:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 20 started sync module instance, tier type = 2024-04-27T03:18:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:32.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd8959ff60 con 0x55bd89596d30 2024-04-27T03:18:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94272541557584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a1740 con 0x55bd89596d30 2024-04-27T03:18:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a1f90 con 0x55bd89592840 2024-04-27T03:18:32.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94272541562688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a2110 con 0x55bd89592840 2024-04-27T03:18:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.548+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a2c30 con 0x55bd89596d30 2024-04-27T03:18:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94272541557584] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94272541562688] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94272541566720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a2bf0 con 0x55bd89596d30 2024-04-27T03:18:32.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a34d0 con 0x7f8eb0036070 2024-04-27T03:18:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x7f8eb0036070 2024-04-27T03:18:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94272541569056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a2bf0 con 0x7f8eb0036070 2024-04-27T03:18:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a3f80 con 0x55bd89596d30 2024-04-27T03:18:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.552+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94272541566720] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94272541569056] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x7f8eb0036070 2024-04-27T03:18:32.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94272541570112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a3f80 con 0x55bd89596d30 2024-04-27T03:18:32.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a4370 con 0x55bd8959bd90 2024-04-27T03:18:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94272541561488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a4bc0 con 0x55bd8959bd90 2024-04-27T03:18:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a55c0 con 0x55bd8959bd90 2024-04-27T03:18:32.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94272541570112] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94272541571728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a59b0 con 0x55bd8959bd90 2024-04-27T03:18:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a63f0 con 0x7f8eb0036070 2024-04-27T03:18:32.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.556+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94272541561488] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'183 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x7f8eb0036070 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94272541580992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a63b0 con 0x7f8eb0036070 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=0 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=1 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=2 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=3 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=4 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=5 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94272541571728] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=6 2024-04-27T03:18:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94272541580992] v27'184 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x7f8eb0036070 2024-04-27T03:18:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 20 add_watcher() i=7 2024-04-27T03:18:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a50d0 con 0x55bd89592840 2024-04-27T03:18:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.560+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94272541577424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895a68d0 con 0x55bd89592840 2024-04-27T03:18:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94272541577424] v27'195 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8d6f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 5 note: GC not initialized 2024-04-27T03:18:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8d6e7e4640 20 reqs_thread_entry: start 2024-04-27T03:18:32.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 20 Filter name: none 2024-04-27T03:18:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8d67fff640 20 reqs_thread_entry: start 2024-04-27T03:18:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ccc30 con 0x55bd89592840 2024-04-27T03:18:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(62 foo [call out=48b,read 0~403 out=403b,getxattrs out=75b] v0'0 uv14 ondisk = 0) ==== 231+0+526 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 20 rados_obj.operate() r=0 bl.length=403 2024-04-27T03:18:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:32.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1e6ca5ed:users.uid::foo:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~391 in=391b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895d2db0 con 0x55bd89592840 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(63 foo [delete,create,call,call,writefull 0~391] v27'15 uv15 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 10 moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.564+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94272541763616 in=579b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ce6b0 con 0x55bd89596d30 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94272541570112 notify 115964117031 ret 0) ==== 609+0+0 (crc 0 0 0) 0x55bd8958a690 con 0x55bd89596d30 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8d8ffff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964117031 cookie 94272541570112 notifier 4653 bl.length()=567 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8d8ffff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo info.flags=0x17 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8d8ffff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:32.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8d8ffff640 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x7f8d7c003eb0 con 0x55bd89596d30 2024-04-27T03:18:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94272541763616 out=8b] v0'0 uv183 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94272541763616 notify 115964117031 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f8eb0063040 con 0x55bd89596d30 2024-04-27T03:18:32.576 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"foo","display_name":"Foo","email":"foo@foo.com","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"foo","access_key":"9te6NH5mcdcq0Tc5i8i1","secret_key":"Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu","active":true,"create_date":"2024-04-27T03:18:06.177162Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:18:06.177152Z","tags":[],"group_ids":[]} 2024-04-27T03:18:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8d6f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.568+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94272541577424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x55bd89592840 2024-04-27T03:18:32.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94272541577424] v27'196 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94272541557584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x55bd89596d30 2024-04-27T03:18:32.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94272541557584] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8ebad8d980 20 remove_watcher() i=0 2024-04-27T03:18:32.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8ebad8d980 2 removed watcher, disabling cache 2024-04-27T03:18:32.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94272541562688] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x55bd89592840 2024-04-27T03:18:32.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94272541562688] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x55bd89592840 2024-04-27T03:18:32.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8ebad8d980 20 remove_watcher() i=1 2024-04-27T03:18:32.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.572+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94272541566720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x55bd89596d30 2024-04-27T03:18:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.576+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94272541566720] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.576+0000 7f8ebad8d980 20 remove_watcher() i=2 2024-04-27T03:18:32.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.576+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94272541569056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x7f8eb0036070 2024-04-27T03:18:32.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.576+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94272541569056] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x7f8eb0036070 2024-04-27T03:18:32.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.576+0000 7f8ebad8d980 20 remove_watcher() i=3 2024-04-27T03:18:32.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.576+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94272541570112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x55bd89596d30 2024-04-27T03:18:32.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8eb90a9640 1 -- 172.21.15.110:0/2170768681 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94272541570112] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8eb0033030 con 0x55bd89596d30 2024-04-27T03:18:32.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8ebad8d980 20 remove_watcher() i=4 2024-04-27T03:18:32.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94272541561488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x55bd8959bd90 2024-04-27T03:18:32.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94272541561488] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8ebad8d980 20 remove_watcher() i=5 2024-04-27T03:18:32.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94272541571728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x55bd8959bd90 2024-04-27T03:18:32.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8eb88a8640 1 -- 172.21.15.110:0/2170768681 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94272541571728] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea80b2070 con 0x55bd8959bd90 2024-04-27T03:18:32.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8ebad8d980 20 remove_watcher() i=6 2024-04-27T03:18:32.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.580+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94272541580992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55bd895ca960 con 0x7f8eb0036070 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94272541580992] v27'185 uv183 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8ea003c1c0 con 0x7f8eb0036070 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 20 remove_watcher() i=7 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd89596d30 msgr2=0x55bd89599120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd89596d30 0x55bd89599120 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89592840 msgr2=0x55bd89594d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89592840 0x55bd89594d00 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8eb0036070 msgr2=0x7f8eb003b620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8eb0036070 0x7f8eb003b620 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bd8959bd90 msgr2=0x55bd8959e180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55bd8959bd90 0x55bd8959e180 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d84034650 msgr2=0x7f8d84036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d84034650 0x7f8d84036b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80c8ad0 tx=0x7f8ea80bf980 comp rx=0 tx=0).stop 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 msgr2=0x55bd895871e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8eb80a7640 1 -- 172.21.15.110:0/2170768681 reap_dead start 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895871e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8eb00168b0 tx=0x7f8eb00498c0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 shutdown_connections 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/2170768681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd8957fc90 0x55bd895871e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 >> 172.21.15.110:0/2170768681 conn(0x55bd8957c020 msgr2=0x55bd89581510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 shutdown_connections 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/2170768681 wait complete. 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d90034650 msgr2=0x7f8d90036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d90034650 0x7f8d90036b10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55bd89562ec0 tx=0x7f8ea80e3560 comp rx=0 tx=0).stop 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd89572ae0 msgr2=0x55bd89572f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd89572ae0 0x55bd89572f00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000aab0 tx=0x7f8eb000af70 comp rx=0 tx=0).stop 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 shutdown_connections 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8d90034650 0x7f8d90036b10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/3114031922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd89572ae0 0x55bd89572f00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 >> 172.21.15.110:0/3114031922 conn(0x55bd895678f0 msgr2=0x55bd89568400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 shutdown_connections 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/3114031922 wait complete. 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd8955ff40 msgr2=0x55bd89562330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd8955ff40 0x55bd89562330 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89465de0 msgr2=0x55bd8955eca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89465de0 0x55bd8955eca0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8da00346a0 msgr2=0x7f8da0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8da00346a0 0x7f8da0036b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0000f70 tx=0x7f8ea00012c0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 msgr2=0x55bd89557ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89557ca0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8ea801b010 tx=0x7f8ea8019810 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 shutdown_connections 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55bd8955ff40 0x55bd89562330 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55bd89465de0 0x55bd8955eca0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8da00346a0 0x7f8da0036b60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 --2- 172.21.15.110:0/896857505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55bd894654c0 0x55bd89557ca0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 >> 172.21.15.110:0/896857505 conn(0x55bd893d8ba0 msgr2=0x55bd8954b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 shutdown_connections 2024-04-27T03:18:32.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.584+0000 7f8ebad8d980 1 -- 172.21.15.110:0/896857505 wait complete. 2024-04-27T03:18:32.604 DEBUG:tasks.util.rgw: json result: {'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True, 'create_date': '2024-04-27T03:18:06.177162Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:18:06.177152Z', 'tags': [], 'group_ids': []} 2024-04-27T03:18:32.619 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:32.619 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo'] 2024-04-27T03:18:32.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo'] 2024-04-27T03:18:32.619 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo 2024-04-27T03:18:32.677 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:18:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:18:32.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.668+0000 7fb475adb980 1 Processor -- start 2024-04-27T03:18:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb475adb980 1 -- start start 2024-04-27T03:18:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb475adb980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55e0f042d9c0 0x55e0f042fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb475adb980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55e0f02e60d0 con 0x55e0f04294b0 2024-04-27T03:18:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb475adb980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55e0f02db430 con 0x55e0f042d9c0 2024-04-27T03:18:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb472df5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55e0f042d9c0 0x55e0f042fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb472df5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55e0f042d9c0 0x55e0f042fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41252/0 (socket says 172.21.15.110:41252) 2024-04-27T03:18:32.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb472df5640 1 -- 172.21.15.110:0/2575322129 learned_addr learned my addr 172.21.15.110:0/2575322129 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb472df5640 1 -- 172.21.15.110:0/2575322129 >> v1:172.21.15.77:6789/0 conn(0x55e0f04294b0 legacy=0x55e0f0429890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb472df5640 1 -- 172.21.15.110:0/2575322129 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f02d4510 con 0x55e0f042d9c0 2024-04-27T03:18:32.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.672+0000 7fb472df5640 1 --2- 172.21.15.110:0/2575322129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f042d9c0 0x55e0f042fdb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb46800a5c0 tx=0x7fb46801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b0f4207214e373 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb4725f4640 1 -- 172.21.15.110:0/2575322129 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4680a6070 con 0x55e0f042d9c0 2024-04-27T03:18:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb4725f4640 1 -- 172.21.15.110:0/2575322129 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4680af070 con 0x55e0f042d9c0 2024-04-27T03:18:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb4725f4640 1 -- 172.21.15.110:0/2575322129 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4680aa5c0 con 0x55e0f042d9c0 2024-04-27T03:18:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb475adb980 1 -- 172.21.15.110:0/2575322129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f042d9c0 msgr2=0x55e0f042fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb475adb980 1 --2- 172.21.15.110:0/2575322129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f042d9c0 0x55e0f042fdb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb46800a5c0 tx=0x7fb46801b340 comp rx=0 tx=0).stop 2024-04-27T03:18:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb475adb980 1 -- 172.21.15.110:0/2575322129 shutdown_connections 2024-04-27T03:18:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb475adb980 1 --2- 172.21.15.110:0/2575322129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f042d9c0 0x55e0f042fdb0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb475adb980 1 -- 172.21.15.110:0/2575322129 >> 172.21.15.110:0/2575322129 conn(0x55e0f039cb90 msgr2=0x55e0f039cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb475adb980 1 -- 172.21.15.110:0/2575322129 shutdown_connections 2024-04-27T03:18:32.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.676+0000 7fb475adb980 1 -- 172.21.15.110:0/2575322129 wait complete. 2024-04-27T03:18:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.688+0000 7fb475adb980 1 Processor -- start 2024-04-27T03:18:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.688+0000 7fb475adb980 1 -- start start 2024-04-27T03:18:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f05145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e0f032a2d0 con 0x55e0f04294b0 2024-04-27T03:18:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f05145d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f05145d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41254/0 (socket says 172.21.15.110:41254) 2024-04-27T03:18:32.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 -- 172.21.15.110:0/743281146 learned_addr learned my addr 172.21.15.110:0/743281146 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 -- 172.21.15.110:0/743281146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f02b0ae0 con 0x55e0f04294b0 2024-04-27T03:18:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 --2- 172.21.15.110:0/743281146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f05145d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb464001ea0 tx=0x7fb46400b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7275835f3bb72d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb36160a640 1 -- 172.21.15.110:0/743281146 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46400a040 con 0x55e0f04294b0 2024-04-27T03:18:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb36160a640 1 -- 172.21.15.110:0/743281146 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46400f620 con 0x55e0f04294b0 2024-04-27T03:18:32.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb36160a640 1 -- 172.21.15.110:0/743281146 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46400f890 con 0x55e0f04294b0 2024-04-27T03:18:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- 172.21.15.110:0/743281146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 msgr2=0x55e0f05145d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 --2- 172.21.15.110:0/743281146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f05145d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb464001ea0 tx=0x7fb46400b680 comp rx=0 tx=0).stop 2024-04-27T03:18:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- 172.21.15.110:0/743281146 shutdown_connections 2024-04-27T03:18:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 --2- 172.21.15.110:0/743281146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f05145d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- 172.21.15.110:0/743281146 >> 172.21.15.110:0/743281146 conn(0x55e0f039cb90 msgr2=0x55e0f0510160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- 172.21.15.110:0/743281146 shutdown_connections 2024-04-27T03:18:32.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- 172.21.15.110:0/743281146 wait complete. 2024-04-27T03:18:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 Processor -- start 2024-04-27T03:18:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- start start 2024-04-27T03:18:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f051b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb475adb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e0f0432de0 con 0x55e0f04294b0 2024-04-27T03:18:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f051b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f051b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41264/0 (socket says 172.21.15.110:41264) 2024-04-27T03:18:32.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.692+0000 7fb4735f6640 1 -- 172.21.15.110:0/3254696761 learned_addr learned my addr 172.21.15.110:0/3254696761 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb4735f6640 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f051bb50 con 0x55e0f04294b0 2024-04-27T03:18:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb4735f6640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f051b610 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb464001fd0 tx=0x7fb464000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb35fe07640 1 -- 172.21.15.110:0/3254696761 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb464001520 con 0x55e0f04294b0 2024-04-27T03:18:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e0f051bde0 con 0x55e0f04294b0 2024-04-27T03:18:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb35fe07640 1 -- 172.21.15.110:0/3254696761 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46401ca10 con 0x55e0f04294b0 2024-04-27T03:18:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb35fe07640 1 -- 172.21.15.110:0/3254696761 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46402c060 con 0x55e0f04294b0 2024-04-27T03:18:32.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f051ea60 con 0x55e0f04294b0 2024-04-27T03:18:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb35fe07640 1 -- 172.21.15.110:0/3254696761 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb464017050 con 0x55e0f04294b0 2024-04-27T03:18:32.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb35fe07640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb358034650 0x7fb358036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb35fe07640 1 -- 172.21.15.110:0/3254696761 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb464015020 con 0x55e0f04294b0 2024-04-27T03:18:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb472df5640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb358034650 0x7fb358036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f042d9c0 0x55e0f05226b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0522bf0 con 0x55e0f042d9c0 2024-04-27T03:18:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb473df7640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f042d9c0 0x55e0f05226b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb472df5640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb358034650 0x7fb358036b10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb4680ae000 tx=0x7fb4680a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb473df7640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f042d9c0 0x55e0f05226b0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb473df7640 1 -- 172.21.15.110:0/3254696761 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb46c00a4d0 con 0x55e0f042d9c0 2024-04-27T03:18:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f05239a0 0x55e0f0525d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05263c0 con 0x55e0f05239a0 2024-04-27T03:18:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.696+0000 7fb4735f6640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f05239a0 0x55e0f0525d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb4735f6640 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f05239a0 0x55e0f0525d90 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb4735f6640 1 -- 172.21.15.110:0/3254696761 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f05239a0 2024-04-27T03:18:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0527c70 con 0x55e0f042d9c0 2024-04-27T03:18:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb473df7640 1 -- 172.21.15.110:0/3254696761 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fb46c00a4d0 con 0x55e0f042d9c0 2024-04-27T03:18:32.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0526ea0 con 0x55e0f05239a0 2024-04-27T03:18:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb4735f6640 1 -- 172.21.15.110:0/3254696761 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f05239a0 2024-04-27T03:18:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0527290 con 0x55e0f042d9c0 2024-04-27T03:18:32.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb473df7640 1 -- 172.21.15.110:0/3254696761 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fb46c00a4d0 con 0x55e0f042d9c0 2024-04-27T03:18:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb475adb980 1 Processor -- start 2024-04-27T03:18:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb475adb980 1 -- start start 2024-04-27T03:18:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb475adb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0536570 0x55e0f0536990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb475adb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e0f032e310 con 0x55e0f0536570 2024-04-27T03:18:32.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb473df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0536570 0x55e0f0536990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb473df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0536570 0x55e0f0536990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41272/0 (socket says 172.21.15.110:41272) 2024-04-27T03:18:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.700+0000 7fb473df7640 1 -- 172.21.15.110:0/877153071 learned_addr learned my addr 172.21.15.110:0/877153071 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb473df7640 1 -- 172.21.15.110:0/877153071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f0536ed0 con 0x55e0f0536570 2024-04-27T03:18:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb473df7640 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0536570 0x55e0f0536990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00dfc0 tx=0x7fb46c0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb35d602640 1 -- 172.21.15.110:0/877153071 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46c0157d0 con 0x55e0f0536570 2024-04-27T03:18:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb35d602640 1 -- 172.21.15.110:0/877153071 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46c015970 con 0x55e0f0536570 2024-04-27T03:18:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb35d602640 1 -- 172.21.15.110:0/877153071 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46c015c70 con 0x55e0f0536570 2024-04-27T03:18:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e0f0537970 con 0x55e0f0536570 2024-04-27T03:18:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f0537e40 con 0x55e0f0536570 2024-04-27T03:18:32.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb35d602640 1 -- 172.21.15.110:0/877153071 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb46c029440 con 0x55e0f0536570 2024-04-27T03:18:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb35d602640 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb34c034650 0x7fb34c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb35d602640 1 -- 172.21.15.110:0/877153071 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb46c00e040 con 0x55e0f0536570 2024-04-27T03:18:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb4735f6640 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb34c034650 0x7fb34c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb475adb980 1 Processor -- start 2024-04-27T03:18:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb475adb980 1 -- start start 2024-04-27T03:18:32.714 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb4735f6640 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb34c034650 0x7fb34c036b10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55e0f05269c0 tx=0x7fb4640486d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb475adb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f0543b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb475adb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e0f0514d20 con 0x55e0f0543720 2024-04-27T03:18:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb473df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f0543b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb473df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f0543b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41280/0 (socket says 172.21.15.110:41280) 2024-04-27T03:18:32.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.704+0000 7fb473df7640 1 -- 172.21.15.110:0/534387025 learned_addr learned my addr 172.21.15.110:0/534387025 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb473df7640 1 -- 172.21.15.110:0/534387025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f0544110 con 0x55e0f0543720 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb473df7640 1 --2- 172.21.15.110:0/534387025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f0543b40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb46c015f40 tx=0x7fb46c03c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e781b9c050c9a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb356ffd640 1 -- 172.21.15.110:0/534387025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46c04c110 con 0x55e0f0543720 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb356ffd640 1 -- 172.21.15.110:0/534387025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46c0283a0 con 0x55e0f0543720 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb356ffd640 1 -- 172.21.15.110:0/534387025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46c04c2b0 con 0x55e0f0543720 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- 172.21.15.110:0/534387025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 msgr2=0x55e0f0543b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 --2- 172.21.15.110:0/534387025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f0543b40 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb46c015f40 tx=0x7fb46c03c280 comp rx=0 tx=0).stop 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- 172.21.15.110:0/534387025 shutdown_connections 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 --2- 172.21.15.110:0/534387025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f0543b40 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- 172.21.15.110:0/534387025 >> 172.21.15.110:0/534387025 conn(0x55e0f053fab0 msgr2=0x55e0f053db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- 172.21.15.110:0/534387025 shutdown_connections 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- 172.21.15.110:0/534387025 wait complete. 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 Processor -- start 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- start start 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f054abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55e0f0544880 con 0x55e0f0543720 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb473df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f054abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb473df7640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f054abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:41284/0 (socket says 172.21.15.110:41284) 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 learned_addr learned my addr 172.21.15.110:0/3534579961 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0f054b0f0 con 0x55e0f0543720 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb473df7640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f054abb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb46c012750 tx=0x7fb46c04a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb3557fa640 1 -- 172.21.15.110:0/3534579961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46c036070 con 0x55e0f0543720 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb3557fa640 1 -- 172.21.15.110:0/3534579961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46c0494e0 con 0x55e0f0543720 2024-04-27T03:18:32.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e0f054b380 con 0x55e0f0543720 2024-04-27T03:18:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb3557fa640 1 -- 172.21.15.110:0/3534579961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb46c041040 con 0x55e0f0543720 2024-04-27T03:18:32.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.708+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0f054e060 con 0x55e0f0543720 2024-04-27T03:18:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb3557fa640 1 -- 172.21.15.110:0/3534579961 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb46c039070 con 0x55e0f0543720 2024-04-27T03:18:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb3557fa640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb33c034650 0x7fb33c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb3557fa640 1 -- 172.21.15.110:0/3534579961 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6110+0+0 (secure 0 0 0) 0x7fb46c045020 con 0x55e0f0543720 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb4735f6640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb33c034650 0x7fb33c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f0556210 0x55e0f05586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0558e20 con 0x55e0f0556210 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb472df5640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f0556210 0x55e0f05586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb4735f6640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb33c034650 0x7fb33c036b10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb46401dfd0 tx=0x7fb464037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb472df5640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f0556210 0x55e0f05586d0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 realm 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0559250 con 0x55e0f0556210 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0559b20 con 0x55e0f0556210 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f055a540 0x55e0f055c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f055d0e0 con 0x55e0f055a540 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb473df7640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f055a540 0x55e0f055c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.712+0000 7fb473df7640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f055a540 0x55e0f055c970 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb475adb980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f055d930 con 0x55e0f0556210 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb475adb980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb475adb980 20 searching for the correct realm 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e0f055f6a0 0x55e0f0561a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055f6a0 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb4735f6640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e0f055f6a0 0x55e0f0561a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb4735f6640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e0f055f6a0 0x55e0f0561a90 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb473df7640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb46c014030 0x7fb46c039770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x7fb46c014030 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb472df5640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb46c014030 0x7fb46c039770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb472df5640 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb46c014030 0x7fb46c039770 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.716+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055f6a0 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 27 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x7fb46c014030 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x7fb46c014030 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 27 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055f6a0 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.720+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 27 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055f6a0 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(25 [pgnls start_epoch 27 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(26 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055f6a0 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x7fb46c014030 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(28 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f0556210 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(29 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x7fb46c014030 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(30 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055f6a0 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.724+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x7fb46c014030 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(32 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055f6a0 2024-04-27T03:18:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 27 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 27 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562110 con 0x55e0f055a540 2024-04-27T03:18:32.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 27 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f055e2e0 con 0x55e0f0556210 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.728+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f055df20 con 0x55e0f055a540 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f055ec40 con 0x55e0f0556210 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 zone default found 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 4 Realm: () 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:18:32.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 zonegroup default 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0562a20 con 0x7fb46c014030 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564cd0 con 0x55e0f055f6a0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0565070 con 0x55e0f055f6a0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:18:32.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 started sync module instance, tier type = 2024-04-27T03:18:32.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:18:32.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.732+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564110 con 0x55e0f055a540 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94424593220864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564920 con 0x55e0f055a540 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0565df0 con 0x55e0f0556210 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(46 notify.1 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94424593225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0566690 con 0x55e0f0556210 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05670b0 con 0x55e0f055a540 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94424593220864] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.736+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(47 notify.1 [watch watch cookie 94424593225920] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94424593230752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05661e0 con 0x55e0f055a540 2024-04-27T03:18:32.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0567980 con 0x7fb46c014030 2024-04-27T03:18:32.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(50 notify.3 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94424593233104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05661e0 con 0x7fb46c014030 2024-04-27T03:18:32.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0568420 con 0x55e0f055a540 2024-04-27T03:18:32.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94424593230752] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(51 notify.3 [watch watch cookie 94424593233104] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94424593234160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05683e0 con 0x55e0f055a540 2024-04-27T03:18:32.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.740+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0568810 con 0x55e0f055f6a0 2024-04-27T03:18:32.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94424593224720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05690b0 con 0x55e0f055f6a0 2024-04-27T03:18:32.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0569ab0 con 0x55e0f055f6a0 2024-04-27T03:18:32.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94424593234160] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94424593224720] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94424593235728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0568c00 con 0x55e0f055f6a0 2024-04-27T03:18:32.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f056a350 con 0x7fb46c014030 2024-04-27T03:18:32.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(58 notify.7 [create] v27'186 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94424593243808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f056a770 con 0x7fb46c014030 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 20 add_watcher() i=0 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 20 add_watcher() i=1 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 20 add_watcher() i=2 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 20 add_watcher() i=3 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 20 add_watcher() i=4 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 20 add_watcher() i=5 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94424593235728] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.744+0000 7fb475adb980 20 add_watcher() i=6 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(59 notify.7 [watch watch cookie 94424593243808] v27'187 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb475adb980 20 add_watcher() i=7 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb475adb980 2 all 8 watchers are set, enabling cache 2024-04-27T03:18:32.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f056d3f0 con 0x55e0f0556210 2024-04-27T03:18:32.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94424593241536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f056a770 con 0x55e0f0556210 2024-04-27T03:18:32.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94424593241536] v27'197 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb329fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb475adb980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:18:32.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb475adb980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:18:32.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.748+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f056ce70 con 0x55e0f055a540 2024-04-27T03:18:32.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.752+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v27'55 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.752+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f056ce10 con 0x7fb46c014030 2024-04-27T03:18:32.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.752+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(63 gc.1 [create,call,call,call] v27'32 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.752+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05766c0 con 0x55e0f055f6a0 2024-04-27T03:18:32.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.756+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v27'32 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.756+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0576b30 con 0x55e0f0556210 2024-04-27T03:18:32.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.756+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 gc.3 [create,call,call,call] v27'198 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.756+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0575b20 con 0x55e0f055a540 2024-04-27T03:18:32.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.760+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v27'73 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.760+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0575b20 con 0x55e0f055f6a0 2024-04-27T03:18:32.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.760+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v27'55 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.760+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0575b20 con 0x55e0f055a540 2024-04-27T03:18:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.764+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v27'35 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.764+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0575b20 con 0x55e0f0556210 2024-04-27T03:18:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.764+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 gc.7 [create,call,call,call] v27'50 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.764+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577690 con 0x55e0f055a540 2024-04-27T03:18:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.768+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v27'25 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.768+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577690 con 0x55e0f055a540 2024-04-27T03:18:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.768+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v27'31 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.772+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.772+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v27'74 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.772+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055f6a0 2024-04-27T03:18:32.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.772+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v27'28 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.772+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.776+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v27'31 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.776+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f0556210 2024-04-27T03:18:32.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.776+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(75 gc.13 [create,call,call,call] v27'61 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.776+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f0556210 2024-04-27T03:18:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.780+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 gc.14 [create,call,call,call] v27'199 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.780+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.784+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v27'65 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.784+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.784+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v27'75 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.784+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055f6a0 2024-04-27T03:18:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.788+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v27'38 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.788+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.788+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v27'33 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.788+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f0556210 2024-04-27T03:18:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.792+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(81 gc.19 [create,call,call,call] v27'200 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.792+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.796+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v27'66 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.796+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.796+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v27'41 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.796+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055f6a0 2024-04-27T03:18:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.800+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v27'33 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.800+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.800+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(85 gc.23 [create,call,call,call] v27'43 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.800+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055f6a0 2024-04-27T03:18:32.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.804+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v27'27 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.804+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055f6a0 2024-04-27T03:18:32.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.804+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v27'56 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.804+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055f6a0 2024-04-27T03:18:32.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.808+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v27'34 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.808+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.812+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(89 gc.27 [create,call,call,call] v27'37 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.812+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f0556210 2024-04-27T03:18:32.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.812+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v27'51 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.812+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f055a540 2024-04-27T03:18:32.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.816+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(91 gc.29 [create,call,call,call] v27'67 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.816+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x55e0f0556210 2024-04-27T03:18:32.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.816+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v27'62 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.816+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0577a70 con 0x7fb46c014030 2024-04-27T03:18:32.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(93 gc.31 [create,call,call,call] v27'31 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb328fe1640 20 reqs_thread_entry: start 2024-04-27T03:18:32.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 20 init_complete bucket index max shards: 11 2024-04-27T03:18:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 20 Filter name: none 2024-04-27T03:18:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb2a27fc640 20 reqs_thread_entry: start 2024-04-27T03:18:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:18:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 20 rados->read ofs=0 len=0 2024-04-27T03:18:32.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0574e90 con 0x55e0f0556210 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(94 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:95 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0574a10 con 0x7fb46c014030 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(95 foo.buckets [call out=183b] v0'0 uv41 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.829 INFO:teuthology.orchestra.run.smithi110.stderr:could not remove user: unable to remove user, must specify purge data to remove user with buckets 2024-04-27T03:18:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb329fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:18:32.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.820+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94424593241536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0550610 con 0x55e0f0556210 2024-04-27T03:18:32.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.824+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(96 data_loggenerations_metadata [watch unwatch cookie 94424593241536] v27'201 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.824+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94424593220864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05695c0 con 0x55e0f055a540 2024-04-27T03:18:32.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(97 notify.0 [watch unwatch cookie 94424593220864] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb475adb980 20 remove_watcher() i=0 2024-04-27T03:18:32.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb475adb980 2 removed watcher, disabling cache 2024-04-27T03:18:32.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:98 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94424593225920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564500 con 0x55e0f0556210 2024-04-27T03:18:32.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(98 notify.1 [watch unwatch cookie 94424593225920] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x55e0f0556210 2024-04-27T03:18:32.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb475adb980 20 remove_watcher() i=1 2024-04-27T03:18:32.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94424593230752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564500 con 0x55e0f055a540 2024-04-27T03:18:32.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(99 notify.2 [watch unwatch cookie 94424593230752] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb475adb980 20 remove_watcher() i=2 2024-04-27T03:18:32.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.828+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:100 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94424593233104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564500 con 0x7fb46c014030 2024-04-27T03:18:32.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.832+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(100 notify.3 [watch unwatch cookie 94424593233104] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.832+0000 7fb475adb980 20 remove_watcher() i=3 2024-04-27T03:18:32.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.832+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94424593234160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f05674d0 con 0x55e0f055a540 2024-04-27T03:18:32.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.832+0000 7fb473df7640 1 -- 172.21.15.110:0/3534579961 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(101 notify.4 [watch unwatch cookie 94424593234160] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46c038030 con 0x55e0f055a540 2024-04-27T03:18:32.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.832+0000 7fb475adb980 20 remove_watcher() i=4 2024-04-27T03:18:32.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.832+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:102 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94424593224720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564500 con 0x55e0f055f6a0 2024-04-27T03:18:32.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(102 notify.5 [watch unwatch cookie 94424593224720] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb475adb980 20 remove_watcher() i=5 2024-04-27T03:18:32.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:103 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94424593235728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0565410 con 0x55e0f055f6a0 2024-04-27T03:18:32.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb4735f6640 1 -- 172.21.15.110:0/3534579961 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(103 notify.6 [watch unwatch cookie 94424593235728] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb46402e070 con 0x55e0f055f6a0 2024-04-27T03:18:32.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb475adb980 20 remove_watcher() i=6 2024-04-27T03:18:32.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:104 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94424593243808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55e0f0564500 con 0x7fb46c014030 2024-04-27T03:18:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(104 notify.7 [watch unwatch cookie 94424593243808] v27'188 uv186 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fb4680aaa20 con 0x7fb46c014030 2024-04-27T03:18:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.836+0000 7fb475adb980 20 remove_watcher() i=7 2024-04-27T03:18:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f055a540 msgr2=0x55e0f055c970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f055a540 0x55e0f055c970 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f0556210 msgr2=0x55e0f05586d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f0556210 0x55e0f05586d0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb46c014030 msgr2=0x7fb46c039770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fb46c014030 0x7fb46c039770 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e0f055f6a0 msgr2=0x55e0f0561a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55e0f055f6a0 0x55e0f0561a90 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb33c034650 msgr2=0x7fb33c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb33c034650 0x7fb33c036b10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb46401dfd0 tx=0x7fb464037000 comp rx=0 tx=0).stop 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb472df5640 1 -- 172.21.15.110:0/3534579961 reap_dead start 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 msgr2=0x55e0f054abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f054abb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb46c012750 tx=0x7fb46c04a850 comp rx=0 tx=0).stop 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 shutdown_connections 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3534579961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0543720 0x55e0f054abb0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 >> 172.21.15.110:0/3534579961 conn(0x55e0f053fab0 msgr2=0x55e0f0544fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 shutdown_connections 2024-04-27T03:18:32.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3534579961 wait complete. 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb34c034650 msgr2=0x7fb34c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb34c034650 0x7fb34c036b10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55e0f05269c0 tx=0x7fb4640486d0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0536570 msgr2=0x55e0f0536990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0536570 0x55e0f0536990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00dfc0 tx=0x7fb46c0178e0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 shutdown_connections 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb34c034650 0x7fb34c036b10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/877153071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f0536570 0x55e0f0536990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 >> 172.21.15.110:0/877153071 conn(0x55e0f052b380 msgr2=0x55e0f052be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 shutdown_connections 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/877153071 wait complete. 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f05239a0 msgr2=0x55e0f0525d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f05239a0 0x55e0f0525d90 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f042d9c0 msgr2=0x55e0f05226b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f042d9c0 0x55e0f05226b0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb358034650 msgr2=0x7fb358036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb358034650 0x7fb358036b10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb4680ae000 tx=0x7fb4680a6000 comp rx=0 tx=0).stop 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 msgr2=0x55e0f051b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f051b610 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb464001fd0 tx=0x7fb464000bd0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 shutdown_connections 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55e0f05239a0 0x55e0f0525d90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55e0f042d9c0 0x55e0f05226b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb358034650 0x7fb358036b10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 --2- 172.21.15.110:0/3254696761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55e0f04294b0 0x55e0f051b610 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 >> 172.21.15.110:0/3254696761 conn(0x55e0f039cb90 msgr2=0x55e0f050ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 shutdown_connections 2024-04-27T03:18:32.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:18:32.840+0000 7fb475adb980 1 -- 172.21.15.110:0/3254696761 wait complete. 2024-04-27T03:18:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 17 2024-04-27T03:18:32.880 INFO:tasks.radosgw_admin:RL: create_bucket myfoo foo bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:18:32.887 INFO:tasks.radosgw_admin:RL: put_obj myfoo foo bytes_out=6 bytes_in=0 failed=False 2024-04-27T03:19:07.889 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj'] 2024-04-27T03:19:07.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--categories', 'delete_obj'] 2024-04-27T03:19:07.890 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --categories delete_obj 2024-04-27T03:19:07.929 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:07.930 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f33029980 1 Processor -- start 2024-04-27T03:19:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f33029980 1 -- start start 2024-04-27T03:19:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f33029980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5626e92649c0 0x5626e9266db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f33029980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5626e911d0d0 con 0x5626e92604b0 2024-04-27T03:19:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f33029980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5626e9112430 con 0x5626e92649c0 2024-04-27T03:19:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f30343640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5626e92649c0 0x5626e9266db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f30b44640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5626e92604b0 0x5626e9260890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36154/0 (socket says 172.21.15.110:36154) 2024-04-27T03:19:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.919+0000 7f4f30b44640 1 -- 172.21.15.110:0/3173870300 learned_addr learned my addr 172.21.15.110:0/3173870300 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f30343640 1 -- 172.21.15.110:0/3173870300 >> v1:172.21.15.77:6789/0 conn(0x5626e92604b0 legacy=0x5626e9260890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:07.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f30343640 1 -- 172.21.15.110:0/3173870300 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e910b510 con 0x5626e92649c0 2024-04-27T03:19:07.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f30343640 1 --2- 172.21.15.110:0/3173870300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9266db0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400a480 tx=0x7f4f2401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb919994b9b17841 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f2fb42640 1 -- 172.21.15.110:0/3173870300 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f240b0070 con 0x5626e92649c0 2024-04-27T03:19:07.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f2fb42640 1 -- 172.21.15.110:0/3173870300 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f240ab540 con 0x5626e92649c0 2024-04-27T03:19:07.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f2fb42640 1 -- 172.21.15.110:0/3173870300 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f240ab840 con 0x5626e92649c0 2024-04-27T03:19:07.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f33029980 1 -- 172.21.15.110:0/3173870300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 msgr2=0x5626e9266db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:07.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f33029980 1 --2- 172.21.15.110:0/3173870300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9266db0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400a480 tx=0x7f4f2401b280 comp rx=0 tx=0).stop 2024-04-27T03:19:07.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f33029980 1 -- 172.21.15.110:0/3173870300 shutdown_connections 2024-04-27T03:19:07.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f33029980 1 --2- 172.21.15.110:0/3173870300 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9266db0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:07.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f33029980 1 -- 172.21.15.110:0/3173870300 >> 172.21.15.110:0/3173870300 conn(0x5626e91d3b90 msgr2=0x5626e91d3fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:07.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f33029980 1 -- 172.21.15.110:0/3173870300 shutdown_connections 2024-04-27T03:19:07.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.923+0000 7f4f33029980 1 -- 172.21.15.110:0/3173870300 wait complete. 2024-04-27T03:19:07.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.939+0000 7f4f33029980 1 Processor -- start 2024-04-27T03:19:07.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.939+0000 7f4f33029980 1 -- start start 2024-04-27T03:19:07.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.939+0000 7f4f33029980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92604b0 0x5626e934b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.939+0000 7f4f33029980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626e91612d0 con 0x5626e92604b0 2024-04-27T03:19:07.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92604b0 0x5626e934b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92604b0 0x5626e934b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39888/0 (socket says 172.21.15.110:39888) 2024-04-27T03:19:07.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 -- 172.21.15.110:0/4093451915 learned_addr learned my addr 172.21.15.110:0/4093451915 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 -- 172.21.15.110:0/4093451915 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e90e7ae0 con 0x5626e92604b0 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 --2- 172.21.15.110:0/4093451915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92604b0 0x5626e934b680 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4f200017b0 tx=0x7f4f20001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82380f815425cf67 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4e1ee0d640 1 -- 172.21.15.110:0/4093451915 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f20016070 con 0x5626e92604b0 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4e1ee0d640 1 -- 172.21.15.110:0/4093451915 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f20004530 con 0x5626e92604b0 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4e1ee0d640 1 -- 172.21.15.110:0/4093451915 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f20004870 con 0x5626e92604b0 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 -- 172.21.15.110:0/4093451915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92604b0 msgr2=0x5626e934b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 --2- 172.21.15.110:0/4093451915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92604b0 0x5626e934b680 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4f200017b0 tx=0x7f4f20001c70 comp rx=0 tx=0).stop 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 -- 172.21.15.110:0/4093451915 shutdown_connections 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 --2- 172.21.15.110:0/4093451915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92604b0 0x5626e934b680 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 -- 172.21.15.110:0/4093451915 >> 172.21.15.110:0/4093451915 conn(0x5626e91d3b90 msgr2=0x5626e9347210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 -- 172.21.15.110:0/4093451915 shutdown_connections 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 -- 172.21.15.110:0/4093451915 wait complete. 2024-04-27T03:19:07.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 Processor -- start 2024-04-27T03:19:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 -- start start 2024-04-27T03:19:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9352920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f33029980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626e9269de0 con 0x5626e92649c0 2024-04-27T03:19:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9352920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9352920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39900/0 (socket says 172.21.15.110:39900) 2024-04-27T03:19:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.943+0000 7f4f30b44640 1 -- 172.21.15.110:0/3938943537 learned_addr learned my addr 172.21.15.110:0/3938943537 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f30b44640 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e9352e60 con 0x5626e92649c0 2024-04-27T03:19:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f30b44640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9352920 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4f20014040 tx=0x7f4f20001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4e1d60a640 1 -- 172.21.15.110:0/3938943537 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f20020020 con 0x5626e92649c0 2024-04-27T03:19:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4e1d60a640 1 -- 172.21.15.110:0/3938943537 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f20017040 con 0x5626e92649c0 2024-04-27T03:19:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5626e93530f0 con 0x5626e92649c0 2024-04-27T03:19:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4e1d60a640 1 -- 172.21.15.110:0/3938943537 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f20016070 con 0x5626e92649c0 2024-04-27T03:19:07.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e9355d70 con 0x5626e92649c0 2024-04-27T03:19:07.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4e1d60a640 1 -- 172.21.15.110:0/3938943537 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4f20023020 con 0x5626e92649c0 2024-04-27T03:19:07.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4e1d60a640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e18034650 0x7f4e18036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4e1d60a640 1 -- 172.21.15.110:0/3938943537 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f4f20048f40 con 0x5626e92649c0 2024-04-27T03:19:07.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f30343640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e18034650 0x7f4e18036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e9357840 0x5626e9359c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e935a1c0 con 0x5626e9357840 2024-04-27T03:19:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f31345640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e9357840 0x5626e9359c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f30343640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e18034650 0x7f4e18036b10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4f240ad000 tx=0x7f4f240a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f31345640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e9357840 0x5626e9359c30 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f31345640 1 -- 172.21.15.110:0/3938943537 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4f280083f0 con 0x5626e9357840 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e935afb0 0x5626e935d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.947+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e935d9e0 con 0x5626e935afb0 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f30b44640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e935afb0 0x5626e935d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f30b44640 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e935afb0 0x5626e935d3a0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f30b44640 1 -- 172.21.15.110:0/3938943537 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e935afb0 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e935f290 con 0x5626e9357840 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f31345640 1 -- 172.21.15.110:0/3938943537 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f4f280083f0 con 0x5626e9357840 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e935e4c0 con 0x5626e935afb0 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f30b44640 1 -- 172.21.15.110:0/3938943537 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e935afb0 2024-04-27T03:19:07.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e935e8b0 con 0x5626e9357840 2024-04-27T03:19:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f31345640 1 -- 172.21.15.110:0/3938943537 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f4f280083f0 con 0x5626e9357840 2024-04-27T03:19:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f33029980 1 Processor -- start 2024-04-27T03:19:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f33029980 1 -- start start 2024-04-27T03:19:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f33029980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e936dbd0 0x5626e936dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.951+0000 7f4f33029980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626e9165310 con 0x5626e936dbd0 2024-04-27T03:19:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f31345640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e936dbd0 0x5626e936dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f31345640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e936dbd0 0x5626e936dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39906/0 (socket says 172.21.15.110:39906) 2024-04-27T03:19:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f31345640 1 -- 172.21.15.110:0/104902325 learned_addr learned my addr 172.21.15.110:0/104902325 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f31345640 1 -- 172.21.15.110:0/104902325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e936e530 con 0x5626e936dbd0 2024-04-27T03:19:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f31345640 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e936dbd0 0x5626e936dff0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800aab0 tx=0x7f4f2800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4e167fc640 1 -- 172.21.15.110:0/104902325 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f28007d30 con 0x5626e936dbd0 2024-04-27T03:19:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4e167fc640 1 -- 172.21.15.110:0/104902325 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f2801d070 con 0x5626e936dbd0 2024-04-27T03:19:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5626e936efd0 con 0x5626e936dbd0 2024-04-27T03:19:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4e167fc640 1 -- 172.21.15.110:0/104902325 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f28018440 con 0x5626e936dbd0 2024-04-27T03:19:07.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e936f4a0 con 0x5626e936dbd0 2024-04-27T03:19:07.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4e167fc640 1 -- 172.21.15.110:0/104902325 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4f28013070 con 0x5626e936dbd0 2024-04-27T03:19:07.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4e167fc640 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e080346a0 0x7f4e08036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4e167fc640 1 -- 172.21.15.110:0/104902325 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f4f2804b650 con 0x5626e936dbd0 2024-04-27T03:19:07.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f30b44640 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e080346a0 0x7f4e08036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f33029980 1 Processor -- start 2024-04-27T03:19:07.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f33029980 1 -- start start 2024-04-27T03:19:07.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f30b44640 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e080346a0 0x7f4e08036b60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5626e935dfe0 tx=0x7f4f20030040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f33029980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e937b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.955+0000 7f4f33029980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626e934bdd0 con 0x5626e937adb0 2024-04-27T03:19:07.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e937b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e937b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39916/0 (socket says 172.21.15.110:39916) 2024-04-27T03:19:07.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 -- 172.21.15.110:0/3370309046 learned_addr learned my addr 172.21.15.110:0/3370309046 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:07.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 -- 172.21.15.110:0/3370309046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e937b7a0 con 0x5626e937adb0 2024-04-27T03:19:07.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 --2- 172.21.15.110:0/3370309046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e937b1d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4f280498a0 tx=0x7f4f28049b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710d057a57948b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4e07fff640 1 -- 172.21.15.110:0/3370309046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f28047070 con 0x5626e937adb0 2024-04-27T03:19:07.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4e07fff640 1 -- 172.21.15.110:0/3370309046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f28022460 con 0x5626e937adb0 2024-04-27T03:19:07.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4e07fff640 1 -- 172.21.15.110:0/3370309046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f28018af0 con 0x5626e937adb0 2024-04-27T03:19:07.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 -- 172.21.15.110:0/3370309046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 msgr2=0x5626e937b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:07.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 --2- 172.21.15.110:0/3370309046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e937b1d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4f280498a0 tx=0x7f4f28049b50 comp rx=0 tx=0).stop 2024-04-27T03:19:07.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 -- 172.21.15.110:0/3370309046 shutdown_connections 2024-04-27T03:19:07.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 --2- 172.21.15.110:0/3370309046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e937b1d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:07.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 -- 172.21.15.110:0/3370309046 >> 172.21.15.110:0/3370309046 conn(0x5626e93770f0 msgr2=0x5626e9375150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:07.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 -- 172.21.15.110:0/3370309046 shutdown_connections 2024-04-27T03:19:07.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 -- 172.21.15.110:0/3370309046 wait complete. 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 Processor -- start 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 -- start start 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e9382240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f33029980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5626e937bf00 con 0x5626e937adb0 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e9382240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e9382240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39922/0 (socket says 172.21.15.110:39922) 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 learned_addr learned my addr 172.21.15.110:0/2765972945 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e9382780 con 0x5626e937adb0 2024-04-27T03:19:07.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.959+0000 7f4f31345640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e9382240 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4f28048770 tx=0x7f4f28049830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4e067fc640 1 -- 172.21.15.110:0/2765972945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f28031070 con 0x5626e937adb0 2024-04-27T03:19:07.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4e067fc640 1 -- 172.21.15.110:0/2765972945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f28020900 con 0x5626e937adb0 2024-04-27T03:19:07.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5626e9382a10 con 0x5626e937adb0 2024-04-27T03:19:07.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4e067fc640 1 -- 172.21.15.110:0/2765972945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f2804b0d0 con 0x5626e937adb0 2024-04-27T03:19:07.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e93856f0 con 0x5626e937adb0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4e067fc640 1 -- 172.21.15.110:0/2765972945 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4f28034070 con 0x5626e937adb0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4e067fc640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4dfc034650 0x7f4dfc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4e067fc640 1 -- 172.21.15.110:0/2765972945 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f4f28037db0 con 0x5626e937adb0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f30b44640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4dfc034650 0x7f4dfc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e938d8c0 0x5626e938fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93904b0 con 0x5626e938d8c0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f30343640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e938d8c0 0x5626e938fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f30b44640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4dfc034650 0x7f4dfc036b10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000f880 tx=0x7f4f20039000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f30343640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e938d8c0 0x5626e938fd60 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 realm 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93908e0 con 0x5626e938d8c0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93911b0 con 0x5626e938d8c0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e9391bd0 0x5626e9393fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e9394730 con 0x5626e9391bd0 2024-04-27T03:19:07.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.963+0000 7f4f31345640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e9391bd0 0x5626e9393fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f31345640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e9391bd0 0x5626e9393fc0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f33029980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e9394fd0 con 0x5626e938d8c0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f33029980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f33029980 20 searching for the correct realm 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5626e9396d40 0x5626e9399130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9396d40 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f30b44640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5626e9396d40 0x5626e9399130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f30b44640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5626e9396d40 0x5626e9399130 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f31345640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4f28036070 0x7f4f28045a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x7f4f28036070 2024-04-27T03:19:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f30343640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4f28036070 0x7f4f28045a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.967+0000 7f4f30343640 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4f28036070 0x7f4f28045a80 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:07.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9396d40 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x7f4f28036070 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x7f4f28036070 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.971+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9396d40 2024-04-27T03:19:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9396d40 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x7f4f28036070 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9396d40 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x7f4f28036070 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.975+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e938d8c0 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x7f4f28036070 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9396d40 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x7f4f28036070 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9396d40 2024-04-27T03:19:07.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.979+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5626e9399710 con 0x5626e9391bd0 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e9395950 con 0x5626e938d8c0 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e9395dd0 con 0x5626e9391bd0 2024-04-27T03:19:07.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e9396230 con 0x5626e938d8c0 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 zone default found 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 4 Realm: () 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 zonegroup default 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e939a020 con 0x7f4f28036070 2024-04-27T03:19:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e939ba30 con 0x5626e9396d40 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados->read ofs=0 len=0 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e939bec0 con 0x5626e9396d40 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 started sync module instance, tier type = 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:07.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.983+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939ae90 con 0x5626e9391bd0 2024-04-27T03:19:07.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.987+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.987+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94725121618560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939aaf0 con 0x5626e9391bd0 2024-04-27T03:19:07.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.987+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939cdf0 con 0x5626e938d8c0 2024-04-27T03:19:07.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.987+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:07.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.987+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94725121624384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939d210 con 0x5626e938d8c0 2024-04-27T03:19:07.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.987+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939dcd0 con 0x5626e9391bd0 2024-04-27T03:19:07.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.987+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94725121618560] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:07.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94725121624384] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94725121628064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939cd60 con 0x5626e9391bd0 2024-04-27T03:19:08.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939e570 con 0x7f4f28036070 2024-04-27T03:19:08.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94725121630400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939cd60 con 0x7f4f28036070 2024-04-27T03:19:08.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939eff0 con 0x5626e9391bd0 2024-04-27T03:19:08.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94725121628064] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.991+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94725121630400] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94725121631456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939eb00 con 0x5626e9391bd0 2024-04-27T03:19:08.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939f890 con 0x5626e9396d40 2024-04-27T03:19:08.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94725121635296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939eb00 con 0x5626e9396d40 2024-04-27T03:19:08.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e93a0700 con 0x5626e9396d40 2024-04-27T03:19:08.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94725121631456] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94725121635296] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94725121638992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e93a0aa0 con 0x5626e9396d40 2024-04-27T03:19:08.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.995+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e93a1330 con 0x7f4f28036070 2024-04-27T03:19:08.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94725121638992] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'189 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94725121622816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e93a0f50 con 0x7f4f28036070 2024-04-27T03:19:08.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=0 2024-04-27T03:19:08.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=1 2024-04-27T03:19:08.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=2 2024-04-27T03:19:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=3 2024-04-27T03:19:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=4 2024-04-27T03:19:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=5 2024-04-27T03:19:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=6 2024-04-27T03:19:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94725121622816] v29'190 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 20 add_watcher() i=7 2024-04-27T03:19:08.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:08.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93a4810 con 0x5626e938d8c0 2024-04-27T03:19:08.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:07.999+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94725121641584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e93a1d30 con 0x5626e938d8c0 2024-04-27T03:19:08.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94725121641584] v29'202 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4de6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:08.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:08.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 5 note: GC not initialized 2024-04-27T03:19:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4de5fe3640 20 reqs_thread_entry: start 2024-04-27T03:19:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 20 Filter name: none 2024-04-27T03:19:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4d577fe640 20 reqs_thread_entry: start 2024-04-27T03:19:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.e 6:74cd889a:usage::usage.0:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c8410 con 0x5626e9391bd0 2024-04-27T03:19:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 usage.0 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:63 6.3 6:c0e78a64:usage::usage.1:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c8410 con 0x5626e9391bd0 2024-04-27T03:19:08.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(63 usage.1 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.003+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 6.d 6:b564efa4:usage::usage.2:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e9391bd0 2024-04-27T03:19:08.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(64 usage.2 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:65 6.2 6:42d64401:usage::usage.3:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e9396d40 2024-04-27T03:19:08.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(65 usage.3 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.7 6:e09b6f01:usage::usage.4:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e9391bd0 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(66 usage.4 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.6 6:672810f1:usage::usage.5:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e9396d40 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 usage.5 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 6.4 6:25a86931:usage::usage.6:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e938d8c0 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(68 usage.6 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.17 6:ebbe7322:usage::usage.7:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e938d8c0 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 usage.7 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 6.6 6:607e1922:usage::usage.8:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e9396d40 2024-04-27T03:19:08.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(70 usage.8 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 6.b 6:d7809252:usage::usage.9:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e9396d40 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(71 usage.9 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c83d0 con 0x5626e9396d40 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(72 usage.10 [call out=1268b] v0'0 uv29 ondisk = 0) ==== 152+0+1268 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:73 6.4 6:21f845ae:usage::usage.11:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e938d8c0 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(73 usage.11 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 6.f 6:f0386354:usage::usage.12:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x7f4f28036070 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 usage.12 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 6.c 6:3735eec4:usage::usage.13:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9396d40 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.007+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(75 usage.13 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 6.18 6:1f520d0a:usage::usage.14:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x7f4f28036070 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(76 usage.14 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:77 6.0 6:029cee12:usage::usage.15:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e938d8c0 2024-04-27T03:19:08.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(77 usage.15 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:78 6.1e 6:7f3e030e:usage::usage.16:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9396d40 2024-04-27T03:19:08.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(78 usage.16 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 6.10 6:0fbdcb32:usage::usage.17:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e938d8c0 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(79 usage.17 [call out=163b] v0'0 uv5 ondisk = 0) ==== 152+0+163 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.19 6:993b03e9:usage::usage.18:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9391bd0 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(80 usage.18 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 6.1a 6:58fcd270:usage::usage.19:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9391bd0 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(81 usage.19 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:82 6.1f 6:fcd63d49:usage::usage.20:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9396d40 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(82 usage.20 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.5 6:a2c1a412:usage::usage.21:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9391bd0 2024-04-27T03:19:08.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(83 usage.21 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.011+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.13 6:cdb71d68:usage::usage.22:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9396d40 2024-04-27T03:19:08.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(84 usage.22 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.17 6:ef3bca03:usage::usage.23:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e938d8c0 2024-04-27T03:19:08.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 usage.23 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:86 6.e 6:7412960a:usage::usage.24:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9391bd0 2024-04-27T03:19:08.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(86 usage.24 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:87 6.7 6:e62d1024:usage::usage.25:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9391bd0 2024-04-27T03:19:08.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(87 usage.25 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:88 6.d 6:b172cd59:usage::usage.26:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9391bd0 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(88 usage.26 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:89 6.1 6:83912023:usage::usage.27:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e938d8c0 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(89 usage.27 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:90 6.1e 6:7c73bdb3:usage::usage.28:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9396d40 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(90 usage.28 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:91 6.16 6:6e59fa35:usage::usage.29:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9396d40 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(91 usage.29 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:92 6.1c 6:38fe6885:usage::usage.30:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e9391bd0 2024-04-27T03:19:08.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(92 usage.30 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:93 6.1d 6:bb28dff0:usage::usage.31:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c9fe0 con 0x5626e938d8c0 2024-04-27T03:19:08.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.015+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(93 usage.31 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.019+0000 7f4de6fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.019+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94725121641584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e93c6630 con 0x5626e938d8c0 2024-04-27T03:19:08.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.019+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(94 data_loggenerations_metadata [watch unwatch cookie 94725121641584] v29'203 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.019+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94725121618560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e93a0c40 con 0x5626e9391bd0 2024-04-27T03:19:08.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.023+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(95 notify.0 [watch unwatch cookie 94725121618560] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.023+0000 7f4f33029980 20 remove_watcher() i=0 2024-04-27T03:19:08.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.023+0000 7f4f33029980 2 removed watcher, disabling cache 2024-04-27T03:19:08.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.023+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94725121624384] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939b280 con 0x5626e938d8c0 2024-04-27T03:19:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.023+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(96 notify.1 [watch unwatch cookie 94725121624384] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x5626e938d8c0 2024-04-27T03:19:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.023+0000 7f4f33029980 20 remove_watcher() i=1 2024-04-27T03:19:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.023+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94725121628064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939b280 con 0x5626e9391bd0 2024-04-27T03:19:08.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.027+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(97 notify.2 [watch unwatch cookie 94725121628064] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.027+0000 7f4f33029980 20 remove_watcher() i=2 2024-04-27T03:19:08.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.027+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94725121630400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939b280 con 0x7f4f28036070 2024-04-27T03:19:08.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.027+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(98 notify.3 [watch unwatch cookie 94725121630400] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.027+0000 7f4f33029980 20 remove_watcher() i=3 2024-04-27T03:19:08.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.027+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94725121631456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939e0c0 con 0x5626e9391bd0 2024-04-27T03:19:08.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f31345640 1 -- 172.21.15.110:0/2765972945 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(99 notify.4 [watch unwatch cookie 94725121631456] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f28053040 con 0x5626e9391bd0 2024-04-27T03:19:08.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f33029980 20 remove_watcher() i=4 2024-04-27T03:19:08.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94725121635296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939b280 con 0x5626e9396d40 2024-04-27T03:19:08.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(100 notify.5 [watch unwatch cookie 94725121635296] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f33029980 20 remove_watcher() i=5 2024-04-27T03:19:08.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94725121638992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939b280 con 0x5626e9396d40 2024-04-27T03:19:08.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f30b44640 1 -- 172.21.15.110:0/2765972945 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(101 notify.6 [watch unwatch cookie 94725121638992] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f2000f070 con 0x5626e9396d40 2024-04-27T03:19:08.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f33029980 20 remove_watcher() i=6 2024-04-27T03:19:08.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.031+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94725121622816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5626e939b280 con 0x7f4f28036070 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(102 notify.7 [watch unwatch cookie 94725121622816] v29'191 uv189 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4f240b3040 con 0x7f4f28036070 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 20 remove_watcher() i=7 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e9391bd0 msgr2=0x5626e9393fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e9391bd0 0x5626e9393fc0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e938d8c0 msgr2=0x5626e938fd60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e938d8c0 0x5626e938fd60 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4f28036070 msgr2=0x7f4f28045a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4f28036070 0x7f4f28045a80 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5626e9396d40 msgr2=0x5626e9399130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5626e9396d40 0x5626e9399130 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4dfc034650 msgr2=0x7f4dfc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4dfc034650 0x7f4dfc036b10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4f2000f880 tx=0x7f4f20039000 comp rx=0 tx=0).stop 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 msgr2=0x5626e9382240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e9382240 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4f28048770 tx=0x7f4f28049830 comp rx=0 tx=0).stop 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f30343640 1 -- 172.21.15.110:0/2765972945 reap_dead start 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 shutdown_connections 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/2765972945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e937adb0 0x5626e9382240 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 >> 172.21.15.110:0/2765972945 conn(0x5626e93770f0 msgr2=0x5626e9374750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 shutdown_connections 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/2765972945 wait complete. 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e080346a0 msgr2=0x7f4e08036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e080346a0 0x7f4e08036b60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5626e935dfe0 tx=0x7f4f20030040 comp rx=0 tx=0).stop 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e936dbd0 msgr2=0x5626e936dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e936dbd0 0x5626e936dff0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800aab0 tx=0x7f4f2800af70 comp rx=0 tx=0).stop 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 shutdown_connections 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e080346a0 0x7f4e08036b60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/104902325 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e936dbd0 0x5626e936dff0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 >> 172.21.15.110:0/104902325 conn(0x5626e93629e0 msgr2=0x5626e93634f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 shutdown_connections 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/104902325 wait complete. 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e935afb0 msgr2=0x5626e935d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e935afb0 0x5626e935d3a0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e9357840 msgr2=0x5626e9359c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e9357840 0x5626e9359c30 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e18034650 msgr2=0x7f4e18036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e18034650 0x7f4e18036b10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4f240ad000 tx=0x7f4f240a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 msgr2=0x5626e9352920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9352920 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4f20014040 tx=0x7f4f20001260 comp rx=0 tx=0).stop 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 shutdown_connections 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5626e935afb0 0x5626e935d3a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5626e9357840 0x5626e9359c30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f4e18034650 0x7f4e18036b10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 --2- 172.21.15.110:0/3938943537 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5626e92649c0 0x5626e9352920 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 >> 172.21.15.110:0/3938943537 conn(0x5626e91d3b90 msgr2=0x5626e9345d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 shutdown_connections 2024-04-27T03:19:08.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.035+0000 7f4f33029980 1 -- 172.21.15.110:0/3938943537 wait complete. 2024-04-27T03:19:08.056 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[{"user":"anonymous","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"anonymous","categories":[]}]},{"user":"foo","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4}]},{"bucket":"myfoo2","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo3","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo4","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]}]}],"summary":[{"user":"anonymous","categories":[],"total":{"bytes_sent":0,"bytes_received":0,"ops":0,"successful_ops":0}},{"user":"foo","categories":[{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4}],"total":{"bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4}}]} 2024-04-27T03:19:08.056 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'anonymous', 'categories': []}]}, {'user': 'foo', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}]}, {'bucket': 'myfoo2', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo3', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo4', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}]}], 'summary': [{'user': 'anonymous', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}}]} 2024-04-27T03:19:08.057 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show'] 2024-04-27T03:19:08.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show'] 2024-04-27T03:19:08.057 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show 2024-04-27T03:19:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9948f2f980 1 Processor -- start 2024-04-27T03:19:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9948f2f980 1 -- start start 2024-04-27T03:19:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9948f2f980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x562dba75b4a0 0x562dba75b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9948f2f980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x562dba6180d0 con 0x562dba75bdc0 2024-04-27T03:19:08.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9948f2f980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x562dba60d430 con 0x562dba75b4a0 2024-04-27T03:19:08.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9946a4a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x562dba75b4a0 0x562dba75b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9946249640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x562dba75bdc0 0x562dba75c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36170/0 (socket says 172.21.15.110:36170) 2024-04-27T03:19:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9946249640 1 -- 172.21.15.110:0/2548160207 learned_addr learned my addr 172.21.15.110:0/2548160207 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9945a48640 1 -- 172.21.15.110:0/2548160207 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1596977152 0 0) 0x562dba6180d0 con 0x562dba75bdc0 2024-04-27T03:19:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9945a48640 1 -- 172.21.15.110:0/2548160207 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9930001e20 con 0x562dba75bdc0 2024-04-27T03:19:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9946a4a640 1 -- 172.21.15.110:0/2548160207 >> v1:172.21.15.77:6789/0 conn(0x562dba75bdc0 legacy=0x562dba75c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.131+0000 7f9946a4a640 1 -- 172.21.15.110:0/2548160207 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dba606510 con 0x562dba75b4a0 2024-04-27T03:19:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9946a4a640 1 --2- 172.21.15.110:0/2548160207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba75b880 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f993800a480 tx=0x7f993801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97b9ab35de8646e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9945a48640 1 -- 172.21.15.110:0/2548160207 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99380b0070 con 0x562dba75b4a0 2024-04-27T03:19:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9945a48640 1 -- 172.21.15.110:0/2548160207 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99380ab540 con 0x562dba75b4a0 2024-04-27T03:19:08.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9945a48640 1 -- 172.21.15.110:0/2548160207 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99380ab840 con 0x562dba75b4a0 2024-04-27T03:19:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9948f2f980 1 -- 172.21.15.110:0/2548160207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 msgr2=0x562dba75b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9948f2f980 1 --2- 172.21.15.110:0/2548160207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba75b880 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f993800a480 tx=0x7f993801b280 comp rx=0 tx=0).stop 2024-04-27T03:19:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9948f2f980 1 -- 172.21.15.110:0/2548160207 shutdown_connections 2024-04-27T03:19:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9948f2f980 1 --2- 172.21.15.110:0/2548160207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba75b880 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9948f2f980 1 -- 172.21.15.110:0/2548160207 >> 172.21.15.110:0/2548160207 conn(0x562dba6ceb80 msgr2=0x562dba6cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9948f2f980 1 -- 172.21.15.110:0/2548160207 shutdown_connections 2024-04-27T03:19:08.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.135+0000 7f9948f2f980 1 -- 172.21.15.110:0/2548160207 wait complete. 2024-04-27T03:19:08.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9948f2f980 1 Processor -- start 2024-04-27T03:19:08.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9948f2f980 1 -- start start 2024-04-27T03:19:08.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9948f2f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba846b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9948f2f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562dba65c2d0 con 0x562dba75b4a0 2024-04-27T03:19:08.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9946a4a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba846b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9946a4a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba846b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39926/0 (socket says 172.21.15.110:39926) 2024-04-27T03:19:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9946a4a640 1 -- 172.21.15.110:0/4183258638 learned_addr learned my addr 172.21.15.110:0/4183258638 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9946a4a640 1 -- 172.21.15.110:0/4183258638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dba5e2ae0 con 0x562dba75b4a0 2024-04-27T03:19:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9946a4a640 1 --2- 172.21.15.110:0/4183258638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba846b60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f99380ae040 tx=0x7f9938003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d48d094f9ca52d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9834e09640 1 -- 172.21.15.110:0/4183258638 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99380b0040 con 0x562dba75b4a0 2024-04-27T03:19:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9834e09640 1 -- 172.21.15.110:0/4183258638 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99380ab9d0 con 0x562dba75b4a0 2024-04-27T03:19:08.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9834e09640 1 -- 172.21.15.110:0/4183258638 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99380abcd0 con 0x562dba75b4a0 2024-04-27T03:19:08.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9948f2f980 1 -- 172.21.15.110:0/4183258638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 msgr2=0x562dba846b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.151+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4183258638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba846b60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f99380ae040 tx=0x7f9938003030 comp rx=0 tx=0).stop 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- 172.21.15.110:0/4183258638 shutdown_connections 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4183258638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75b4a0 0x562dba846b60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- 172.21.15.110:0/4183258638 >> 172.21.15.110:0/4183258638 conn(0x562dba6ceb80 msgr2=0x562dba8426f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- 172.21.15.110:0/4183258638 shutdown_connections 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- 172.21.15.110:0/4183258638 wait complete. 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 Processor -- start 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- start start 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75bdc0 0x562dba84dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562dba7654f0 con 0x562dba75bdc0 2024-04-27T03:19:08.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9946a4a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75bdc0 0x562dba84dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9946a4a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75bdc0 0x562dba84dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39932/0 (socket says 172.21.15.110:39932) 2024-04-27T03:19:08.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9946a4a640 1 -- 172.21.15.110:0/4083355795 learned_addr learned my addr 172.21.15.110:0/4083355795 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9946a4a640 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dba84e210 con 0x562dba75bdc0 2024-04-27T03:19:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9946a4a640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75bdc0 0x562dba84dcd0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f99380b9040 tx=0x7f9938018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9833606640 1 -- 172.21.15.110:0/4083355795 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99380c3070 con 0x562dba75bdc0 2024-04-27T03:19:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9833606640 1 -- 172.21.15.110:0/4083355795 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99380b67b0 con 0x562dba75bdc0 2024-04-27T03:19:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562dba84e4a0 con 0x562dba75bdc0 2024-04-27T03:19:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9833606640 1 -- 172.21.15.110:0/4083355795 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99380b0070 con 0x562dba75bdc0 2024-04-27T03:19:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dba851030 con 0x562dba75bdc0 2024-04-27T03:19:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9833606640 1 -- 172.21.15.110:0/4083355795 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f99380c6070 con 0x562dba75bdc0 2024-04-27T03:19:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.155+0000 7f9833606640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f982c034650 0x7f982c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9833606640 1 -- 172.21.15.110:0/4083355795 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f99380bc070 con 0x562dba75bdc0 2024-04-27T03:19:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9946249640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f982c034650 0x7f982c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba852b00 0x562dba854ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8554c0 con 0x562dba852b00 2024-04-27T03:19:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f994724b640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba852b00 0x562dba854ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9946249640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f982c034650 0x7f982c036b10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f993c001dc0 tx=0x7f993c001360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f994724b640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba852b00 0x562dba854ef0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f994724b640 1 -- 172.21.15.110:0/4083355795 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f994000a4d0 con 0x562dba852b00 2024-04-27T03:19:08.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba8562b0 0x562dba8586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba858cd0 con 0x562dba8562b0 2024-04-27T03:19:08.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9946a4a640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba8562b0 0x562dba8586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9946a4a640 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba8562b0 0x562dba8586a0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9946a4a640 1 -- 172.21.15.110:0/4083355795 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f99380cf070 con 0x562dba8562b0 2024-04-27T03:19:08.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba85a580 con 0x562dba852b00 2024-04-27T03:19:08.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f994724b640 1 -- 172.21.15.110:0/4083355795 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f994000a4d0 con 0x562dba852b00 2024-04-27T03:19:08.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8597b0 con 0x562dba8562b0 2024-04-27T03:19:08.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.159+0000 7f9946a4a640 1 -- 172.21.15.110:0/4083355795 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f99380cf070 con 0x562dba8562b0 2024-04-27T03:19:08.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba859ba0 con 0x562dba852b00 2024-04-27T03:19:08.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f994724b640 1 -- 172.21.15.110:0/4083355795 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f994000a4d0 con 0x562dba852b00 2024-04-27T03:19:08.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9948f2f980 1 Processor -- start 2024-04-27T03:19:08.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9948f2f980 1 -- start start 2024-04-27T03:19:08.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9948f2f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba868e90 0x562dba8692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9948f2f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562dba660310 con 0x562dba868e90 2024-04-27T03:19:08.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f994724b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba868e90 0x562dba8692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f994724b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba868e90 0x562dba8692b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39944/0 (socket says 172.21.15.110:39944) 2024-04-27T03:19:08.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f994724b640 1 -- 172.21.15.110:0/156956297 learned_addr learned my addr 172.21.15.110:0/156956297 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f994724b640 1 -- 172.21.15.110:0/156956297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dba8697f0 con 0x562dba868e90 2024-04-27T03:19:08.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f994724b640 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba868e90 0x562dba8692b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f994000dfc0 tx=0x7f99400178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9830e01640 1 -- 172.21.15.110:0/156956297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99400157d0 con 0x562dba868e90 2024-04-27T03:19:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9830e01640 1 -- 172.21.15.110:0/156956297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9940015970 con 0x562dba868e90 2024-04-27T03:19:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9830e01640 1 -- 172.21.15.110:0/156956297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9940015c70 con 0x562dba868e90 2024-04-27T03:19:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562dba86a290 con 0x562dba868e90 2024-04-27T03:19:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.163+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dba86a760 con 0x562dba868e90 2024-04-27T03:19:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9830e01640 1 -- 172.21.15.110:0/156956297 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9940029440 con 0x562dba868e90 2024-04-27T03:19:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9830e01640 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9820034650 0x7f9820036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9830e01640 1 -- 172.21.15.110:0/156956297 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f994000e040 con 0x562dba868e90 2024-04-27T03:19:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9946a4a640 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9820034650 0x7f9820036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 Processor -- start 2024-04-27T03:19:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- start start 2024-04-27T03:19:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9946a4a640 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9820034650 0x7f9820036b10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x562dba8592d0 tx=0x7f99380e2800 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba876490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562dba8471f0 con 0x562dba876090 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f994724b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba876490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f994724b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba876490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39952/0 (socket says 172.21.15.110:39952) 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f994724b640 1 -- 172.21.15.110:0/1309146850 learned_addr learned my addr 172.21.15.110:0/1309146850 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f994724b640 1 -- 172.21.15.110:0/1309146850 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dba876a60 con 0x562dba876090 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f994724b640 1 --2- 172.21.15.110:0/1309146850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba876490 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9940015f40 tx=0x7f994004aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236cd7866fc0865e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f982a7fc640 1 -- 172.21.15.110:0/1309146850 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99400283a0 con 0x562dba876090 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f982a7fc640 1 -- 172.21.15.110:0/1309146850 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9940028540 con 0x562dba876090 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f982a7fc640 1 -- 172.21.15.110:0/1309146850 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f994001eb90 con 0x562dba876090 2024-04-27T03:19:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- 172.21.15.110:0/1309146850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 msgr2=0x562dba876490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 --2- 172.21.15.110:0/1309146850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba876490 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9940015f40 tx=0x7f994004aa00 comp rx=0 tx=0).stop 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- 172.21.15.110:0/1309146850 shutdown_connections 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 --2- 172.21.15.110:0/1309146850 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba876490 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- 172.21.15.110:0/1309146850 >> 172.21.15.110:0/1309146850 conn(0x562dba8723d0 msgr2=0x562dba870430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- 172.21.15.110:0/1309146850 shutdown_connections 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- 172.21.15.110:0/1309146850 wait complete. 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 Processor -- start 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- start start 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba87d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f9948f2f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x562dba877200 con 0x562dba876090 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.167+0000 7f994724b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba87d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f994724b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba87d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39960/0 (socket says 172.21.15.110:39960) 2024-04-27T03:19:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 learned_addr learned my addr 172.21.15.110:0/3996962940 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dba87da00 con 0x562dba876090 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f994724b640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba87d4c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f994004aa40 tx=0x7f994004a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9828ff9640 1 -- 172.21.15.110:0/3996962940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9940036070 con 0x562dba876090 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9828ff9640 1 -- 172.21.15.110:0/3996962940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f994004c240 con 0x562dba876090 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562dba87dc90 con 0x562dba876090 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9828ff9640 1 -- 172.21.15.110:0/3996962940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9940041040 con 0x562dba876090 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dba880970 con 0x562dba876090 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9828ff9640 1 -- 172.21.15.110:0/3996962940 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9940039070 con 0x562dba876090 2024-04-27T03:19:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9828ff9640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9810034650 0x7f9810036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9828ff9640 1 -- 172.21.15.110:0/3996962940 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f9940032d50 con 0x562dba876090 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9946a4a640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9810034650 0x7f9810036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba888b40 0x562dba88afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba88b720 con 0x562dba888b40 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9946249640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba888b40 0x562dba88afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9946a4a640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9810034650 0x7f9810036b10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9938003070 tx=0x7f99380de000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9946249640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba888b40 0x562dba88afd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f993c00e660 con 0x562dba888b40 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 20 realm 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba888050 con 0x562dba888b40 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.171+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f993c00e660 con 0x562dba888b40 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba88c1f0 con 0x562dba888b40 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f993c00e660 con 0x562dba888b40 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba88cdd0 0x562dba88f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba88fa00 con 0x562dba88cdd0 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f994724b640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba88cdd0 0x562dba88f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f994724b640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba88cdd0 0x562dba88f290 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8902a0 con 0x562dba888b40 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f993c00e660 con 0x562dba888b40 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 20 searching for the correct realm 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562dba892010 0x562dba894400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba892010 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9946a4a640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562dba892010 0x562dba894400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9946a4a640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562dba892010 0x562dba894400 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.175+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f994724b640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9940014030 0x7f9940020860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x7f9940014030 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9940014030 0x7f9940020860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9940014030 0x7f9940020860 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c00e660 con 0x7f9940014030 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c00e660 con 0x562dba888b40 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba892010 2024-04-27T03:19:08.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.179+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x7f9940014030 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x7f9940014030 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x7f9940014030 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f993c029040 con 0x7f9940014030 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba892010 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba892010 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.183+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x7f9940014030 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f993c029040 con 0x7f9940014030 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba892010 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x7f9940014030 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x7f9940014030 2024-04-27T03:19:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba888b40 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x7f9940014030 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x7f9940014030 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba892010 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x7f9940014030 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f993c029040 con 0x7f9940014030 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba892010 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.187+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x562dba894a10 con 0x562dba88cdd0 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba890c50 con 0x562dba888b40 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f993c029040 con 0x562dba888b40 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba891070 con 0x562dba88cdd0 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba891490 con 0x562dba888b40 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 zone default found 2024-04-27T03:19:08.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 4 Realm: () 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 zonegroup default 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba894e90 con 0x7f9940014030 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba890860 con 0x562dba892010 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba896de0 con 0x562dba892010 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 started sync module instance, tier type = 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.191+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8961b0 con 0x562dba88cdd0 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.195+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.195+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94754403083680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8976f0 con 0x562dba88cdd0 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.195+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba897f40 con 0x562dba888b40 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.195+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.195+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94754403088112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba898040 con 0x562dba888b40 2024-04-27T03:19:08.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.195+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba898b60 con 0x562dba88cdd0 2024-04-27T03:19:08.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.195+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94754403083680] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.199+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94754403088112] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.199+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.199+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94754403089168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba898b20 con 0x562dba88cdd0 2024-04-27T03:19:08.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.199+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba899400 con 0x7f9940014030 2024-04-27T03:19:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.199+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.199+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94754403094352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba898b20 con 0x7f9940014030 2024-04-27T03:19:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.199+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba899eb0 con 0x562dba88cdd0 2024-04-27T03:19:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94754403094352] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94754403089168] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94754403092016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba899eb0 con 0x562dba88cdd0 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba89a2a0 con 0x562dba892010 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94754403095408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba89aaf0 con 0x562dba892010 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba89b4f0 con 0x562dba892010 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94754403092016] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.203+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94754403095408] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94754403097024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba89a640 con 0x562dba892010 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba89bd90 con 0x7f9940014030 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94754403097024] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'192 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94754403102720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba89a640 con 0x7f9940014030 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=0 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=1 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=2 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=3 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=4 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=5 2024-04-27T03:19:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=6 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94754403102720] v29'193 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 20 add_watcher() i=7 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba89eeb0 con 0x562dba888b40 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.207+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94754403104992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba89c360 con 0x562dba888b40 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94754403104992] v29'204 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f97fd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 5 note: GC not initialized 2024-04-27T03:19:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9777fff640 20 reqs_thread_entry: start 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 20 Filter name: none 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9775ffb640 20 reqs_thread_entry: start 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.e 6:74cd889a:usage::usage.0:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2b30 con 0x562dba88cdd0 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 usage.0 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:63 6.3 6:c0e78a64:usage::usage.1:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2b30 con 0x562dba88cdd0 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(63 usage.1 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.211+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 6.d 6:b564efa4:usage::usage.2:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba88cdd0 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(64 usage.2 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:65 6.2 6:42d64401:usage::usage.3:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba892010 2024-04-27T03:19:08.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(65 usage.3 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.7 6:e09b6f01:usage::usage.4:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba88cdd0 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(66 usage.4 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.6 6:672810f1:usage::usage.5:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba892010 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 usage.5 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 6.4 6:25a86931:usage::usage.6:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba888b40 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(68 usage.6 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.17 6:ebbe7322:usage::usage.7:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba888b40 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 usage.7 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 6.6 6:607e1922:usage::usage.8:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba892010 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(70 usage.8 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 6.b 6:d7809252:usage::usage.9:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba892010 2024-04-27T03:19:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(71 usage.9 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba892010 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(72 usage.10 [call out=1268b] v0'0 uv29 ondisk = 0) ==== 152+0+1268 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:73 6.4 6:21f845ae:usage::usage.11:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4610 con 0x562dba888b40 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(73 usage.11 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 6.f 6:f0386354:usage::usage.12:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4610 con 0x7f9940014030 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 usage.12 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:75 6.c 6:3735eec4:usage::usage.13:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4610 con 0x562dba892010 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(75 usage.13 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.215+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:76 6.18 6:1f520d0a:usage::usage.14:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4610 con 0x7f9940014030 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(76 usage.14 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:77 6.0 6:029cee12:usage::usage.15:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4610 con 0x562dba888b40 2024-04-27T03:19:08.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(77 usage.15 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:78 6.1e 6:7f3e030e:usage::usage.16:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4610 con 0x562dba892010 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(78 usage.16 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 6.10 6:0fbdcb32:usage::usage.17:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4550 con 0x562dba888b40 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(79 usage.17 [call out=163b] v0'0 uv5 ondisk = 0) ==== 152+0+163 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.19 6:993b03e9:usage::usage.18:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4530 con 0x562dba88cdd0 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(80 usage.18 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 6.1a 6:58fcd270:usage::usage.19:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4530 con 0x562dba88cdd0 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(81 usage.19 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:82 6.1f 6:fcd63d49:usage::usage.20:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c44e0 con 0x562dba892010 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(82 usage.20 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.5 6:a2c1a412:usage::usage.21:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c44c0 con 0x562dba88cdd0 2024-04-27T03:19:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(83 usage.21 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.13 6:cdb71d68:usage::usage.22:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c44c0 con 0x562dba892010 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(84 usage.22 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.219+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.17 6:ef3bca03:usage::usage.23:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c44c0 con 0x562dba888b40 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 usage.23 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:86 6.e 6:7412960a:usage::usage.24:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c44c0 con 0x562dba88cdd0 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(86 usage.24 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:87 6.7 6:e62d1024:usage::usage.25:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4440 con 0x562dba88cdd0 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(87 usage.25 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:88 6.d 6:b172cd59:usage::usage.26:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4440 con 0x562dba88cdd0 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(88 usage.26 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:89 6.1 6:83912023:usage::usage.27:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4440 con 0x562dba888b40 2024-04-27T03:19:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(89 usage.27 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:90 6.1e 6:7c73bdb3:usage::usage.28:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4440 con 0x562dba892010 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(90 usage.28 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:91 6.16 6:6e59fa35:usage::usage.29:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4420 con 0x562dba892010 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(91 usage.29 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:92 6.1c 6:38fe6885:usage::usage.30:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4420 con 0x562dba88cdd0 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(92 usage.30 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:93 6.1d 6:bb28dff0:usage::usage.31:head [call rgw.user_usage_log_read in=60b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c4420 con 0x562dba888b40 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.223+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(93 usage.31 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.227+0000 7f97fd7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.227+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94754403104992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c0920 con 0x562dba888b40 2024-04-27T03:19:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.227+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(94 data_loggenerations_metadata [watch unwatch cookie 94754403104992] v29'205 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.227+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94754403083680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8bffc0 con 0x562dba88cdd0 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.231+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(95 notify.0 [watch unwatch cookie 94754403083680] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.231+0000 7f9948f2f980 20 remove_watcher() i=0 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.231+0000 7f9948f2f980 2 removed watcher, disabling cache 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.231+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94754403088112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba888b40 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.231+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(96 notify.1 [watch unwatch cookie 94754403088112] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x562dba888b40 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.231+0000 7f9948f2f980 20 remove_watcher() i=1 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.231+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94754403089168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba88cdd0 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.235+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(97 notify.2 [watch unwatch cookie 94754403089168] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.235+0000 7f9948f2f980 20 remove_watcher() i=2 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.235+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:98 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94754403094352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x7f9940014030 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.235+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(98 notify.3 [watch unwatch cookie 94754403094352] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.235+0000 7f9948f2f980 20 remove_watcher() i=3 2024-04-27T03:19:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.235+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94754403092016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba88cdd0 2024-04-27T03:19:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f994724b640 1 -- 172.21.15.110:0/3996962940 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(99 notify.4 [watch unwatch cookie 94754403092016] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f9940038030 con 0x562dba88cdd0 2024-04-27T03:19:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9948f2f980 20 remove_watcher() i=4 2024-04-27T03:19:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94754403095408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba892010 2024-04-27T03:19:08.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(100 notify.5 [watch unwatch cookie 94754403095408] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9948f2f980 20 remove_watcher() i=5 2024-04-27T03:19:08.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94754403097024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x562dba892010 2024-04-27T03:19:08.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9946a4a640 1 -- 172.21.15.110:0/3996962940 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(101 notify.6 [watch unwatch cookie 94754403097024] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f99380cf070 con 0x562dba892010 2024-04-27T03:19:08.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9948f2f980 20 remove_watcher() i=6 2024-04-27T03:19:08.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94754403102720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x562dba8c2af0 con 0x7f9940014030 2024-04-27T03:19:08.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(102 notify.7 [watch unwatch cookie 94754403102720] v29'194 uv192 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f993c03c1c0 con 0x7f9940014030 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.239+0000 7f9948f2f980 20 remove_watcher() i=7 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba88cdd0 msgr2=0x562dba88f290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba88cdd0 0x562dba88f290 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba888b40 msgr2=0x562dba88afd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba888b40 0x562dba88afd0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9940014030 msgr2=0x7f9940020860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9940014030 0x7f9940020860 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562dba892010 msgr2=0x562dba894400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x562dba892010 0x562dba894400 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9810034650 msgr2=0x7f9810036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9810034650 0x7f9810036b10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9938003070 tx=0x7f99380de000 comp rx=0 tx=0).stop 2024-04-27T03:19:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 msgr2=0x562dba87d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba87d4c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f994004aa40 tx=0x7f994004a850 comp rx=0 tx=0).stop 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9946249640 1 -- 172.21.15.110:0/3996962940 reap_dead start 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 shutdown_connections 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/3996962940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba876090 0x562dba87d4c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 >> 172.21.15.110:0/3996962940 conn(0x562dba8723d0 msgr2=0x562dba86fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 shutdown_connections 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/3996962940 wait complete. 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9820034650 msgr2=0x7f9820036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9820034650 0x7f9820036b10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x562dba8592d0 tx=0x7f99380e2800 comp rx=0 tx=0).stop 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba868e90 msgr2=0x562dba8692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba868e90 0x562dba8692b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f994000dfc0 tx=0x7f99400178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 shutdown_connections 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9820034650 0x7f9820036b10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/156956297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba868e90 0x562dba8692b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 >> 172.21.15.110:0/156956297 conn(0x562dba85dca0 msgr2=0x562dba85e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 shutdown_connections 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/156956297 wait complete. 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba8562b0 msgr2=0x562dba8586a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba8562b0 0x562dba8586a0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba852b00 msgr2=0x562dba854ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba852b00 0x562dba854ef0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f982c034650 msgr2=0x7f982c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f982c034650 0x7f982c036b10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f993c001dc0 tx=0x7f993c001360 comp rx=0 tx=0).stop 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75bdc0 msgr2=0x562dba84dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75bdc0 0x562dba84dcd0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f99380b9040 tx=0x7f9938018020 comp rx=0 tx=0).stop 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 shutdown_connections 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x562dba8562b0 0x562dba8586a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x562dba852b00 0x562dba854ef0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f982c034650 0x7f982c036b10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 --2- 172.21.15.110:0/4083355795 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x562dba75bdc0 0x562dba84dcd0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 >> 172.21.15.110:0/4083355795 conn(0x562dba6ceb80 msgr2=0x562dba841200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 shutdown_connections 2024-04-27T03:19:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.243+0000 7f9948f2f980 1 -- 172.21.15.110:0/4083355795 wait complete. 2024-04-27T03:19:08.263 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[{"user":"anonymous","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"anonymous","categories":[{"category":"list_buckets","bytes_sent":204,"bytes_received":0,"ops":1,"successful_ops":1}]}]},{"user":"foo","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"list_buckets","bytes_sent":322,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":5,"successful_ops":5},{"category":"delete_bucket","bytes_sent":236,"bytes_received":0,"ops":2,"successful_ops":1},{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4},{"category":"get_acls","bytes_sent":1026,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"get_obj","bytes_sent":4,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":515,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"put_acls","bytes_sent":0,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"put_obj","bytes_sent":0,"bytes_received":30000042,"ops":11,"successful_ops":11}]},{"bucket":"myfoo2","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo3","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo4","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"anonymous","categories":[{"category":"list_buckets","bytes_sent":204,"bytes_received":0,"ops":1,"successful_ops":1}],"total":{"bytes_sent":204,"bytes_received":0,"ops":1,"successful_ops":1}},{"user":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":8,"successful_ops":8},{"category":"delete_bucket","bytes_sent":236,"bytes_received":0,"ops":5,"successful_ops":4},{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4},{"category":"get_acls","bytes_sent":1026,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"get_obj","bytes_sent":4,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":515,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_buckets","bytes_sent":322,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"put_acls","bytes_sent":0,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"put_obj","bytes_sent":0,"bytes_received":30000042,"ops":11,"successful_ops":11}],"total":{"bytes_sent":2103,"bytes_received":30000042,"ops":35,"successful_ops":34}}]} 2024-04-27T03:19:08.263 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 204, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'foo', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 5, 'successful_ops': 5}, {'category': 'delete_bucket', 'bytes_sent': 236, 'bytes_received': 0, 'ops': 2, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}]}, {'bucket': 'myfoo2', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 204, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 204, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}}, {'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}, {'category': 'delete_bucket', 'bytes_sent': 236, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}], 'total': {'bytes_sent': 2103, 'bytes_received': 30000042, 'ops': 35, 'successful_ops': 34}}]} 2024-04-27T03:19:08.264 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo'] 2024-04-27T03:19:08.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo'] 2024-04-27T03:19:08.264 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo 2024-04-27T03:19:08.342 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:08.342 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:08.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff503808980 1 Processor -- start 2024-04-27T03:19:08.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff503808980 1 -- start start 2024-04-27T03:19:08.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff503808980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d615511150 0x55d615511530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff503808980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55d6153d10d0 con 0x55d615511a70 2024-04-27T03:19:08.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff503808980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55d6153c6430 con 0x55d615511150 2024-04-27T03:19:08.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff501323640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d615511150 0x55d615511530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff501323640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d615511150 0x55d615511530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39968/0 (socket says 172.21.15.110:39968) 2024-04-27T03:19:08.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff500b22640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55d615511a70 0x55d615512480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36184/0 (socket says 172.21.15.110:36184) 2024-04-27T03:19:08.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff501323640 1 -- 172.21.15.110:0/3556561166 learned_addr learned my addr 172.21.15.110:0/3556561166 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff501323640 1 -- 172.21.15.110:0/3556561166 >> v1:172.21.15.77:6789/0 conn(0x55d615511a70 legacy=0x55d615512480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.335+0000 7ff501323640 1 -- 172.21.15.110:0/3556561166 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6153bf510 con 0x55d615511150 2024-04-27T03:19:08.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff501323640 1 --2- 172.21.15.110:0/3556561166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615511530 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400a5c0 tx=0x7ff4f401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e52bb863a23ec server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff4f3fff640 1 -- 172.21.15.110:0/3556561166 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4f40a6070 con 0x55d615511150 2024-04-27T03:19:08.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff4f3fff640 1 -- 172.21.15.110:0/3556561166 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f40af070 con 0x55d615511150 2024-04-27T03:19:08.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff4f3fff640 1 -- 172.21.15.110:0/3556561166 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4f40aa5c0 con 0x55d615511150 2024-04-27T03:19:08.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff503808980 1 -- 172.21.15.110:0/3556561166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 msgr2=0x55d615511530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff503808980 1 --2- 172.21.15.110:0/3556561166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615511530 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400a5c0 tx=0x7ff4f401b340 comp rx=0 tx=0).stop 2024-04-27T03:19:08.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff503808980 1 -- 172.21.15.110:0/3556561166 shutdown_connections 2024-04-27T03:19:08.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff503808980 1 --2- 172.21.15.110:0/3556561166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615511530 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff503808980 1 -- 172.21.15.110:0/3556561166 >> 172.21.15.110:0/3556561166 conn(0x55d61550ca20 msgr2=0x55d61550ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff503808980 1 -- 172.21.15.110:0/3556561166 shutdown_connections 2024-04-27T03:19:08.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.339+0000 7ff503808980 1 -- 172.21.15.110:0/3556561166 wait complete. 2024-04-27T03:19:08.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 Processor -- start 2024-04-27T03:19:08.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 -- start start 2024-04-27T03:19:08.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d6155fdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d6154152d0 con 0x55d615511150 2024-04-27T03:19:08.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff501323640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d6155fdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff501323640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d6155fdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39980/0 (socket says 172.21.15.110:39980) 2024-04-27T03:19:08.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff501323640 1 -- 172.21.15.110:0/3022998180 learned_addr learned my addr 172.21.15.110:0/3022998180 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff501323640 1 -- 172.21.15.110:0/3022998180 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d61539bae0 con 0x55d615511150 2024-04-27T03:19:08.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff501323640 1 --2- 172.21.15.110:0/3022998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d6155fdac0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40199f0 tx=0x7ff4f4003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23d2765d0da6019 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff3ef606640 1 -- 172.21.15.110:0/3022998180 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4f40aaa00 con 0x55d615511150 2024-04-27T03:19:08.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff3ef606640 1 -- 172.21.15.110:0/3022998180 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f40aaba0 con 0x55d615511150 2024-04-27T03:19:08.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff3ef606640 1 -- 172.21.15.110:0/3022998180 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4f40bd3f0 con 0x55d615511150 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 -- 172.21.15.110:0/3022998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 msgr2=0x55d6155fdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 --2- 172.21.15.110:0/3022998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d6155fdac0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40199f0 tx=0x7ff4f4003030 comp rx=0 tx=0).stop 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 -- 172.21.15.110:0/3022998180 shutdown_connections 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 --2- 172.21.15.110:0/3022998180 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d6155fdac0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 -- 172.21.15.110:0/3022998180 >> 172.21.15.110:0/3022998180 conn(0x55d61550ca20 msgr2=0x55d6154861a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 -- 172.21.15.110:0/3022998180 shutdown_connections 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.355+0000 7ff503808980 1 -- 172.21.15.110:0/3022998180 wait complete. 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 Processor -- start 2024-04-27T03:19:08.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 -- start start 2024-04-27T03:19:08.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615604a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d61551b1f0 con 0x55d615511150 2024-04-27T03:19:08.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff501323640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615604a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff501323640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615604a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:39988/0 (socket says 172.21.15.110:39988) 2024-04-27T03:19:08.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff501323640 1 -- 172.21.15.110:0/1608586541 learned_addr learned my addr 172.21.15.110:0/1608586541 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff501323640 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d615604f60 con 0x55d615511150 2024-04-27T03:19:08.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff501323640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615604a20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff4f401b170 tx=0x7ff4f40180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff3ede03640 1 -- 172.21.15.110:0/1608586541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4f40bd7c0 con 0x55d615511150 2024-04-27T03:19:08.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff3ede03640 1 -- 172.21.15.110:0/1608586541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f40bd960 con 0x55d615511150 2024-04-27T03:19:08.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d6156051f0 con 0x55d615511150 2024-04-27T03:19:08.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff3ede03640 1 -- 172.21.15.110:0/1608586541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4f40bdc40 con 0x55d615511150 2024-04-27T03:19:08.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d615607e70 con 0x55d615511150 2024-04-27T03:19:08.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff3ede03640 1 -- 172.21.15.110:0/1608586541 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff4f40ce020 con 0x55d615511150 2024-04-27T03:19:08.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff3ede03640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3e8034650 0x7ff3e8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff3ede03640 1 -- 172.21.15.110:0/1608586541 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7ff4f40cc020 con 0x55d615511150 2024-04-27T03:19:08.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff500b22640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3e8034650 0x7ff3e8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615511a70 0x55d61560b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.359+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61560bf20 con 0x55d615511a70 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501b24640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615511a70 0x55d61560b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff500b22640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3e8034650 0x7ff3e8036b10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8001e50 tx=0x7ff4f80013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501b24640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615511a70 0x55d61560b950 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501b24640 1 -- 172.21.15.110:0/1608586541 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff4fc00a4d0 con 0x55d615511a70 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d61560ccd0 0x55d61560f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61560f720 con 0x55d61560ccd0 2024-04-27T03:19:08.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501323640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d61560ccd0 0x55d61560f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501323640 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d61560ccd0 0x55d61560f0c0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501323640 1 -- 172.21.15.110:0/1608586541 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61560ccd0 2024-04-27T03:19:08.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615610fd0 con 0x55d615511a70 2024-04-27T03:19:08.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501b24640 1 -- 172.21.15.110:0/1608586541 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7ff4fc00a4d0 con 0x55d615511a70 2024-04-27T03:19:08.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615610200 con 0x55d61560ccd0 2024-04-27T03:19:08.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501323640 1 -- 172.21.15.110:0/1608586541 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61560ccd0 2024-04-27T03:19:08.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d6156105f0 con 0x55d615511a70 2024-04-27T03:19:08.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.363+0000 7ff501b24640 1 -- 172.21.15.110:0/1608586541 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7ff4fc00a4d0 con 0x55d615511a70 2024-04-27T03:19:08.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff503808980 1 Processor -- start 2024-04-27T03:19:08.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff503808980 1 -- start start 2024-04-27T03:19:08.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff503808980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d61561f940 0x55d61561fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff503808980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d615419310 con 0x55d61561f940 2024-04-27T03:19:08.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff501b24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d61561f940 0x55d61561fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff501b24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d61561f940 0x55d61561fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40000/0 (socket says 172.21.15.110:40000) 2024-04-27T03:19:08.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff501b24640 1 -- 172.21.15.110:0/2174165098 learned_addr learned my addr 172.21.15.110:0/2174165098 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff501b24640 1 -- 172.21.15.110:0/2174165098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6156202a0 con 0x55d61561f940 2024-04-27T03:19:08.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff501b24640 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d61561f940 0x55d61561fd60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00df90 tx=0x7ff4fc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff3e6ffd640 1 -- 172.21.15.110:0/2174165098 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4fc0157d0 con 0x55d61561f940 2024-04-27T03:19:08.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff3e6ffd640 1 -- 172.21.15.110:0/2174165098 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4fc015970 con 0x55d61561f940 2024-04-27T03:19:08.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d615620d40 con 0x55d61561f940 2024-04-27T03:19:08.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff3e6ffd640 1 -- 172.21.15.110:0/2174165098 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4fc015c50 con 0x55d61561f940 2024-04-27T03:19:08.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d615621210 con 0x55d61561f940 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff3e6ffd640 1 -- 172.21.15.110:0/2174165098 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff4fc029440 con 0x55d61561f940 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff3e6ffd640 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3d8034640 0x7ff3d8036b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff3e6ffd640 1 -- 172.21.15.110:0/2174165098 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7ff4fc00e040 con 0x55d61561f940 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.367+0000 7ff501323640 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3d8034640 0x7ff3d8036b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 Processor -- start 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- start start 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501323640 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3d8034640 0x7ff3d8036b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55d61560c770 tx=0x7ff4f40e3820 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d6155f9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d6155fe150 con 0x55d6155f97d0 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d6155f9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d6155f9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40002/0 (socket says 172.21.15.110:40002) 2024-04-27T03:19:08.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 -- 172.21.15.110:0/451073570 learned_addr learned my addr 172.21.15.110:0/451073570 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 -- 172.21.15.110:0/451073570 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6155f8000 con 0x55d6155f97d0 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 --2- 172.21.15.110:0/451073570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d6155f9bf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc01e390 tx=0x7ff4fc01e480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ae846026320c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff3e4ff9640 1 -- 172.21.15.110:0/451073570 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4fc012440 con 0x55d6155f97d0 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff3e4ff9640 1 -- 172.21.15.110:0/451073570 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4fc041070 con 0x55d6155f97d0 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff3e4ff9640 1 -- 172.21.15.110:0/451073570 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4fc029c90 con 0x55d6155f97d0 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- 172.21.15.110:0/451073570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 msgr2=0x55d6155f9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 --2- 172.21.15.110:0/451073570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d6155f9bf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc01e390 tx=0x7ff4fc01e480 comp rx=0 tx=0).stop 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- 172.21.15.110:0/451073570 shutdown_connections 2024-04-27T03:19:08.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 --2- 172.21.15.110:0/451073570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d6155f9bf0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- 172.21.15.110:0/451073570 >> 172.21.15.110:0/451073570 conn(0x55d615628e80 msgr2=0x55d615626ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- 172.21.15.110:0/451073570 shutdown_connections 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- 172.21.15.110:0/451073570 wait complete. 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 Processor -- start 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- start start 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d615636210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff503808980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d6155f84d0 con 0x55d6155f97d0 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d615636210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d615636210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40014/0 (socket says 172.21.15.110:40014) 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.371+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 learned_addr learned my addr 172.21.15.110:0/4137552257 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d615636750 con 0x55d6155f97d0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff501b24640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d615636210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc04a7f0 tx=0x7ff4fc04a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff3d6ffd640 1 -- 172.21.15.110:0/4137552257 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4fc0491d0 con 0x55d6155f97d0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff3d6ffd640 1 -- 172.21.15.110:0/4137552257 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4fc049390 con 0x55d6155f97d0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d6156369e0 con 0x55d6155f97d0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff3d6ffd640 1 -- 172.21.15.110:0/4137552257 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4fc04c070 con 0x55d6155f97d0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6156396c0 con 0x55d6155f97d0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff3d6ffd640 1 -- 172.21.15.110:0/4137552257 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff4fc03a020 con 0x55d6155f97d0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff3d6ffd640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3cc0346a0 0x7ff3cc036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff3d6ffd640 1 -- 172.21.15.110:0/4137552257 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7ff4fc038020 con 0x55d6155f97d0 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff501323640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3cc0346a0 0x7ff3cc036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615641680 0x55d615643b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615644260 con 0x55d615641680 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff500b22640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615641680 0x55d615643b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff501323640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3cc0346a0 0x7ff3cc036b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55d615608fb0 tx=0x7ff4f40e9000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff500b22640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615641680 0x55d615643b10 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff4f800f6a0 con 0x55d615641680 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 realm 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615640b90 con 0x55d615641680 2024-04-27T03:19:08.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff4f800f6a0 con 0x55d615641680 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615644d30 con 0x55d615641680 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff4f800f6a0 con 0x55d615641680 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d615645900 0x55d615647dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.375+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615648530 con 0x55d615645900 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501b24640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d615645900 0x55d615647dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501b24640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d615645900 0x55d615647dc0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff503808980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615648dd0 con 0x55d615641680 2024-04-27T03:19:08.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7ff4f800f6a0 con 0x55d615641680 2024-04-27T03:19:08.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff503808980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:08.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff503808980 20 searching for the correct realm 2024-04-27T03:19:08.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d61564ab40 0x55d61564cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d61564ab40 2024-04-27T03:19:08.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501323640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d61564ab40 0x55d61564cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501323640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d61564ab40 0x55d61564cf30 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501b24640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff4fc033070 0x7ff4fc03c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x7ff4fc033070 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.379+0000 7ff500b22640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff4fc033070 0x7ff4fc03c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff4fc033070 0x7ff4fc03c580 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f800f6a0 con 0x7ff4fc033070 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f800f6a0 con 0x55d615641680 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d61564ab40 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x7ff4fc033070 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x7ff4fc033070 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.383+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x7ff4fc033070 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7ff4f802b040 con 0x7ff4fc033070 2024-04-27T03:19:08.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d61564ab40 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d61564ab40 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x7ff4fc033070 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7ff4f802b040 con 0x7ff4fc033070 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.387+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d61564ab40 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x7ff4fc033070 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x7ff4fc033070 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615641680 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f802b040 con 0x55d615641680 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x7ff4fc033070 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d61564ab40 2024-04-27T03:19:08.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x7ff4fc033070 2024-04-27T03:19:08.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d61564ab40 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.391+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55d61564d510 con 0x55d615645900 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:08.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615649750 con 0x55d615641680 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615649bd0 con 0x55d615645900 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61564a030 con 0x55d615641680 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 zone default found 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 4 Realm: () 2024-04-27T03:19:08.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 zonegroup default 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61564de40 con 0x7ff4fc033070 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61564f820 con 0x55d61564ab40 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61564fd40 con 0x55d61564ab40 2024-04-27T03:19:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 started sync module instance, tier type = 2024-04-27T03:19:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:08.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.395+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d61564ecb0 con 0x55d615645900 2024-04-27T03:19:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94377970299040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d61564e910 con 0x55d615645900 2024-04-27T03:19:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615650c70 con 0x55d615641680 2024-04-27T03:19:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94377970304960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615651090 con 0x55d615641680 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615651ae0 con 0x55d615645900 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94377970299040] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.399+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94377970304960] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94377970308560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615650be0 con 0x55d615645900 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615652380 con 0x7ff4fc033070 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94377970310864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615650be0 con 0x7ff4fc033070 2024-04-27T03:19:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615652e40 con 0x55d615645900 2024-04-27T03:19:08.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94377970308560] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94377970310864] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94377970311920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615652e00 con 0x55d615645900 2024-04-27T03:19:08.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.403+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615653230 con 0x55d61564ab40 2024-04-27T03:19:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94377970303392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615653ad0 con 0x55d61564ab40 2024-04-27T03:19:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615654490 con 0x55d61564ab40 2024-04-27T03:19:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94377970311920] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94377970303392] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94377970319328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615653ad0 con 0x55d61564ab40 2024-04-27T03:19:08.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.407+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615654480 con 0x7ff4fc033070 2024-04-27T03:19:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94377970319328] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'195 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94377970316832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615653a40 con 0x7ff4fc033070 2024-04-27T03:19:08.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=0 2024-04-27T03:19:08.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=1 2024-04-27T03:19:08.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=2 2024-04-27T03:19:08.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=3 2024-04-27T03:19:08.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=4 2024-04-27T03:19:08.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=5 2024-04-27T03:19:08.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=6 2024-04-27T03:19:08.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94377970316832] v29'196 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 20 add_watcher() i=7 2024-04-27T03:19:08.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:08.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d615658260 con 0x55d615641680 2024-04-27T03:19:08.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.411+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94377970322736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d6156556e0 con 0x55d615641680 2024-04-27T03:19:08.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94377970322736] v29'206 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff3b7fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 5 note: GC not initialized 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff3b6fe5640 20 reqs_thread_entry: start 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 20 Filter name: none 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff3b4fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61567b610 con 0x55d615641680 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55d61567cbe0 con 0x55d61564ab40 2024-04-27T03:19:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 usage.10 [call out=1268b] v0'0 uv29 ondisk = 0) ==== 152+0+1268 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff3b7fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.415+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94377970322736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d61563b850 con 0x55d615641680 2024-04-27T03:19:08.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.419+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 data_loggenerations_metadata [watch unwatch cookie 94377970322736] v29'207 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.419+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94377970299040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615654d30 con 0x55d615645900 2024-04-27T03:19:08.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.419+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(65 notify.0 [watch unwatch cookie 94377970299040] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.419+0000 7ff503808980 20 remove_watcher() i=0 2024-04-27T03:19:08.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.419+0000 7ff503808980 2 removed watcher, disabling cache 2024-04-27T03:19:08.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.419+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94377970304960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615654d30 con 0x55d615641680 2024-04-27T03:19:08.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.423+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 notify.1 [watch unwatch cookie 94377970304960] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x55d615641680 2024-04-27T03:19:08.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.423+0000 7ff503808980 20 remove_watcher() i=1 2024-04-27T03:19:08.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.423+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94377970308560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d6156507c0 con 0x55d615645900 2024-04-27T03:19:08.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.423+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [watch unwatch cookie 94377970308560] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.423+0000 7ff503808980 20 remove_watcher() i=2 2024-04-27T03:19:08.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.423+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:68 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94377970310864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615654d30 con 0x7ff4fc033070 2024-04-27T03:19:08.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(68 notify.3 [watch unwatch cookie 94377970310864] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff503808980 20 remove_watcher() i=3 2024-04-27T03:19:08.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94377970311920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d6156515d0 con 0x55d615645900 2024-04-27T03:19:08.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff501b24640 1 -- 172.21.15.110:0/4137552257 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.4 [watch unwatch cookie 94377970311920] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4fc026020 con 0x55d615645900 2024-04-27T03:19:08.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff503808980 20 remove_watcher() i=4 2024-04-27T03:19:08.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94377970303392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d6156507c0 con 0x55d61564ab40 2024-04-27T03:19:08.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.5 [watch unwatch cookie 94377970303392] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff503808980 20 remove_watcher() i=5 2024-04-27T03:19:08.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.427+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94377970319328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d615654d30 con 0x55d61564ab40 2024-04-27T03:19:08.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff501323640 1 -- 172.21.15.110:0/4137552257 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(71 notify.6 [watch unwatch cookie 94377970319328] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f40c6cd0 con 0x55d61564ab40 2024-04-27T03:19:08.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 20 remove_watcher() i=6 2024-04-27T03:19:08.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94377970316832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55d6156507c0 con 0x7ff4fc033070 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(72 notify.7 [watch unwatch cookie 94377970316832] v29'197 uv195 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff4f803e040 con 0x7ff4fc033070 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 20 remove_watcher() i=7 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d615645900 msgr2=0x55d615647dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d615645900 0x55d615647dc0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615641680 msgr2=0x55d615643b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615641680 0x55d615643b10 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff4fc033070 msgr2=0x7ff4fc03c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff4fc033070 0x7ff4fc03c580 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d61564ab40 msgr2=0x55d61564cf30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d61564ab40 0x55d61564cf30 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3cc0346a0 msgr2=0x7ff3cc036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3cc0346a0 0x7ff3cc036b60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55d615608fb0 tx=0x7ff4f40e9000 comp rx=0 tx=0).stop 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff500b22640 1 -- 172.21.15.110:0/4137552257 reap_dead start 2024-04-27T03:19:08.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 msgr2=0x55d615636210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d615636210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc04a7f0 tx=0x7ff4fc04a820 comp rx=0 tx=0).stop 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 shutdown_connections 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/4137552257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d6155f97d0 0x55d615636210 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 >> 172.21.15.110:0/4137552257 conn(0x55d615628e80 msgr2=0x55d6156264e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 shutdown_connections 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/4137552257 wait complete. 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3d8034640 msgr2=0x7ff3d8036b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3d8034640 0x7ff3d8036b00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55d61560c770 tx=0x7ff4f40e3820 comp rx=0 tx=0).stop 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d61561f940 msgr2=0x55d61561fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d61561f940 0x55d61561fd60 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00df90 tx=0x7ff4fc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 shutdown_connections 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3d8034640 0x7ff3d8036b00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/2174165098 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d61561f940 0x55d61561fd60 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 >> 172.21.15.110:0/2174165098 conn(0x55d615614750 msgr2=0x55d615615260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 shutdown_connections 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/2174165098 wait complete. 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d61560ccd0 msgr2=0x55d61560f0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d61560ccd0 0x55d61560f0c0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615511a70 msgr2=0x55d61560b950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615511a70 0x55d61560b950 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3e8034650 msgr2=0x7ff3e8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3e8034650 0x7ff3e8036b10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8001e50 tx=0x7ff4f80013b0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 msgr2=0x55d615604a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615604a20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff4f401b170 tx=0x7ff4f40180e0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 shutdown_connections 2024-04-27T03:19:08.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d61560ccd0 0x55d61560f0c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d615511a70 0x55d61560b950 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff3e8034650 0x7ff3e8036b10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 --2- 172.21.15.110:0/1608586541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d615511150 0x55d615604a20 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 >> 172.21.15.110:0/1608586541 conn(0x55d61550ca20 msgr2=0x55d6155f7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 shutdown_connections 2024-04-27T03:19:08.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.431+0000 7ff503808980 1 -- 172.21.15.110:0/1608586541 wait complete. 2024-04-27T03:19:08.453 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[{"user":"foo","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"list_buckets","bytes_sent":322,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":5,"successful_ops":5},{"category":"delete_bucket","bytes_sent":236,"bytes_received":0,"ops":2,"successful_ops":1},{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4},{"category":"get_acls","bytes_sent":1026,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"get_obj","bytes_sent":4,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":515,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"put_acls","bytes_sent":0,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"put_obj","bytes_sent":0,"bytes_received":30000042,"ops":11,"successful_ops":11}]},{"bucket":"myfoo2","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo3","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo4","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":8,"successful_ops":8},{"category":"delete_bucket","bytes_sent":236,"bytes_received":0,"ops":5,"successful_ops":4},{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4},{"category":"get_acls","bytes_sent":1026,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"get_obj","bytes_sent":4,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":515,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_buckets","bytes_sent":322,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"put_acls","bytes_sent":0,"bytes_received":0,"ops":2,"successful_ops":2},{"category":"put_obj","bytes_sent":0,"bytes_received":30000042,"ops":11,"successful_ops":11}],"total":{"bytes_sent":2103,"bytes_received":30000042,"ops":35,"successful_ops":34}}]} 2024-04-27T03:19:08.453 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 5, 'successful_ops': 5}, {'category': 'delete_bucket', 'bytes_sent': 236, 'bytes_received': 0, 'ops': 2, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}]}, {'bucket': 'myfoo2', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}, {'category': 'delete_bucket', 'bytes_sent': 236, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}, {'category': 'get_acls', 'bytes_sent': 1026, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'get_obj', 'bytes_sent': 4, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_bucket', 'bytes_sent': 515, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'list_buckets', 'bytes_sent': 322, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_acls', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}], 'total': {'bytes_sent': 2103, 'bytes_received': 30000042, 'ops': 35, 'successful_ops': 34}}]} 2024-04-27T03:19:08.454 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'create_bucket'] 2024-04-27T03:19:08.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'create_bucket'] 2024-04-27T03:19:08.454 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories create_bucket 2024-04-27T03:19:08.533 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:08.533 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:08.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f3ba40980 1 Processor -- start 2024-04-27T03:19:08.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f3ba40980 1 -- start start 2024-04-27T03:19:08.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f3ba40980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f11f968860 0x55f11f9f9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f3ba40980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55f11f8b30d0 con 0x55f11f9f9890 2024-04-27T03:19:08.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f3ba40980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55f11f8a8430 con 0x55f11f968860 2024-04-27T03:19:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f3955b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f11f968860 0x55f11f9f9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f38d5a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55f11f9f9890 0x55f11f9fbc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36198/0 (socket says 172.21.15.110:36198) 2024-04-27T03:19:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.523+0000 7f8f38d5a640 1 -- 172.21.15.110:0/3197690257 learned_addr learned my addr 172.21.15.110:0/3197690257 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3955b640 1 -- 172.21.15.110:0/3197690257 >> v1:172.21.15.77:6789/0 conn(0x55f11f9f9890 legacy=0x55f11f9fbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3955b640 1 -- 172.21.15.110:0/3197690257 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11f8a1510 con 0x55f11f968860 2024-04-27T03:19:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3955b640 1 --2- 172.21.15.110:0/3197690257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11f9f9350 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00a480 tx=0x7f8f2c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72de63e681699233 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f2bfff640 1 -- 172.21.15.110:0/3197690257 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f2c0b0070 con 0x55f11f968860 2024-04-27T03:19:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f2bfff640 1 -- 172.21.15.110:0/3197690257 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0ab540 con 0x55f11f968860 2024-04-27T03:19:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f2bfff640 1 -- 172.21.15.110:0/3197690257 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f2c0ab840 con 0x55f11f968860 2024-04-27T03:19:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3ba40980 1 -- 172.21.15.110:0/3197690257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 msgr2=0x55f11f9f9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/3197690257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11f9f9350 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00a480 tx=0x7f8f2c01b280 comp rx=0 tx=0).stop 2024-04-27T03:19:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3ba40980 1 -- 172.21.15.110:0/3197690257 shutdown_connections 2024-04-27T03:19:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/3197690257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11f9f9350 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3ba40980 1 -- 172.21.15.110:0/3197690257 >> 172.21.15.110:0/3197690257 conn(0x55f11f9eea30 msgr2=0x55f11f9f0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3ba40980 1 -- 172.21.15.110:0/3197690257 shutdown_connections 2024-04-27T03:19:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.527+0000 7f8f3ba40980 1 -- 172.21.15.110:0/3197690257 wait complete. 2024-04-27T03:19:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.543+0000 7f8f3ba40980 1 Processor -- start 2024-04-27T03:19:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.543+0000 7f8f3ba40980 1 -- start start 2024-04-27T03:19:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.543+0000 7f8f3ba40980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.543+0000 7f8f3ba40980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f11f8f72d0 con 0x55f11f968860 2024-04-27T03:19:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.543+0000 7f8f3955b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.543+0000 7f8f3955b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40034/0 (socket says 172.21.15.110:40034) 2024-04-27T03:19:08.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.543+0000 7f8f3955b640 1 -- 172.21.15.110:0/1355419428 learned_addr learned my addr 172.21.15.110:0/1355419428 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3955b640 1 -- 172.21.15.110:0/1355419428 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11f87dae0 con 0x55f11f968860 2024-04-27T03:19:08.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3955b640 1 --2- 172.21.15.110:0/1355419428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae1650 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0ae040 tx=0x7f8f2c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b6f44a3be4ad5bc server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8e27606640 1 -- 172.21.15.110:0/1355419428 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f2c0b0040 con 0x55f11f968860 2024-04-27T03:19:08.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8e27606640 1 -- 172.21.15.110:0/1355419428 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0ab9d0 con 0x55f11f968860 2024-04-27T03:19:08.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8e27606640 1 -- 172.21.15.110:0/1355419428 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f2c0abcf0 con 0x55f11f968860 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1355419428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 msgr2=0x55f11fae1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1355419428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae1650 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0ae040 tx=0x7f8f2c003030 comp rx=0 tx=0).stop 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1355419428 shutdown_connections 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1355419428 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae1650 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1355419428 >> 172.21.15.110:0/1355419428 conn(0x55f11f9eea30 msgr2=0x55f11fadd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1355419428 shutdown_connections 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1355419428 wait complete. 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 Processor -- start 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 -- start start 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3ba40980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f11f9ffe00 con 0x55f11f968860 2024-04-27T03:19:08.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3955b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3955b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40040/0 (socket says 172.21.15.110:40040) 2024-04-27T03:19:08.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3955b640 1 -- 172.21.15.110:0/2725731866 learned_addr learned my addr 172.21.15.110:0/2725731866 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3955b640 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11fae8a50 con 0x55f11f968860 2024-04-27T03:19:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.547+0000 7f8f3955b640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae8510 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c018eb0 tx=0x7f8f2c018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8e25e03640 1 -- 172.21.15.110:0/2725731866 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f2c0b0070 con 0x55f11f968860 2024-04-27T03:19:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8e25e03640 1 -- 172.21.15.110:0/2725731866 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0b68f0 con 0x55f11f968860 2024-04-27T03:19:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f11fae8ce0 con 0x55f11f968860 2024-04-27T03:19:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8e25e03640 1 -- 172.21.15.110:0/2725731866 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f2c0b6bd0 con 0x55f11f968860 2024-04-27T03:19:08.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f11faeb960 con 0x55f11f968860 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8e25e03640 1 -- 172.21.15.110:0/2725731866 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8f2c0b7420 con 0x55f11f968860 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8e25e03640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e20034650 0x7f8e20036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8e25e03640 1 -- 172.21.15.110:0/2725731866 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f8f2c0bc070 con 0x55f11f968860 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f38d5a640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e20034650 0x7f8e20036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11f9f9890 0x55f11faef440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11faef9d0 con 0x55f11f9f9890 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11f9f9890 0x55f11faef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f38d5a640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e20034650 0x7f8e20036b10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8f30001da0 tx=0x7f8f30001300 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11f9f9890 0x55f11faef440 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2725731866 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8f3400a4d0 con 0x55f11f9f9890 2024-04-27T03:19:08.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11faf07c0 0x55f11faf2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11faf31e0 con 0x55f11faf07c0 2024-04-27T03:19:08.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3955b640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11faf07c0 0x55f11faf2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.551+0000 7f8f3955b640 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11faf07c0 0x55f11faf2bb0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3955b640 1 -- 172.21.15.110:0/2725731866 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11faf07c0 2024-04-27T03:19:08.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11faf4a90 con 0x55f11f9f9890 2024-04-27T03:19:08.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2725731866 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f8f3400a4d0 con 0x55f11f9f9890 2024-04-27T03:19:08.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11faf3cc0 con 0x55f11faf07c0 2024-04-27T03:19:08.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3955b640 1 -- 172.21.15.110:0/2725731866 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11faf07c0 2024-04-27T03:19:08.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11faf40b0 con 0x55f11f9f9890 2024-04-27T03:19:08.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2725731866 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f8f3400a4d0 con 0x55f11f9f9890 2024-04-27T03:19:08.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3ba40980 1 Processor -- start 2024-04-27T03:19:08.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3ba40980 1 -- start start 2024-04-27T03:19:08.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3ba40980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb03390 0x55f11fb037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f3ba40980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f11f8fb310 con 0x55f11fb03390 2024-04-27T03:19:08.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f39d5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb03390 0x55f11fb037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f39d5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb03390 0x55f11fb037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40052/0 (socket says 172.21.15.110:40052) 2024-04-27T03:19:08.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f39d5c640 1 -- 172.21.15.110:0/1726099781 learned_addr learned my addr 172.21.15.110:0/1726099781 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f39d5c640 1 -- 172.21.15.110:0/1726099781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11fb03cf0 con 0x55f11fb03390 2024-04-27T03:19:08.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.555+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb03390 0x55f11fb037b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400dfc0 tx=0x7f8f340178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8e1effd640 1 -- 172.21.15.110:0/1726099781 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f340157d0 con 0x55f11fb03390 2024-04-27T03:19:08.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8e1effd640 1 -- 172.21.15.110:0/1726099781 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f34015970 con 0x55f11fb03390 2024-04-27T03:19:08.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8e1effd640 1 -- 172.21.15.110:0/1726099781 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f34015c70 con 0x55f11fb03390 2024-04-27T03:19:08.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f11fb04790 con 0x55f11fb03390 2024-04-27T03:19:08.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f11fb04c60 con 0x55f11fb03390 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8e1effd640 1 -- 172.21.15.110:0/1726099781 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8f34029440 con 0x55f11fb03390 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8e1effd640 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e100346a0 0x7f8e10036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8e1effd640 1 -- 172.21.15.110:0/1726099781 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f8f3400e040 con 0x55f11fb03390 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3955b640 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e100346a0 0x7f8e10036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3ba40980 1 Processor -- start 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3ba40980 1 -- start start 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3955b640 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e100346a0 0x7f8e10036b60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55f11faf37e0 tx=0x7f8f2c0e2bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3ba40980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb109b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f3ba40980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f11fae1ce0 con 0x55f11fb10590 2024-04-27T03:19:08.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f39d5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f39d5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb109b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40064/0 (socket says 172.21.15.110:40064) 2024-04-27T03:19:08.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.559+0000 7f8f39d5c640 1 -- 172.21.15.110:0/1400311056 learned_addr learned my addr 172.21.15.110:0/1400311056 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f39d5c640 1 -- 172.21.15.110:0/1400311056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11fb10f80 con 0x55f11fb10590 2024-04-27T03:19:08.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/1400311056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb109b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8f34015f40 tx=0x7f8f3403c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0edb84b4a3028e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8e1cff9640 1 -- 172.21.15.110:0/1400311056 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f340283a0 con 0x55f11fb10590 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8e1cff9640 1 -- 172.21.15.110:0/1400311056 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f34028540 con 0x55f11fb10590 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8e1cff9640 1 -- 172.21.15.110:0/1400311056 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f3401ecc0 con 0x55f11fb10590 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1400311056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 msgr2=0x55f11fb109b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1400311056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb109b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8f34015f40 tx=0x7f8f3403c280 comp rx=0 tx=0).stop 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1400311056 shutdown_connections 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1400311056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb109b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1400311056 >> 172.21.15.110:0/1400311056 conn(0x55f11fb0c8d0 msgr2=0x55f11fb0a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1400311056 shutdown_connections 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1400311056 wait complete. 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 Processor -- start 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- start start 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb179f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f11fb116a0 con 0x55f11fb10590 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f39d5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb179f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f39d5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb179f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40072/0 (socket says 172.21.15.110:40072) 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 learned_addr learned my addr 172.21.15.110:0/2428667801 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11fb17f30 con 0x55f11fb10590 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb179f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8f34049790 tx=0x7f8f3404a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8e0effd640 1 -- 172.21.15.110:0/2428667801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f34038070 con 0x55f11fb10590 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8e0effd640 1 -- 172.21.15.110:0/2428667801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f3404c0d0 con 0x55f11fb10590 2024-04-27T03:19:08.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8e0effd640 1 -- 172.21.15.110:0/2428667801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f3404c3f0 con 0x55f11fb10590 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f11fb181c0 con 0x55f11fb10590 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.563+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f11fb1aea0 con 0x55f11fb10590 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8e0effd640 1 -- 172.21.15.110:0/2428667801 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8f3404aa00 con 0x55f11fb10590 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8e0effd640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e040346a0 0x7f8e04036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8e0effd640 1 -- 172.21.15.110:0/2428667801 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f8f34034d80 con 0x55f11fb10590 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3955b640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e040346a0 0x7f8e04036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11fb23070 0x55f11fb25510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb25c60 con 0x55f11fb23070 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f38d5a640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11fb23070 0x55f11fb25510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3955b640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e040346a0 0x7f8e04036b60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0aa780 tx=0x7f8f2c0de000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f38d5a640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11fb23070 0x55f11fb25510 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8f3000e640 con 0x55f11fb23070 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 realm 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb22580 con 0x55f11fb23070 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8f3000e640 con 0x55f11fb23070 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb26730 con 0x55f11fb23070 2024-04-27T03:19:08.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8f3000e640 con 0x55f11fb23070 2024-04-27T03:19:08.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11fb27330 0x55f11fb297f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb29f60 con 0x55f11fb27330 2024-04-27T03:19:08.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11fb27330 0x55f11fb297f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.567+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11fb27330 0x55f11fb297f0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3ba40980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2a800 con 0x55f11fb23070 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f8f3000e640 con 0x55f11fb23070 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3ba40980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3ba40980 20 searching for the correct realm 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f11fb2c570 0x55f11fb2e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb2c570 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3955b640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f11fb2c570 0x55f11fb2e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3955b640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f11fb2c570 0x55f11fb2e960 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f39d5c640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8f34014030 0x7f8f340133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x7f8f34014030 2024-04-27T03:19:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f38d5a640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8f34014030 0x7f8f340133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f38d5a640 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8f34014030 0x7f8f340133b0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f3000e640 con 0x7f8f34014030 2024-04-27T03:19:08.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f3000e640 con 0x55f11fb23070 2024-04-27T03:19:08.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.571+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb2c570 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x7f8f34014030 2024-04-27T03:19:08.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x7f8f34014030 2024-04-27T03:19:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x7f8f34014030 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f8f30029040 con 0x7f8f34014030 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.575+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb2c570 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb2c570 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x7f8f34014030 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f8f30029040 con 0x7f8f34014030 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb2c570 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x7f8f34014030 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x7f8f34014030 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb23070 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x7f8f34014030 2024-04-27T03:19:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x7f8f34014030 2024-04-27T03:19:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb2c570 2024-04-27T03:19:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.579+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x7f8f34014030 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f30029040 con 0x7f8f34014030 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb2c570 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ef70 con 0x55f11fb27330 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2b1b0 con 0x55f11fb23070 2024-04-27T03:19:08.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8f30029040 con 0x55f11fb23070 2024-04-27T03:19:08.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.583+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2b600 con 0x55f11fb27330 2024-04-27T03:19:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2ba20 con 0x55f11fb23070 2024-04-27T03:19:08.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 zone default found 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 4 Realm: () 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 zonegroup default 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2f3f0 con 0x7f8f34014030 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x7f8f34014030 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb2adf0 con 0x55f11fb2c570 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb31370 con 0x55f11fb2c570 2024-04-27T03:19:08.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 started sync module instance, tier type = 2024-04-27T03:19:08.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:08.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.587+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb30710 con 0x55f11fb27330 2024-04-27T03:19:08.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94494107306752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb30370 con 0x55f11fb27330 2024-04-27T03:19:08.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb322a0 con 0x55f11fb23070 2024-04-27T03:19:08.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94494107311600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb326c0 con 0x55f11fb23070 2024-04-27T03:19:08.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb33180 con 0x55f11fb27330 2024-04-27T03:19:08.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94494107306752] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94494107311600] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94494107315280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb32210 con 0x55f11fb27330 2024-04-27T03:19:08.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.591+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb33a20 con 0x7f8f34014030 2024-04-27T03:19:08.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x7f8f34014030 2024-04-27T03:19:08.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94494107317616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb32210 con 0x7f8f34014030 2024-04-27T03:19:08.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb344c0 con 0x55f11fb27330 2024-04-27T03:19:08.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94494107317616] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x7f8f34014030 2024-04-27T03:19:08.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94494107315280] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94494107310032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb34480 con 0x55f11fb27330 2024-04-27T03:19:08.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.595+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb348b0 con 0x55f11fb2c570 2024-04-27T03:19:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94494107318672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb35150 con 0x55f11fb2c570 2024-04-27T03:19:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb35b50 con 0x55f11fb2c570 2024-04-27T03:19:08.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94494107310032] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94494107318672] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94494107323552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb35660 con 0x55f11fb2c570 2024-04-27T03:19:08.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb36560 con 0x7f8f34014030 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94494107323552] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'198 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x7f8f34014030 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94494107320240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb36560 con 0x7f8f34014030 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 20 add_watcher() i=0 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 20 add_watcher() i=1 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 20 add_watcher() i=2 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 20 add_watcher() i=3 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 20 add_watcher() i=4 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 20 add_watcher() i=5 2024-04-27T03:19:08.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.599+0000 7f8f3ba40980 20 add_watcher() i=6 2024-04-27T03:19:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94494107320240] v29'199 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x7f8f34014030 2024-04-27T03:19:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f3ba40980 20 add_watcher() i=7 2024-04-27T03:19:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f3ba40980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:08.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb397f0 con 0x55f11fb23070 2024-04-27T03:19:08.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94494107329056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb36af0 con 0x55f11fb23070 2024-04-27T03:19:08.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94494107329056] v29'208 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8def7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f3ba40980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:08.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.603+0000 7f8f3ba40980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:08.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 5 note: GC not initialized 2024-04-27T03:19:08.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8dee7e4640 20 reqs_thread_entry: start 2024-04-27T03:19:08.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:08.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 20 Filter name: none 2024-04-27T03:19:08.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8d5ffff640 20 reqs_thread_entry: start 2024-04-27T03:19:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb5cc20 con 0x55f11fb23070 2024-04-27T03:19:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb5e170 con 0x55f11fb2c570 2024-04-27T03:19:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 usage.10 [call out=1268b] v0'0 uv29 ondisk = 0) ==== 152+0+1268 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8def7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.607+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94494107329056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb5b800 con 0x55f11fb23070 2024-04-27T03:19:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.611+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 data_loggenerations_metadata [watch unwatch cookie 94494107329056] v29'209 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.611+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94494107306752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb36220 con 0x55f11fb27330 2024-04-27T03:19:08.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.611+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(65 notify.0 [watch unwatch cookie 94494107306752] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.611+0000 7f8f3ba40980 20 remove_watcher() i=0 2024-04-27T03:19:08.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.611+0000 7f8f3ba40980 2 removed watcher, disabling cache 2024-04-27T03:19:08.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.611+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94494107311600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb36220 con 0x55f11fb23070 2024-04-27T03:19:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.615+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 notify.1 [watch unwatch cookie 94494107311600] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x55f11fb23070 2024-04-27T03:19:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.615+0000 7f8f3ba40980 20 remove_watcher() i=1 2024-04-27T03:19:08.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.615+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94494107315280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb31df0 con 0x55f11fb27330 2024-04-27T03:19:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.615+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [watch unwatch cookie 94494107315280] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.615+0000 7f8f3ba40980 20 remove_watcher() i=2 2024-04-27T03:19:08.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.615+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:68 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94494107317616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb36220 con 0x7f8f34014030 2024-04-27T03:19:08.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.619+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(68 notify.3 [watch unwatch cookie 94494107317616] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x7f8f34014030 2024-04-27T03:19:08.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.619+0000 7f8f3ba40980 20 remove_watcher() i=3 2024-04-27T03:19:08.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.619+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94494107310032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb32c50 con 0x55f11fb27330 2024-04-27T03:19:08.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.619+0000 7f8f39d5c640 1 -- 172.21.15.110:0/2428667801 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.4 [watch unwatch cookie 94494107310032] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f34026020 con 0x55f11fb27330 2024-04-27T03:19:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.619+0000 7f8f3ba40980 20 remove_watcher() i=4 2024-04-27T03:19:08.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.619+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94494107318672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb31df0 con 0x55f11fb2c570 2024-04-27T03:19:08.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.619+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.5 [watch unwatch cookie 94494107318672] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 20 remove_watcher() i=5 2024-04-27T03:19:08.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94494107323552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb36220 con 0x55f11fb2c570 2024-04-27T03:19:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3955b640 1 -- 172.21.15.110:0/2428667801 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(71 notify.6 [watch unwatch cookie 94494107323552] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f2c0b3040 con 0x55f11fb2c570 2024-04-27T03:19:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 20 remove_watcher() i=6 2024-04-27T03:19:08.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94494107320240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55f11fb32c50 con 0x7f8f34014030 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(72 notify.7 [watch unwatch cookie 94494107320240] v29'200 uv198 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8f3003c1c0 con 0x7f8f34014030 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 20 remove_watcher() i=7 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11fb27330 msgr2=0x55f11fb297f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11fb27330 0x55f11fb297f0 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11fb23070 msgr2=0x55f11fb25510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11fb23070 0x55f11fb25510 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8f34014030 msgr2=0x7f8f340133b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8f34014030 0x7f8f340133b0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f11fb2c570 msgr2=0x55f11fb2e960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f11fb2c570 0x55f11fb2e960 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e040346a0 msgr2=0x7f8e04036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e040346a0 0x7f8e04036b60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0aa780 tx=0x7f8f2c0de000 comp rx=0 tx=0).stop 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 msgr2=0x55f11fb179f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb179f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8f34049790 tx=0x7f8f3404a850 comp rx=0 tx=0).stop 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f38d5a640 1 -- 172.21.15.110:0/2428667801 reap_dead start 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 shutdown_connections 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2428667801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb10590 0x55f11fb179f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 >> 172.21.15.110:0/2428667801 conn(0x55f11fb0c8d0 msgr2=0x55f11fb09f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 shutdown_connections 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2428667801 wait complete. 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e100346a0 msgr2=0x7f8e10036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e100346a0 0x7f8e10036b60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55f11faf37e0 tx=0x7f8f2c0e2bf0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb03390 msgr2=0x55f11fb037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb03390 0x55f11fb037b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400dfc0 tx=0x7f8f340178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 shutdown_connections 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e100346a0 0x7f8e10036b60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/1726099781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11fb03390 0x55f11fb037b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 >> 172.21.15.110:0/1726099781 conn(0x55f11faf81a0 msgr2=0x55f11faf8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 shutdown_connections 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.623+0000 7f8f3ba40980 1 -- 172.21.15.110:0/1726099781 wait complete. 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11faf07c0 msgr2=0x55f11faf2bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11faf07c0 0x55f11faf2bb0 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11f9f9890 msgr2=0x55f11faef440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11f9f9890 0x55f11faef440 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e20034650 msgr2=0x7f8e20036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e20034650 0x7f8e20036b10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8f30001da0 tx=0x7f8f30001300 comp rx=0 tx=0).stop 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 msgr2=0x55f11fae8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae8510 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c018eb0 tx=0x7f8f2c018020 comp rx=0 tx=0).stop 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 shutdown_connections 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f11faf07c0 0x55f11faf2bb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f11f9f9890 0x55f11faef440 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8e20034650 0x7f8e20036b10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 --2- 172.21.15.110:0/2725731866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f11f968860 0x55f11fae8510 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 >> 172.21.15.110:0/2725731866 conn(0x55f11f9eea30 msgr2=0x55f11fadbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 shutdown_connections 2024-04-27T03:19:08.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.627+0000 7f8f3ba40980 1 -- 172.21.15.110:0/2725731866 wait complete. 2024-04-27T03:19:08.646 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[{"user":"foo","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":5,"successful_ops":5}]},{"bucket":"myfoo2","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo3","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo4","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"foo","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":8,"successful_ops":8}],"total":{"bytes_sent":0,"bytes_received":0,"ops":8,"successful_ops":8}}]} 2024-04-27T03:19:08.646 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 5, 'successful_ops': 5}]}, {'bucket': 'myfoo2', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 8, 'successful_ops': 8}}]} 2024-04-27T03:19:08.647 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'put_obj'] 2024-04-27T03:19:08.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'put_obj'] 2024-04-27T03:19:08.647 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories put_obj 2024-04-27T03:19:08.723 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:08.723 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:08.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c4990980 1 Processor -- start 2024-04-27T03:19:08.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c4990980 1 -- start start 2024-04-27T03:19:08.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c4990980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5622d44308a0 0x5622d44bcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c4990980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5622d437b0d0 con 0x5622d44bd240 2024-04-27T03:19:08.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c4990980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5622d4370430 con 0x5622d44308a0 2024-04-27T03:19:08.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c24ab640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5622d44308a0 0x5622d44bcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c1caa640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5622d44bd240 0x5622d44bf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36210/0 (socket says 172.21.15.110:36210) 2024-04-27T03:19:08.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c1caa640 1 -- 172.21.15.110:0/2462916474 learned_addr learned my addr 172.21.15.110:0/2462916474 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:08.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c14a9640 1 -- 172.21.15.110:0/2462916474 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221447960 0 0) 0x5622d437b0d0 con 0x5622d44bd240 2024-04-27T03:19:08.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c14a9640 1 -- 172.21.15.110:0/2462916474 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96ac001e20 con 0x5622d44bd240 2024-04-27T03:19:08.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c24ab640 1 -- 172.21.15.110:0/2462916474 >> v1:172.21.15.77:6789/0 conn(0x5622d44bd240 legacy=0x5622d44bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.715+0000 7f96c24ab640 1 -- 172.21.15.110:0/2462916474 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d4369510 con 0x5622d44308a0 2024-04-27T03:19:08.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c24ab640 1 --2- 172.21.15.110:0/2462916474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d44bcd00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f96b400a480 tx=0x7f96b401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd28e20699a7455 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c14a9640 1 -- 172.21.15.110:0/2462916474 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b40b0070 con 0x5622d44308a0 2024-04-27T03:19:08.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c14a9640 1 -- 172.21.15.110:0/2462916474 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b40ab540 con 0x5622d44308a0 2024-04-27T03:19:08.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c14a9640 1 -- 172.21.15.110:0/2462916474 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b40ab820 con 0x5622d44308a0 2024-04-27T03:19:08.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c4990980 1 -- 172.21.15.110:0/2462916474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 msgr2=0x5622d44bcd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c4990980 1 --2- 172.21.15.110:0/2462916474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d44bcd00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f96b400a480 tx=0x7f96b401b280 comp rx=0 tx=0).stop 2024-04-27T03:19:08.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c4990980 1 -- 172.21.15.110:0/2462916474 shutdown_connections 2024-04-27T03:19:08.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c4990980 1 --2- 172.21.15.110:0/2462916474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d44bcd00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c4990980 1 -- 172.21.15.110:0/2462916474 >> 172.21.15.110:0/2462916474 conn(0x5622d44b6a30 msgr2=0x5622d44b8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c4990980 1 -- 172.21.15.110:0/2462916474 shutdown_connections 2024-04-27T03:19:08.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.719+0000 7f96c4990980 1 -- 172.21.15.110:0/2462916474 wait complete. 2024-04-27T03:19:08.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c4990980 1 Processor -- start 2024-04-27T03:19:08.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c4990980 1 -- start start 2024-04-27T03:19:08.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c4990980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45adc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c4990980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d43bf2d0 con 0x5622d44308a0 2024-04-27T03:19:08.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c24ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c24ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45adc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40082/0 (socket says 172.21.15.110:40082) 2024-04-27T03:19:08.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c24ab640 1 -- 172.21.15.110:0/639414695 learned_addr learned my addr 172.21.15.110:0/639414695 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c24ab640 1 -- 172.21.15.110:0/639414695 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d4345ae0 con 0x5622d44308a0 2024-04-27T03:19:08.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f96c24ab640 1 --2- 172.21.15.110:0/639414695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45adc40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f96b4017e90 tx=0x7f96b4003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2046429d4867437 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f95b0608640 1 -- 172.21.15.110:0/639414695 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b40b0040 con 0x5622d44308a0 2024-04-27T03:19:08.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f95b0608640 1 -- 172.21.15.110:0/639414695 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b40aca40 con 0x5622d44308a0 2024-04-27T03:19:08.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.735+0000 7f95b0608640 1 -- 172.21.15.110:0/639414695 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b40acd20 con 0x5622d44308a0 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- 172.21.15.110:0/639414695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 msgr2=0x5622d45adc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 --2- 172.21.15.110:0/639414695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45adc40 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f96b4017e90 tx=0x7f96b4003030 comp rx=0 tx=0).stop 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- 172.21.15.110:0/639414695 shutdown_connections 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 --2- 172.21.15.110:0/639414695 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45adc40 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- 172.21.15.110:0/639414695 >> 172.21.15.110:0/639414695 conn(0x5622d44b6a30 msgr2=0x5622d45a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- 172.21.15.110:0/639414695 shutdown_connections 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- 172.21.15.110:0/639414695 wait complete. 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 Processor -- start 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- start start 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d44cc640 con 0x5622d44308a0 2024-04-27T03:19:08.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c24ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c24ab640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40096/0 (socket says 172.21.15.110:40096) 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c24ab640 1 -- 172.21.15.110:0/162480078 learned_addr learned my addr 172.21.15.110:0/162480078 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c24ab640 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d45b5100 con 0x5622d44308a0 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c24ab640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45b4bc0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f96b40ab510 tx=0x7f96b4018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f95aee05640 1 -- 172.21.15.110:0/162480078 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b40c5030 con 0x5622d44308a0 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5622d45b5390 con 0x5622d44308a0 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f95aee05640 1 -- 172.21.15.110:0/162480078 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b40b69c0 con 0x5622d44308a0 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f95aee05640 1 -- 172.21.15.110:0/162480078 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96b40b0070 con 0x5622d44308a0 2024-04-27T03:19:08.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.739+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d45b7f70 con 0x5622d44308a0 2024-04-27T03:19:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f95aee05640 1 -- 172.21.15.110:0/162480078 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f96b40c8070 con 0x5622d44308a0 2024-04-27T03:19:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f95aee05640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f95a8034650 0x7f95a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f95aee05640 1 -- 172.21.15.110:0/162480078 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f96b40c6070 con 0x5622d44308a0 2024-04-27T03:19:08.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c1caa640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f95a8034650 0x7f95a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d44bd240 0x5622d45bbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45bc100 con 0x5622d44bd240 2024-04-27T03:19:08.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c2cac640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d44bd240 0x5622d45bbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c1caa640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f95a8034650 0x7f95a8036b10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f96b8001dc0 tx=0x7f96b8001360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c2cac640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d44bd240 0x5622d45bbbc0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c2cac640 1 -- 172.21.15.110:0/162480078 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f96bc00a4d0 con 0x5622d44bd240 2024-04-27T03:19:08.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45bce60 0x5622d45bf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45bf820 con 0x5622d45bce60 2024-04-27T03:19:08.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c24ab640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45bce60 0x5622d45bf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c24ab640 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45bce60 0x5622d45bf250 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c24ab640 1 -- 172.21.15.110:0/162480078 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45bce60 2024-04-27T03:19:08.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45c10d0 con 0x5622d44bd240 2024-04-27T03:19:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c2cac640 1 -- 172.21.15.110:0/162480078 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f96bc00a4d0 con 0x5622d44bd240 2024-04-27T03:19:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45c0300 con 0x5622d45bce60 2024-04-27T03:19:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c24ab640 1 -- 172.21.15.110:0/162480078 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45bce60 2024-04-27T03:19:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.743+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45c0720 con 0x5622d44bd240 2024-04-27T03:19:08.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c2cac640 1 -- 172.21.15.110:0/162480078 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f96bc00a4d0 con 0x5622d44bd240 2024-04-27T03:19:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c4990980 1 Processor -- start 2024-04-27T03:19:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c4990980 1 -- start start 2024-04-27T03:19:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c4990980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45cf9e0 0x5622d45cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c4990980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d43c3310 con 0x5622d45cf9e0 2024-04-27T03:19:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c2cac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45cf9e0 0x5622d45cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c2cac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45cf9e0 0x5622d45cfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40104/0 (socket says 172.21.15.110:40104) 2024-04-27T03:19:08.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c2cac640 1 -- 172.21.15.110:0/2477835784 learned_addr learned my addr 172.21.15.110:0/2477835784 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c2cac640 1 -- 172.21.15.110:0/2477835784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d45d0340 con 0x5622d45cf9e0 2024-04-27T03:19:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c2cac640 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45cf9e0 0x5622d45cfe00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00df90 tx=0x7f96bc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f95a7fff640 1 -- 172.21.15.110:0/2477835784 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96bc0157d0 con 0x5622d45cf9e0 2024-04-27T03:19:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f95a7fff640 1 -- 172.21.15.110:0/2477835784 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96bc015970 con 0x5622d45cf9e0 2024-04-27T03:19:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5622d45d0de0 con 0x5622d45cf9e0 2024-04-27T03:19:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f95a7fff640 1 -- 172.21.15.110:0/2477835784 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96bc015c50 con 0x5622d45cf9e0 2024-04-27T03:19:08.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d45d12b0 con 0x5622d45cf9e0 2024-04-27T03:19:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f95a7fff640 1 -- 172.21.15.110:0/2477835784 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f96bc029440 con 0x5622d45cf9e0 2024-04-27T03:19:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.747+0000 7f95a7fff640 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f959c034650 0x7f959c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f95a7fff640 1 -- 172.21.15.110:0/2477835784 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f96bc00e040 con 0x5622d45cf9e0 2024-04-27T03:19:08.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c24ab640 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f959c034650 0x7f959c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 Processor -- start 2024-04-27T03:19:08.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- start start 2024-04-27T03:19:08.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c24ab640 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f959c034650 0x7f959c036b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x5622d45bc620 tx=0x7f96b40e32e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45dd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d45ae2d0 con 0x5622d45dcbe0 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45dd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45dd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40116/0 (socket says 172.21.15.110:40116) 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 -- 172.21.15.110:0/2919825153 learned_addr learned my addr 172.21.15.110:0/2919825153 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 -- 172.21.15.110:0/2919825153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d45dd5d0 con 0x5622d45dcbe0 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 --2- 172.21.15.110:0/2919825153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45dd000 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f96bc01e390 tx=0x7f96bc01e480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ceb088d81533d7e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f95a5ffb640 1 -- 172.21.15.110:0/2919825153 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96bc012440 con 0x5622d45dcbe0 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f95a5ffb640 1 -- 172.21.15.110:0/2919825153 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96bc041070 con 0x5622d45dcbe0 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f95a5ffb640 1 -- 172.21.15.110:0/2919825153 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96bc029c90 con 0x5622d45dcbe0 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- 172.21.15.110:0/2919825153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 msgr2=0x5622d45dd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 --2- 172.21.15.110:0/2919825153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45dd000 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f96bc01e390 tx=0x7f96bc01e480 comp rx=0 tx=0).stop 2024-04-27T03:19:08.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- 172.21.15.110:0/2919825153 shutdown_connections 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 --2- 172.21.15.110:0/2919825153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45dd000 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- 172.21.15.110:0/2919825153 >> 172.21.15.110:0/2919825153 conn(0x5622d45d8f20 msgr2=0x5622d45d6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- 172.21.15.110:0/2919825153 shutdown_connections 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- 172.21.15.110:0/2919825153 wait complete. 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 Processor -- start 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- start start 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c4990980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5622d45ddd50 con 0x5622d45dcbe0 2024-04-27T03:19:08.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45e40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40122/0 (socket says 172.21.15.110:40122) 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.751+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 learned_addr learned my addr 172.21.15.110:0/4143943298 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622d45e45e0 con 0x5622d45dcbe0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c2cac640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45e40a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f96bc04a7f0 tx=0x7f96bc04a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f9593fff640 1 -- 172.21.15.110:0/4143943298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96bc0491d0 con 0x5622d45dcbe0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5622d45e4870 con 0x5622d45dcbe0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f9593fff640 1 -- 172.21.15.110:0/4143943298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96bc049370 con 0x5622d45dcbe0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f9593fff640 1 -- 172.21.15.110:0/4143943298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96bc04c070 con 0x5622d45dcbe0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622d45e7550 con 0x5622d45dcbe0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f9593fff640 1 -- 172.21.15.110:0/4143943298 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f96bc03a020 con 0x5622d45dcbe0 2024-04-27T03:19:08.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f9593fff640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f958c034650 0x7f958c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f9593fff640 1 -- 172.21.15.110:0/4143943298 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f96bc038020 con 0x5622d45dcbe0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c24ab640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f958c034650 0x7f958c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d45ef510 0x5622d45f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f20f0 con 0x5622d45ef510 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c1caa640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d45ef510 0x5622d45f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c24ab640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f958c034650 0x7f958c036b10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f96b40e2130 tx=0x7f96b40df000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c1caa640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d45ef510 0x5622d45f19a0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f96b800e660 con 0x5622d45ef510 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 realm 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45eea20 con 0x5622d45ef510 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f96b800e660 con 0x5622d45ef510 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f2bc0 con 0x5622d45ef510 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f96b800e660 con 0x5622d45ef510 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45f37a0 0x5622d45f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.755+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f63d0 con 0x5622d45f37a0 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c2cac640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45f37a0 0x5622d45f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c2cac640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45f37a0 0x5622d45f5c60 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c4990980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f6c70 con 0x5622d45ef510 2024-04-27T03:19:08.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f96b800e660 con 0x5622d45ef510 2024-04-27T03:19:08.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c4990980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c4990980 20 searching for the correct realm 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5622d45f89e0 0x5622d45fadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f89e0 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c24ab640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5622d45f89e0 0x5622d45fadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c24ab640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5622d45f89e0 0x5622d45fadd0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.759+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c2cac640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f96bc033070 0x7f96bc03c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x7f96bc033070 2024-04-27T03:19:08.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f96bc033070 0x7f96bc03c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f96bc033070 0x7f96bc03c580 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b800e660 con 0x7f96bc033070 2024-04-27T03:19:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b800e660 con 0x5622d45ef510 2024-04-27T03:19:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f89e0 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.763+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x7f96bc033070 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x7f96bc033070 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x7f96bc033070 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f96b8029040 con 0x7f96bc033070 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f89e0 2024-04-27T03:19:08.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f89e0 2024-04-27T03:19:08.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x7f96bc033070 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f96b8029040 con 0x7f96bc033070 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.767+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f89e0 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x7f96bc033070 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x7f96bc033070 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45ef510 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x7f96bc033070 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x7f96bc033070 2024-04-27T03:19:08.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f89e0 2024-04-27T03:19:08.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x7f96bc033070 2024-04-27T03:19:08.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b8029040 con 0x7f96bc033070 2024-04-27T03:19:08.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f89e0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.771+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb3e0 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:08.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f7620 con 0x5622d45ef510 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f96b8029040 con 0x5622d45ef510 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f7a40 con 0x5622d45f37a0 2024-04-27T03:19:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f7e60 con 0x5622d45ef510 2024-04-27T03:19:08.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 zone default found 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 4 Realm: () 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 zonegroup default 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fb860 con 0x7f96bc033070 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x7f96bc033070 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45f7260 con 0x5622d45f89e0 2024-04-27T03:19:08.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fd7a0 con 0x5622d45f89e0 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 started sync module instance, tier type = 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:08.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.775+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fcb80 con 0x5622d45f37a0 2024-04-27T03:19:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94707591925616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fe0b0 con 0x5622d45f37a0 2024-04-27T03:19:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fe900 con 0x5622d45ef510 2024-04-27T03:19:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94707591930032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fea80 con 0x5622d45ef510 2024-04-27T03:19:08.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45ff550 con 0x5622d45f37a0 2024-04-27T03:19:08.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94707591925616] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.779+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94707591930032] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.783+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.783+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94707591934016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45ff510 con 0x5622d45f37a0 2024-04-27T03:19:08.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.783+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45ff9b0 con 0x7f96bc033070 2024-04-27T03:19:08.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.783+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x7f96bc033070 2024-04-27T03:19:08.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.783+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94707591931088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45ff970 con 0x7f96bc033070 2024-04-27T03:19:08.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.783+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4600870 con 0x5622d45f37a0 2024-04-27T03:19:08.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.783+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94707591931088] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x7f96bc033070 2024-04-27T03:19:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94707591934016] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94707591928832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d46003c0 con 0x5622d45f37a0 2024-04-27T03:19:08.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601110 con 0x5622d45f89e0 2024-04-27T03:19:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94707591941216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d46003c0 con 0x5622d45f89e0 2024-04-27T03:19:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601f80 con 0x5622d45f89e0 2024-04-27T03:19:08.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94707591928832] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94707591941216] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94707591937440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601f80 con 0x5622d45f89e0 2024-04-27T03:19:08.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.787+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d46027d0 con 0x7f96bc033070 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'201 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x7f96bc033070 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94707591937440] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94707591946592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4602c10 con 0x7f96bc033070 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=0 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=1 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=2 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=3 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=4 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=5 2024-04-27T03:19:08.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=6 2024-04-27T03:19:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94707591946592] v29'202 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x7f96bc033070 2024-04-27T03:19:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 20 add_watcher() i=7 2024-04-27T03:19:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601b60 con 0x5622d45ef510 2024-04-27T03:19:08.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.791+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94707591945056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4603100 con 0x5622d45ef510 2024-04-27T03:19:08.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94707591945056] v29'210 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f9578fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:08.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 5 note: GC not initialized 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f94f37fe640 20 reqs_thread_entry: start 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 20 Filter name: none 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f94f17fa640 20 reqs_thread_entry: start 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d4628fb0 con 0x5622d45ef510 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:08.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5622d462a500 con 0x5622d45f89e0 2024-04-27T03:19:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 usage.10 [call out=1268b] v0'0 uv29 ondisk = 0) ==== 152+0+1268 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f9578fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:08.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.795+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94707591945056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d46275b0 con 0x5622d45ef510 2024-04-27T03:19:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.799+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 data_loggenerations_metadata [watch unwatch cookie 94707591945056] v29'211 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.799+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94707591925616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601b60 con 0x5622d45f37a0 2024-04-27T03:19:08.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.799+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(65 notify.0 [watch unwatch cookie 94707591925616] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.799+0000 7f96c4990980 20 remove_watcher() i=0 2024-04-27T03:19:08.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.799+0000 7f96c4990980 2 removed watcher, disabling cache 2024-04-27T03:19:08.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.799+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94707591930032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601b60 con 0x5622d45ef510 2024-04-27T03:19:08.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.803+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 notify.1 [watch unwatch cookie 94707591930032] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x5622d45ef510 2024-04-27T03:19:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.803+0000 7f96c4990980 20 remove_watcher() i=1 2024-04-27T03:19:08.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.803+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94707591934016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fe0b0 con 0x5622d45f37a0 2024-04-27T03:19:08.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.803+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [watch unwatch cookie 94707591934016] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.803+0000 7f96c4990980 20 remove_watcher() i=2 2024-04-27T03:19:08.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.803+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:68 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94707591931088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601b60 con 0x7f96bc033070 2024-04-27T03:19:08.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(68 notify.3 [watch unwatch cookie 94707591931088] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x7f96bc033070 2024-04-27T03:19:08.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c4990980 20 remove_watcher() i=3 2024-04-27T03:19:08.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94707591928832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601b60 con 0x5622d45f37a0 2024-04-27T03:19:08.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c2cac640 1 -- 172.21.15.110:0/4143943298 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.4 [watch unwatch cookie 94707591928832] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96bc026020 con 0x5622d45f37a0 2024-04-27T03:19:08.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c4990980 20 remove_watcher() i=4 2024-04-27T03:19:08.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94707591941216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d45fdc00 con 0x5622d45f89e0 2024-04-27T03:19:08.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.5 [watch unwatch cookie 94707591941216] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c4990980 20 remove_watcher() i=5 2024-04-27T03:19:08.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.807+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94707591937440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601b60 con 0x5622d45f89e0 2024-04-27T03:19:08.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c24ab640 1 -- 172.21.15.110:0/4143943298 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(71 notify.6 [watch unwatch cookie 94707591937440] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b40b3040 con 0x5622d45f89e0 2024-04-27T03:19:08.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 20 remove_watcher() i=6 2024-04-27T03:19:08.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94707591946592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5622d4601b60 con 0x7f96bc033070 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(72 notify.7 [watch unwatch cookie 94707591946592] v29'203 uv201 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f96b803c1c0 con 0x7f96bc033070 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 20 remove_watcher() i=7 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45f37a0 msgr2=0x5622d45f5c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45f37a0 0x5622d45f5c60 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d45ef510 msgr2=0x5622d45f19a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d45ef510 0x5622d45f19a0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f96bc033070 msgr2=0x7f96bc03c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f96bc033070 0x7f96bc03c580 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5622d45f89e0 msgr2=0x5622d45fadd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5622d45f89e0 0x5622d45fadd0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f958c034650 msgr2=0x7f958c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f958c034650 0x7f958c036b10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f96b40e2130 tx=0x7f96b40df000 comp rx=0 tx=0).stop 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 msgr2=0x5622d45e40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c1caa640 1 -- 172.21.15.110:0/4143943298 reap_dead start 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45e40a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f96bc04a7f0 tx=0x7f96bc04a820 comp rx=0 tx=0).stop 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 shutdown_connections 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/4143943298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45dcbe0 0x5622d45e40a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 >> 172.21.15.110:0/4143943298 conn(0x5622d45d8f20 msgr2=0x5622d45d6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 shutdown_connections 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/4143943298 wait complete. 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f959c034650 msgr2=0x7f959c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f959c034650 0x7f959c036b10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x5622d45bc620 tx=0x7f96b40e32e0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45cf9e0 msgr2=0x5622d45cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45cf9e0 0x5622d45cfe00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00df90 tx=0x7f96bc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 shutdown_connections 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f959c034650 0x7f959c036b10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/2477835784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d45cf9e0 0x5622d45cfe00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 >> 172.21.15.110:0/2477835784 conn(0x5622d45c47f0 msgr2=0x5622d45c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 shutdown_connections 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/2477835784 wait complete. 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45bce60 msgr2=0x5622d45bf250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45bce60 0x5622d45bf250 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d44bd240 msgr2=0x5622d45bbbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d44bd240 0x5622d45bbbc0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f95a8034650 msgr2=0x7f95a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f95a8034650 0x7f95a8036b10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f96b8001dc0 tx=0x7f96b8001360 comp rx=0 tx=0).stop 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 msgr2=0x5622d45b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45b4bc0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f96b40ab510 tx=0x7f96b4018020 comp rx=0 tx=0).stop 2024-04-27T03:19:08.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 shutdown_connections 2024-04-27T03:19:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5622d45bce60 0x5622d45bf250 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5622d44bd240 0x5622d45bbbc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f95a8034650 0x7f95a8036b10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 --2- 172.21.15.110:0/162480078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5622d44308a0 0x5622d45b4bc0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 >> 172.21.15.110:0/162480078 conn(0x5622d44b6a30 msgr2=0x5622d45a82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 shutdown_connections 2024-04-27T03:19:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.811+0000 7f96c4990980 1 -- 172.21.15.110:0/162480078 wait complete. 2024-04-27T03:19:08.833 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[{"user":"foo","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"put_obj","bytes_sent":0,"bytes_received":30000042,"ops":11,"successful_ops":11}]},{"bucket":"myfoo2","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo3","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo4","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]}]}],"summary":[{"user":"foo","categories":[{"category":"put_obj","bytes_sent":0,"bytes_received":30000042,"ops":11,"successful_ops":11}],"total":{"bytes_sent":0,"bytes_received":30000042,"ops":11,"successful_ops":11}}]} 2024-04-27T03:19:08.834 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}]}, {'bucket': 'myfoo2', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo3', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo4', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}], 'total': {'bytes_sent': 0, 'bytes_received': 30000042, 'ops': 11, 'successful_ops': 11}}]} 2024-04-27T03:19:08.834 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'delete_obj'] 2024-04-27T03:19:08.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'delete_obj'] 2024-04-27T03:19:08.834 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories delete_obj 2024-04-27T03:19:08.917 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:08.917 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:08.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35dd5a3980 1 Processor -- start 2024-04-27T03:19:08.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35dd5a3980 1 -- start start 2024-04-27T03:19:08.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35dd5a3980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564c531da7b0 0x564c531dcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35dd5a3980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x564c530950d0 con 0x564c531d62a0 2024-04-27T03:19:08.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35dd5a3980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x564c5308a430 con 0x564c531da7b0 2024-04-27T03:19:08.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35da8bd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564c531da7b0 0x564c531dcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35db0be640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x564c531d62a0 0x564c531d6680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36218/0 (socket says 172.21.15.110:36218) 2024-04-27T03:19:08.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.907+0000 7f35db0be640 1 -- 172.21.15.110:0/2993560783 learned_addr learned my addr 172.21.15.110:0/2993560783 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:08.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da0bc640 1 -- 172.21.15.110:0/2993560783 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4087731201 0 0) 0x564c530950d0 con 0x564c531d62a0 2024-04-27T03:19:08.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da0bc640 1 -- 172.21.15.110:0/2993560783 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35c8001e20 con 0x564c531d62a0 2024-04-27T03:19:08.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da8bd640 1 -- 172.21.15.110:0/2993560783 >> v1:172.21.15.77:6789/0 conn(0x564c531d62a0 legacy=0x564c531d6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da8bd640 1 -- 172.21.15.110:0/2993560783 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c53083510 con 0x564c531da7b0 2024-04-27T03:19:08.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da8bd640 1 --2- 172.21.15.110:0/2993560783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531da7b0 0x564c531dcba0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f35c400a480 tx=0x7f35c401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b746a2ac1fb71b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da0bc640 1 -- 172.21.15.110:0/2993560783 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35c40b0070 con 0x564c531da7b0 2024-04-27T03:19:08.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da0bc640 1 -- 172.21.15.110:0/2993560783 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c40ab540 con 0x564c531da7b0 2024-04-27T03:19:08.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35da0bc640 1 -- 172.21.15.110:0/2993560783 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35c40ab840 con 0x564c531da7b0 2024-04-27T03:19:08.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2993560783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531da7b0 msgr2=0x564c531dcba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.911+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2993560783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531da7b0 0x564c531dcba0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f35c400a480 tx=0x7f35c401b280 comp rx=0 tx=0).stop 2024-04-27T03:19:08.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.915+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2993560783 shutdown_connections 2024-04-27T03:19:08.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.915+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2993560783 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531da7b0 0x564c531dcba0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.915+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2993560783 >> 172.21.15.110:0/2993560783 conn(0x564c531d1b70 msgr2=0x564c531d3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.915+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2993560783 shutdown_connections 2024-04-27T03:19:08.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.915+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2993560783 wait complete. 2024-04-27T03:19:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.927+0000 7f35dd5a3980 1 Processor -- start 2024-04-27T03:19:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.927+0000 7f35dd5a3980 1 -- start start 2024-04-27T03:19:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.927+0000 7f35dd5a3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.927+0000 7f35dd5a3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564c530d92d0 con 0x564c531d62a0 2024-04-27T03:19:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.927+0000 7f35db0be640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.927+0000 7f35db0be640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40126/0 (socket says 172.21.15.110:40126) 2024-04-27T03:19:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.927+0000 7f35db0be640 1 -- 172.21.15.110:0/3272508620 learned_addr learned my addr 172.21.15.110:0/3272508620 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35db0be640 1 -- 172.21.15.110:0/3272508620 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c5305fae0 con 0x564c531d62a0 2024-04-27T03:19:08.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35db0be640 1 --2- 172.21.15.110:0/3272508620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c1260 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f35cc001740 tx=0x7f35cc001c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933eb54326e410f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f34c960a640 1 -- 172.21.15.110:0/3272508620 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc016070 con 0x564c531d62a0 2024-04-27T03:19:08.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f34c960a640 1 -- 172.21.15.110:0/3272508620 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35cc004530 con 0x564c531d62a0 2024-04-27T03:19:08.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f34c960a640 1 -- 172.21.15.110:0/3272508620 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc004850 con 0x564c531d62a0 2024-04-27T03:19:08.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 -- 172.21.15.110:0/3272508620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 msgr2=0x564c532c1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/3272508620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c1260 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f35cc001740 tx=0x7f35cc001c00 comp rx=0 tx=0).stop 2024-04-27T03:19:08.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 -- 172.21.15.110:0/3272508620 shutdown_connections 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/3272508620 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c1260 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 -- 172.21.15.110:0/3272508620 >> 172.21.15.110:0/3272508620 conn(0x564c531d1b70 msgr2=0x564c532bcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 -- 172.21.15.110:0/3272508620 shutdown_connections 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 -- 172.21.15.110:0/3272508620 wait complete. 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 Processor -- start 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 -- start start 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35dd5a3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564c531dfbd0 con 0x564c531d62a0 2024-04-27T03:19:08.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35db0be640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35db0be640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40134/0 (socket says 172.21.15.110:40134) 2024-04-27T03:19:08.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35db0be640 1 -- 172.21.15.110:0/4105744584 learned_addr learned my addr 172.21.15.110:0/4105744584 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.931+0000 7f35db0be640 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c532c8810 con 0x564c531d62a0 2024-04-27T03:19:08.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35db0be640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c82d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f35cc014040 tx=0x7f35cc001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f34c7e07640 1 -- 172.21.15.110:0/4105744584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc016040 con 0x564c531d62a0 2024-04-27T03:19:08.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f34c7e07640 1 -- 172.21.15.110:0/4105744584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35cc011400 con 0x564c531d62a0 2024-04-27T03:19:08.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f34c7e07640 1 -- 172.21.15.110:0/4105744584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35cc011720 con 0x564c531d62a0 2024-04-27T03:19:08.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564c532c8aa0 con 0x564c531d62a0 2024-04-27T03:19:08.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c532cb720 con 0x564c531d62a0 2024-04-27T03:19:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f34c7e07640 1 -- 172.21.15.110:0/4105744584 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f35cc0118c0 con 0x564c531d62a0 2024-04-27T03:19:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f34c7e07640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34c00346a0 0x7f34c0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f34c7e07640 1 -- 172.21.15.110:0/4105744584 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f35cc0496b0 con 0x564c531d62a0 2024-04-27T03:19:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35da8bd640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34c00346a0 0x7f34c0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c531da7b0 0x564c532cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c532cf8b0 con 0x564c531da7b0 2024-04-27T03:19:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35db8bf640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c531da7b0 0x564c532cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35da8bd640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34c00346a0 0x7f34c0036b60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f35c40ad000 tx=0x7f35c40a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35db8bf640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c531da7b0 0x564c532cf370 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35db8bf640 1 -- 172.21.15.110:0/4105744584 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f35d40083f0 con 0x564c531da7b0 2024-04-27T03:19:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c532d0660 0x564c532d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.935+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c532d3080 con 0x564c532d0660 2024-04-27T03:19:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35db0be640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c532d0660 0x564c532d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35db0be640 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c532d0660 0x564c532d2a50 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35db0be640 1 -- 172.21.15.110:0/4105744584 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f35cc02c070 con 0x564c532d0660 2024-04-27T03:19:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c532d4930 con 0x564c531da7b0 2024-04-27T03:19:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35db8bf640 1 -- 172.21.15.110:0/4105744584 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f35d40083f0 con 0x564c531da7b0 2024-04-27T03:19:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c532d3b60 con 0x564c532d0660 2024-04-27T03:19:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35db0be640 1 -- 172.21.15.110:0/4105744584 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f35cc02c070 con 0x564c532d0660 2024-04-27T03:19:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c532d3f50 con 0x564c531da7b0 2024-04-27T03:19:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35db8bf640 1 -- 172.21.15.110:0/4105744584 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f35d40083f0 con 0x564c531da7b0 2024-04-27T03:19:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35dd5a3980 1 Processor -- start 2024-04-27T03:19:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35dd5a3980 1 -- start start 2024-04-27T03:19:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35dd5a3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c5314cae0 0x564c5314af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.939+0000 7f35dd5a3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564c530dd310 con 0x564c5314cae0 2024-04-27T03:19:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db8bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c5314cae0 0x564c5314af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db8bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c5314cae0 0x564c5314af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40138/0 (socket says 172.21.15.110:40138) 2024-04-27T03:19:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db8bf640 1 -- 172.21.15.110:0/331029269 learned_addr learned my addr 172.21.15.110:0/331029269 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db8bf640 1 -- 172.21.15.110:0/331029269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c5314b4c0 con 0x564c5314cae0 2024-04-27T03:19:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db8bf640 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c5314cae0 0x564c5314af80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x564c532c93f0 tx=0x7f35d400af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f34c5602640 1 -- 172.21.15.110:0/331029269 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d4007ce0 con 0x564c5314cae0 2024-04-27T03:19:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564c5314b750 con 0x564c5314cae0 2024-04-27T03:19:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f34c5602640 1 -- 172.21.15.110:0/331029269 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d4018440 con 0x564c5314cae0 2024-04-27T03:19:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f34c5602640 1 -- 172.21.15.110:0/331029269 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d4018740 con 0x564c5314cae0 2024-04-27T03:19:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c5314bc20 con 0x564c5314cae0 2024-04-27T03:19:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f34c5602640 1 -- 172.21.15.110:0/331029269 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f35d4013070 con 0x564c5314cae0 2024-04-27T03:19:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f34c5602640 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34b4034650 0x7f34b4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f34c5602640 1 -- 172.21.15.110:0/331029269 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f35d404b890 con 0x564c5314cae0 2024-04-27T03:19:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db0be640 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34b4034650 0x7f34b4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35dd5a3980 1 Processor -- start 2024-04-27T03:19:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35dd5a3980 1 -- start start 2024-04-27T03:19:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db0be640 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34b4034650 0x7f34b4036b10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x564c532d3680 tx=0x7f35cc047890 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35dd5a3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35dd5a3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564c532c19b0 con 0x564c532f2660 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.943+0000 7f35db8bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40144/0 (socket says 172.21.15.110:40144) 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 -- 172.21.15.110:0/4037820770 learned_addr learned my addr 172.21.15.110:0/4037820770 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 -- 172.21.15.110:0/4037820770 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c532f3030 con 0x564c532f2660 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 --2- 172.21.15.110:0/4037820770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f2a60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f35d4008fd0 tx=0x7f35d4016e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abb184a1784991d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f34beffd640 1 -- 172.21.15.110:0/4037820770 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d4047070 con 0x564c532f2660 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f34beffd640 1 -- 172.21.15.110:0/4037820770 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d4021370 con 0x564c532f2660 2024-04-27T03:19:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f34beffd640 1 -- 172.21.15.110:0/4037820770 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d4022dc0 con 0x564c532f2660 2024-04-27T03:19:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4037820770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 msgr2=0x564c532f2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4037820770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f2a60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f35d4008fd0 tx=0x7f35d4016e60 comp rx=0 tx=0).stop 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4037820770 shutdown_connections 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4037820770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f2a60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4037820770 >> 172.21.15.110:0/4037820770 conn(0x564c532ee9a0 msgr2=0x564c532eca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4037820770 shutdown_connections 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4037820770 wait complete. 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 Processor -- start 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- start start 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564c532f37d0 con 0x564c532f2660 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40146/0 (socket says 172.21.15.110:40146) 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 learned_addr learned my addr 172.21.15.110:0/2846691301 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c532f9ff0 con 0x564c532f2660 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35db8bf640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f9ab0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f35d4008fd0 tx=0x7f35d4049a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f34bd7fa640 1 -- 172.21.15.110:0/2846691301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d4031070 con 0x564c532f2660 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f34bd7fa640 1 -- 172.21.15.110:0/2846691301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d4023470 con 0x564c532f2660 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564c532fa280 con 0x564c532f2660 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f34bd7fa640 1 -- 172.21.15.110:0/2846691301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f35d4047040 con 0x564c532f2660 2024-04-27T03:19:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.947+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c532fcf60 con 0x564c532f2660 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f34bd7fa640 1 -- 172.21.15.110:0/2846691301 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f35d4034070 con 0x564c532f2660 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f34bd7fa640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34a4034600 0x7f34a4036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f34bd7fa640 1 -- 172.21.15.110:0/2846691301 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f35d4037db0 con 0x564c532f2660 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35db0be640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34a4034600 0x7f34a4036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c53305110 0x564c533075a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c53307cf0 con 0x564c53305110 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35da8bd640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c53305110 0x564c533075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35db0be640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34a4034600 0x7f34a4036ac0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35cc02cff0 tx=0x7f35cc037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35da8bd640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c53305110 0x564c533075a0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 realm 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c53308120 con 0x564c53305110 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c533089f0 con 0x564c53305110 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c53309410 0x564c5330b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c5330bf70 con 0x564c53309410 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35db8bf640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c53309410 0x564c5330b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.951+0000 7f35db8bf640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c53309410 0x564c5330b800 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35dd5a3980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c5330c810 con 0x564c53305110 2024-04-27T03:19:08.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35dd5a3980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35dd5a3980 20 searching for the correct realm 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564c5330e580 0x564c53310970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x564c53310f50 con 0x564c5330e580 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db0be640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564c5330e580 0x564c53310970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db0be640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564c5330e580 0x564c53310970 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x564c53310f50 con 0x564c53309410 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db8bf640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f35d4036070 0x7f35d4013700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x564c53310f50 con 0x7f35d4036070 2024-04-27T03:19:08.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35da8bd640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f35d4036070 0x7f35d4013700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.955+0000 7f35da8bd640 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f35d4036070 0x7f35d4013700 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x564c53310f50 con 0x564c53305110 2024-04-27T03:19:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x564c53310f50 con 0x564c53305110 2024-04-27T03:19:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x564c53310f50 con 0x564c5330e580 2024-04-27T03:19:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53305110 2024-04-27T03:19:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53305110 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x7f35d4036070 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53305110 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x7f35d4036070 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.959+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c5330e580 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53305110 2024-04-27T03:19:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c5330e580 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53305110 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x7f35d4036070 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53305110 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c5330e580 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x7f35d4036070 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.963+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53305110 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x7f35d4036070 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c5330e580 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x7f35d4036070 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c5330e580 2024-04-27T03:19:08.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.967+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7f35cc054040 con 0x564c53309410 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:08.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c5330d1c0 con 0x564c53305110 2024-04-27T03:19:08.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c5330d5e0 con 0x564c53309410 2024-04-27T03:19:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:08.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c5330da40 con 0x564c53305110 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 zone default found 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 4 Realm: () 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 zonegroup default 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c53311880 con 0x7f35d4036070 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c53313260 con 0x564c5330e580 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c53313720 con 0x564c5330e580 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 started sync module instance, tier type = 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:08.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.971+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533126f0 con 0x564c53309410 2024-04-27T03:19:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.975+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.975+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94885813234704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533126f0 con 0x564c53309410 2024-04-27T03:19:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.975+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53314200 con 0x564c53305110 2024-04-27T03:19:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.975+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.975+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94885813238992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533141c0 con 0x564c53305110 2024-04-27T03:19:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.975+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53315130 con 0x564c53309410 2024-04-27T03:19:08.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94885813234704] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94885813238992] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94885813241328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53315130 con 0x564c53309410 2024-04-27T03:19:08.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53315580 con 0x7f35d4036070 2024-04-27T03:19:08.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94885813232192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53315540 con 0x7f35d4036070 2024-04-27T03:19:08.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.979+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53316460 con 0x564c53309410 2024-04-27T03:19:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94885813232192] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94885813241328] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94885813242944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53316460 con 0x564c53309410 2024-04-27T03:19:08.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53316850 con 0x564c5330e580 2024-04-27T03:19:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94885813246320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533170a0 con 0x564c5330e580 2024-04-27T03:19:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53317aa0 con 0x564c5330e580 2024-04-27T03:19:08.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94885813242944] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94885813246320] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94885813251056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533175b0 con 0x564c5330e580 2024-04-27T03:19:08.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.983+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533184b0 con 0x7f35d4036070 2024-04-27T03:19:08.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94885813251056] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:08.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'204 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94885813247888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533184b0 con 0x7f35d4036070 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=0 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=1 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=2 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=3 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=4 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=5 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=6 2024-04-27T03:19:08.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94885813247888] v29'205 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 20 add_watcher() i=7 2024-04-27T03:19:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c5331b740 con 0x564c53305110 2024-04-27T03:19:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:08.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.987+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94885813256560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53318a40 con 0x564c53305110 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94885813256560] v29'212 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f3491fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 5 note: GC not initialized 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f3490fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:09.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 20 Filter name: none 2024-04-27T03:19:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f340a7fc640 20 reqs_thread_entry: start 2024-04-27T03:19:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c5333eab0 con 0x564c53305110 2024-04-27T03:19:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:09.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x564c53340000 con 0x564c5330e580 2024-04-27T03:19:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.991+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 usage.10 [call out=1268b] v0'0 uv29 ondisk = 0) ==== 152+0+1268 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.995+0000 7f3491fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.995+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94885813256560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c5333d170 con 0x564c53305110 2024-04-27T03:19:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.995+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 data_loggenerations_metadata [watch unwatch cookie 94885813256560] v29'213 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:09.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.995+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94885813234704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53318170 con 0x564c53309410 2024-04-27T03:19:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(65 notify.0 [watch unwatch cookie 94885813234704] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:09.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35dd5a3980 20 remove_watcher() i=0 2024-04-27T03:19:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35dd5a3980 2 removed watcher, disabling cache 2024-04-27T03:19:09.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94885813238992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53318170 con 0x564c53305110 2024-04-27T03:19:09.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 notify.1 [watch unwatch cookie 94885813238992] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x564c53305110 2024-04-27T03:19:09.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35dd5a3980 20 remove_watcher() i=1 2024-04-27T03:19:09.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94885813241328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53318170 con 0x564c53309410 2024-04-27T03:19:09.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [watch unwatch cookie 94885813241328] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:09.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35dd5a3980 20 remove_watcher() i=2 2024-04-27T03:19:09.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:08.999+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:68 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94885813232192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533145f0 con 0x7f35d4036070 2024-04-27T03:19:09.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.003+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(68 notify.3 [watch unwatch cookie 94885813232192] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:09.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.003+0000 7f35dd5a3980 20 remove_watcher() i=3 2024-04-27T03:19:09.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.003+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94885813242944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c533145f0 con 0x564c53309410 2024-04-27T03:19:09.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.003+0000 7f35db8bf640 1 -- 172.21.15.110:0/2846691301 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.4 [watch unwatch cookie 94885813242944] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35d4053040 con 0x564c53309410 2024-04-27T03:19:09.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.003+0000 7f35dd5a3980 20 remove_watcher() i=4 2024-04-27T03:19:09.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.003+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94885813246320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53314c40 con 0x564c5330e580 2024-04-27T03:19:09.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.5 [watch unwatch cookie 94885813246320] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:09.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 20 remove_watcher() i=5 2024-04-27T03:19:09.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94885813251056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53314c40 con 0x564c5330e580 2024-04-27T03:19:09.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35db0be640 1 -- 172.21.15.110:0/2846691301 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(71 notify.6 [watch unwatch cookie 94885813251056] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35cc02c070 con 0x564c5330e580 2024-04-27T03:19:09.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 20 remove_watcher() i=6 2024-04-27T03:19:09.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94885813247888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x564c53316bf0 con 0x7f35d4036070 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(72 notify.7 [watch unwatch cookie 94885813247888] v29'206 uv204 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f35c40b3040 con 0x7f35d4036070 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 20 remove_watcher() i=7 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c53309410 msgr2=0x564c5330b800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c53309410 0x564c5330b800 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c53305110 msgr2=0x564c533075a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c53305110 0x564c533075a0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f35d4036070 msgr2=0x7f35d4013700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f35d4036070 0x7f35d4013700 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564c5330e580 msgr2=0x564c53310970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.007+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564c5330e580 0x564c53310970 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34a4034600 msgr2=0x7f34a4036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34a4034600 0x7f34a4036ac0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f35cc02cff0 tx=0x7f35cc037000 comp rx=0 tx=0).stop 2024-04-27T03:19:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35da8bd640 1 -- 172.21.15.110:0/2846691301 reap_dead start 2024-04-27T03:19:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 msgr2=0x564c532f9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f9ab0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f35d4008fd0 tx=0x7f35d4049a70 comp rx=0 tx=0).stop 2024-04-27T03:19:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 shutdown_connections 2024-04-27T03:19:09.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/2846691301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c532f2660 0x564c532f9ab0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 >> 172.21.15.110:0/2846691301 conn(0x564c532ee9a0 msgr2=0x564c532ec000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 shutdown_connections 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/2846691301 wait complete. 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34b4034650 msgr2=0x7f34b4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34b4034650 0x7f34b4036b10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x564c532d3680 tx=0x7f35cc047890 comp rx=0 tx=0).stop 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c5314cae0 msgr2=0x564c5314af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c5314cae0 0x564c5314af80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x564c532c93f0 tx=0x7f35d400af40 comp rx=0 tx=0).stop 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 shutdown_connections 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34b4034650 0x7f34b4036b10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/331029269 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c5314cae0 0x564c5314af80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 >> 172.21.15.110:0/331029269 conn(0x564c532d8060 msgr2=0x564c532d8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 shutdown_connections 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/331029269 wait complete. 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c532d0660 msgr2=0x564c532d2a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c532d0660 0x564c532d2a50 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c531da7b0 msgr2=0x564c532cf370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c531da7b0 0x564c532cf370 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34c00346a0 msgr2=0x7f34c0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34c00346a0 0x7f34c0036b60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f35c40ad000 tx=0x7f35c40a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 msgr2=0x564c532c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c82d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f35cc014040 tx=0x7f35cc001260 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 shutdown_connections 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564c532d0660 0x564c532d2a50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564c531da7b0 0x564c532cf370 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f34c00346a0 0x7f34c0036b60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 --2- 172.21.15.110:0/4105744584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564c531d62a0 0x564c532c82d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 >> 172.21.15.110:0/4105744584 conn(0x564c531d1b70 msgr2=0x564c532bb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 shutdown_connections 2024-04-27T03:19:09.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.011+0000 7f35dd5a3980 1 -- 172.21.15.110:0/4105744584 wait complete. 2024-04-27T03:19:09.033 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[{"user":"foo","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4}]},{"bucket":"myfoo2","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo3","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo4","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]}]}],"summary":[{"user":"foo","categories":[{"category":"delete_obj","bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4}],"total":{"bytes_sent":0,"bytes_received":0,"ops":4,"successful_ops":4}}]} 2024-04-27T03:19:09.034 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}]}, {'bucket': 'myfoo2', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo3', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo4', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}}]} 2024-04-27T03:19:09.034 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo', '--categories', 'delete_bucket'] 2024-04-27T03:19:09.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo', '--categories', 'delete_bucket'] 2024-04-27T03:19:09.034 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo --categories delete_bucket 2024-04-27T03:19:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:09.108 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e410b980 1 Processor -- start 2024-04-27T03:19:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e410b980 1 -- start start 2024-04-27T03:19:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e410b980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x558ace8a74c0 0x558ace8a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e410b980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x558ace7640d0 con 0x558ace8a7de0 2024-04-27T03:19:09.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e410b980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x558ace759430 con 0x558ace8a74c0 2024-04-27T03:19:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e1c26640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x558ace8a74c0 0x558ace8a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e1425640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x558ace8a7de0 0x558ace8a87a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36222/0 (socket says 172.21.15.110:36222) 2024-04-27T03:19:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.099+0000 7f28e1425640 1 -- 172.21.15.110:0/2725097109 learned_addr learned my addr 172.21.15.110:0/2725097109 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e1c26640 1 -- 172.21.15.110:0/2725097109 >> v1:172.21.15.77:6789/0 conn(0x558ace8a7de0 legacy=0x558ace8a87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e1c26640 1 -- 172.21.15.110:0/2725097109 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace752510 con 0x558ace8a74c0 2024-04-27T03:19:09.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e1c26640 1 --2- 172.21.15.110:0/2725097109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace8a78a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f28d400a480 tx=0x7f28d401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71126319183175e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e0c24640 1 -- 172.21.15.110:0/2725097109 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d40b0070 con 0x558ace8a74c0 2024-04-27T03:19:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e0c24640 1 -- 172.21.15.110:0/2725097109 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d40ab540 con 0x558ace8a74c0 2024-04-27T03:19:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e0c24640 1 -- 172.21.15.110:0/2725097109 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d40ab840 con 0x558ace8a74c0 2024-04-27T03:19:09.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e410b980 1 -- 172.21.15.110:0/2725097109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 msgr2=0x558ace8a78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e410b980 1 --2- 172.21.15.110:0/2725097109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace8a78a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f28d400a480 tx=0x7f28d401b280 comp rx=0 tx=0).stop 2024-04-27T03:19:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e410b980 1 -- 172.21.15.110:0/2725097109 shutdown_connections 2024-04-27T03:19:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e410b980 1 --2- 172.21.15.110:0/2725097109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace8a78a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e410b980 1 -- 172.21.15.110:0/2725097109 >> 172.21.15.110:0/2725097109 conn(0x558ace81aba0 msgr2=0x558ace81afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e410b980 1 -- 172.21.15.110:0/2725097109 shutdown_connections 2024-04-27T03:19:09.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.103+0000 7f28e410b980 1 -- 172.21.15.110:0/2725097109 wait complete. 2024-04-27T03:19:09.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e410b980 1 Processor -- start 2024-04-27T03:19:09.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e410b980 1 -- start start 2024-04-27T03:19:09.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e410b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace992d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e410b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558ace7a82d0 con 0x558ace8a74c0 2024-04-27T03:19:09.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e1c26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace992d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e1c26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace992d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40156/0 (socket says 172.21.15.110:40156) 2024-04-27T03:19:09.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e1c26640 1 -- 172.21.15.110:0/3323205046 learned_addr learned my addr 172.21.15.110:0/3323205046 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.119+0000 7f28e1c26640 1 -- 172.21.15.110:0/3323205046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace72eae0 con 0x558ace8a74c0 2024-04-27T03:19:09.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e1c26640 1 --2- 172.21.15.110:0/3323205046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace992d70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f28d40ae040 tx=0x7f28d4003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a1c1f9470578ef server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f27cfe07640 1 -- 172.21.15.110:0/3323205046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d40b0040 con 0x558ace8a74c0 2024-04-27T03:19:09.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f27cfe07640 1 -- 172.21.15.110:0/3323205046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d40ab9b0 con 0x558ace8a74c0 2024-04-27T03:19:09.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f27cfe07640 1 -- 172.21.15.110:0/3323205046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d40abc90 con 0x558ace8a74c0 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- 172.21.15.110:0/3323205046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 msgr2=0x558ace992d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 --2- 172.21.15.110:0/3323205046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace992d70 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f28d40ae040 tx=0x7f28d4003030 comp rx=0 tx=0).stop 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- 172.21.15.110:0/3323205046 shutdown_connections 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 --2- 172.21.15.110:0/3323205046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace992d70 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- 172.21.15.110:0/3323205046 >> 172.21.15.110:0/3323205046 conn(0x558ace81aba0 msgr2=0x558ace98e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- 172.21.15.110:0/3323205046 shutdown_connections 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- 172.21.15.110:0/3323205046 wait complete. 2024-04-27T03:19:09.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 Processor -- start 2024-04-27T03:19:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- start start 2024-04-27T03:19:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace999cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558ace8b1510 con 0x558ace8a74c0 2024-04-27T03:19:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e1c26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace999cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e1c26640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace999cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40172/0 (socket says 172.21.15.110:40172) 2024-04-27T03:19:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e1c26640 1 -- 172.21.15.110:0/4256806680 learned_addr learned my addr 172.21.15.110:0/4256806680 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e1c26640 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace99a230 con 0x558ace8a74c0 2024-04-27T03:19:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e1c26640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace999cf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f28d401bb20 tx=0x7f28d4018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f27ce604640 1 -- 172.21.15.110:0/4256806680 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d40c5070 con 0x558ace8a74c0 2024-04-27T03:19:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558ace99a4c0 con 0x558ace8a74c0 2024-04-27T03:19:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f27ce604640 1 -- 172.21.15.110:0/4256806680 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d40b6970 con 0x558ace8a74c0 2024-04-27T03:19:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f27ce604640 1 -- 172.21.15.110:0/4256806680 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28d40b0050 con 0x558ace8a74c0 2024-04-27T03:19:09.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.123+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ace99d0a0 con 0x558ace8a74c0 2024-04-27T03:19:09.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f27ce604640 1 -- 172.21.15.110:0/4256806680 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f28d40c8070 con 0x558ace8a74c0 2024-04-27T03:19:09.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f27ce604640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c8034650 0x7f27c8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f27ce604640 1 -- 172.21.15.110:0/4256806680 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f28d40bc070 con 0x558ace8a74c0 2024-04-27T03:19:09.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e1425640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c8034650 0x7f27c8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace8a7de0 0x558ace9a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9a1100 con 0x558ace8a7de0 2024-04-27T03:19:09.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e2427640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace8a7de0 0x558ace9a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e1425640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c8034650 0x7f27c8036b10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f28d8000f70 tx=0x7f28d80012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e2427640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace8a7de0 0x558ace9a0b80 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e2427640 1 -- 172.21.15.110:0/4256806680 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28dc00a4d0 con 0x558ace8a7de0 2024-04-27T03:19:09.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9a1ef0 0x558ace9a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9a4920 con 0x558ace9a1ef0 2024-04-27T03:19:09.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e1c26640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9a1ef0 0x558ace9a42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e1c26640 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9a1ef0 0x558ace9a42e0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e1c26640 1 -- 172.21.15.110:0/4256806680 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9a1ef0 2024-04-27T03:19:09.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.127+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9a61d0 con 0x558ace8a7de0 2024-04-27T03:19:09.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e2427640 1 -- 172.21.15.110:0/4256806680 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f28dc00a4d0 con 0x558ace8a7de0 2024-04-27T03:19:09.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9a5400 con 0x558ace9a1ef0 2024-04-27T03:19:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e1c26640 1 -- 172.21.15.110:0/4256806680 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9a1ef0 2024-04-27T03:19:09.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9a57f0 con 0x558ace8a7de0 2024-04-27T03:19:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e2427640 1 -- 172.21.15.110:0/4256806680 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f28dc00a4d0 con 0x558ace8a7de0 2024-04-27T03:19:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 Processor -- start 2024-04-27T03:19:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 -- start start 2024-04-27T03:19:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9b4ad0 0x558ace9b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558ace7ac310 con 0x558ace9b4ad0 2024-04-27T03:19:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e2427640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9b4ad0 0x558ace9b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e2427640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9b4ad0 0x558ace9b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40174/0 (socket says 172.21.15.110:40174) 2024-04-27T03:19:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e2427640 1 -- 172.21.15.110:0/3668033086 learned_addr learned my addr 172.21.15.110:0/3668033086 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e2427640 1 -- 172.21.15.110:0/3668033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace9b5430 con 0x558ace9b4ad0 2024-04-27T03:19:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e2427640 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9b4ad0 0x558ace9b4ef0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00dfc0 tx=0x7f28dc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f27c77fe640 1 -- 172.21.15.110:0/3668033086 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28dc0157d0 con 0x558ace9b4ad0 2024-04-27T03:19:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558ace9b5ed0 con 0x558ace9b4ad0 2024-04-27T03:19:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f27c77fe640 1 -- 172.21.15.110:0/3668033086 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28dc015970 con 0x558ace9b4ad0 2024-04-27T03:19:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f27c77fe640 1 -- 172.21.15.110:0/3668033086 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28dc015c70 con 0x558ace9b4ad0 2024-04-27T03:19:09.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.131+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ace9b63a0 con 0x558ace9b4ad0 2024-04-27T03:19:09.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f27c77fe640 1 -- 172.21.15.110:0/3668033086 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f28dc029440 con 0x558ace9b4ad0 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f27c77fe640 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27bc0346a0 0x7f27bc036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f27c77fe640 1 -- 172.21.15.110:0/3668033086 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f28dc00e040 con 0x558ace9b4ad0 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e1c26640 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27bc0346a0 0x7f27bc036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 Processor -- start 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 -- start start 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e1c26640 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27bc0346a0 0x7f27bc036b60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x558ace9a4f20 tx=0x7f28d4000bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558ace993400 con 0x558ace9c1cd0 2024-04-27T03:19:09.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e2427640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e2427640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40184/0 (socket says 172.21.15.110:40184) 2024-04-27T03:19:09.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e2427640 1 -- 172.21.15.110:0/2089813318 learned_addr learned my addr 172.21.15.110:0/2089813318 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e2427640 1 -- 172.21.15.110:0/2089813318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace9c26c0 con 0x558ace9c1cd0 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e2427640 1 --2- 172.21.15.110:0/2089813318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c20f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f28dc015ec0 tx=0x7f28dc015fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984709fddea8452a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f27c57fa640 1 -- 172.21.15.110:0/2089813318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28dc0283a0 con 0x558ace9c1cd0 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f27c57fa640 1 -- 172.21.15.110:0/2089813318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28dc028540 con 0x558ace9c1cd0 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f27c57fa640 1 -- 172.21.15.110:0/2089813318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28dc01ecc0 con 0x558ace9c1cd0 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 -- 172.21.15.110:0/2089813318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 msgr2=0x558ace9c20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 --2- 172.21.15.110:0/2089813318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c20f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f28dc015ec0 tx=0x7f28dc015fb0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 -- 172.21.15.110:0/2089813318 shutdown_connections 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 --2- 172.21.15.110:0/2089813318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c20f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 -- 172.21.15.110:0/2089813318 >> 172.21.15.110:0/2089813318 conn(0x558ace9be010 msgr2=0x558ace9bc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 -- 172.21.15.110:0/2089813318 shutdown_connections 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.135+0000 7f28e410b980 1 -- 172.21.15.110:0/2089813318 wait complete. 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 Processor -- start 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 -- start start 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x558ace9c2de0 con 0x558ace9c1cd0 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e2427640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e2427640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40192/0 (socket says 172.21.15.110:40192) 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 learned_addr learned my addr 172.21.15.110:0/446774240 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ace9c9690 con 0x558ace9c1cd0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e2427640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c9150 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f28dc012750 tx=0x7f28dc04a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f27b37fe640 1 -- 172.21.15.110:0/446774240 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28dc038070 con 0x558ace9c1cd0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f27b37fe640 1 -- 172.21.15.110:0/446774240 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28dc049580 con 0x558ace9c1cd0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558ace9c9920 con 0x558ace9c1cd0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f27b37fe640 1 -- 172.21.15.110:0/446774240 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28dc04c200 con 0x558ace9c1cd0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ace9cc600 con 0x558ace9c1cd0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f27b37fe640 1 -- 172.21.15.110:0/446774240 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f28dc04aa00 con 0x558ace9c1cd0 2024-04-27T03:19:09.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f27b37fe640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27ac034650 0x7f27ac036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f27b37fe640 1 -- 172.21.15.110:0/446774240 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f28dc030070 con 0x558ace9c1cd0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e1c26640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27ac034650 0x7f27ac036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace9d47d0 0x558ace9d6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9d73b0 con 0x558ace9d47d0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.139+0000 7f28e1425640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace9d47d0 0x558ace9d6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1c26640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27ac034650 0x7f27ac036b10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f28d40c5040 tx=0x7f28d40dc000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1425640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace9d47d0 0x558ace9d6c60 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d800e5a0 con 0x558ace9d47d0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 realm 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9d3ce0 con 0x558ace9d47d0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d800e5a0 con 0x558ace9d47d0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9d7eb0 con 0x558ace9d47d0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d800e5a0 con 0x558ace9d47d0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9d8c90 0x558ace9db080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9db700 con 0x558ace9d8c90 2024-04-27T03:19:09.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e2427640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9d8c90 0x558ace9db080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e2427640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9d8c90 0x558ace9db080 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9dbfa0 con 0x558ace9d47d0 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f28d800e5a0 con 0x558ace9d47d0 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 20 searching for the correct realm 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558ace9ddd10 0x558ace9e0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9ddd10 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1c26640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558ace9ddd10 0x558ace9e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.143+0000 7f28e1c26640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558ace9ddd10 0x558ace9e0100 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e2427640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28dc014030 0x7f28dc01f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x7f28dc014030 2024-04-27T03:19:09.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28dc014030 0x7f28dc01f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28dc014030 0x7f28dc01f720 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d800e5a0 con 0x7f28dc014030 2024-04-27T03:19:09.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d800e5a0 con 0x558ace9d47d0 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9ddd10 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.147+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x7f28dc014030 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x7f28dc014030 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x7f28dc014030 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f28d8029040 con 0x7f28dc014030 2024-04-27T03:19:09.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9ddd10 2024-04-27T03:19:09.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.151+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9ddd10 2024-04-27T03:19:09.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x7f28dc014030 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f28d8029040 con 0x7f28dc014030 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9ddd10 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x7f28dc014030 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x7f28dc014030 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d47d0 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x7f28dc014030 2024-04-27T03:19:09.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x7f28dc014030 2024-04-27T03:19:09.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9ddd10 2024-04-27T03:19:09.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x7f28dc014030 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d8029040 con 0x7f28dc014030 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.155+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9ddd10 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0710 con 0x558ace9d8c90 2024-04-27T03:19:09.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9dc950 con 0x558ace9d47d0 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f28d8029040 con 0x558ace9d47d0 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9dcdd0 con 0x558ace9d8c90 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:09.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9dd230 con 0x558ace9d47d0 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 zone default found 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 4 Realm: () 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 zonegroup default 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e0b70 con 0x7f28dc014030 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x7f28dc014030 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e2600 con 0x558ace9ddd10 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.159+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e2af0 con 0x558ace9ddd10 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 20 started sync module instance, tier type = 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:09.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e19f0 con 0x558ace9d8c90 2024-04-27T03:19:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94054660316912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e19f0 con 0x558ace9d8c90 2024-04-27T03:19:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e3650 con 0x558ace9d47d0 2024-04-27T03:19:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94054660321600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e3ac0 con 0x558ace9d47d0 2024-04-27T03:19:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.163+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e4560 con 0x558ace9d8c90 2024-04-27T03:19:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94054660316912] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94054660321600] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94054660325424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e3610 con 0x558ace9d8c90 2024-04-27T03:19:09.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e4e30 con 0x7f28dc014030 2024-04-27T03:19:09.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x7f28dc014030 2024-04-27T03:19:09.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94054660327808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e3610 con 0x7f28dc014030 2024-04-27T03:19:09.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e5870 con 0x558ace9d8c90 2024-04-27T03:19:09.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.167+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94054660325424] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94054660327808] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x7f28dc014030 2024-04-27T03:19:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94054660328864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e53c0 con 0x558ace9d8c90 2024-04-27T03:19:09.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e6110 con 0x558ace9ddd10 2024-04-27T03:19:09.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94054660332640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e53c0 con 0x558ace9ddd10 2024-04-27T03:19:09.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e6e90 con 0x558ace9ddd10 2024-04-27T03:19:09.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94054660328864] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.171+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94054660332640] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94054660314432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e6960 con 0x558ace9ddd10 2024-04-27T03:19:09.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e7770 con 0x7f28dc014030 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'207 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x7f28dc014030 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94054660338304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e76e0 con 0x7f28dc014030 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=0 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=1 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=2 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=3 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=4 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=5 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94054660314432] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=6 2024-04-27T03:19:09.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94054660338304] v29'208 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x7f28dc014030 2024-04-27T03:19:09.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 20 add_watcher() i=7 2024-04-27T03:19:09.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:09.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e64b0 con 0x558ace9d47d0 2024-04-27T03:19:09.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.175+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94054660334768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e8180 con 0x558ace9d47d0 2024-04-27T03:19:09.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94054660334768] v29'214 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f27937fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 5 note: GC not initialized 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f27927fc640 20 reqs_thread_entry: start 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 20 Filter name: none 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f2703fff640 20 reqs_thread_entry: start 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558acea0e050 con 0x558ace9d47d0 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:09.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x558acea0f5a0 con 0x558ace9ddd10 2024-04-27T03:19:09.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.179+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 usage.10 [call out=1268b] v0'0 uv29 ondisk = 0) ==== 152+0+1268 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.183+0000 7f27937fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.183+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94054660334768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558acea0cc30 con 0x558ace9d47d0 2024-04-27T03:19:09.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.183+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 data_loggenerations_metadata [watch unwatch cookie 94054660334768] v29'215 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.183+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94054660316912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e64b0 con 0x558ace9d8c90 2024-04-27T03:19:09.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(65 notify.0 [watch unwatch cookie 94054660316912] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e410b980 20 remove_watcher() i=0 2024-04-27T03:19:09.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e410b980 2 removed watcher, disabling cache 2024-04-27T03:19:09.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94054660321600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e1ef0 con 0x558ace9d47d0 2024-04-27T03:19:09.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 notify.1 [watch unwatch cookie 94054660321600] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x558ace9d47d0 2024-04-27T03:19:09.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e410b980 20 remove_watcher() i=1 2024-04-27T03:19:09.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94054660325424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e1ef0 con 0x558ace9d8c90 2024-04-27T03:19:09.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [watch unwatch cookie 94054660325424] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e410b980 20 remove_watcher() i=2 2024-04-27T03:19:09.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.187+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:68 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94054660327808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e1ef0 con 0x7f28dc014030 2024-04-27T03:19:09.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.191+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(68 notify.3 [watch unwatch cookie 94054660327808] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x7f28dc014030 2024-04-27T03:19:09.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.191+0000 7f28e410b980 20 remove_watcher() i=3 2024-04-27T03:19:09.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.191+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94054660328864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e4980 con 0x558ace9d8c90 2024-04-27T03:19:09.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.191+0000 7f28e2427640 1 -- 172.21.15.110:0/446774240 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.4 [watch unwatch cookie 94054660328864] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28dc026020 con 0x558ace9d8c90 2024-04-27T03:19:09.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.191+0000 7f28e410b980 20 remove_watcher() i=4 2024-04-27T03:19:09.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.191+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94054660332640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e1ef0 con 0x558ace9ddd10 2024-04-27T03:19:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.5 [watch unwatch cookie 94054660332640] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 20 remove_watcher() i=5 2024-04-27T03:19:09.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94054660314432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e1ef0 con 0x558ace9ddd10 2024-04-27T03:19:09.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e1c26640 1 -- 172.21.15.110:0/446774240 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(71 notify.6 [watch unwatch cookie 94054660314432] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d40c7030 con 0x558ace9ddd10 2024-04-27T03:19:09.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 20 remove_watcher() i=6 2024-04-27T03:19:09.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94054660338304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x558ace9e1ef0 con 0x7f28dc014030 2024-04-27T03:19:09.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(72 notify.7 [watch unwatch cookie 94054660338304] v29'209 uv207 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f28d803c1c0 con 0x7f28dc014030 2024-04-27T03:19:09.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 20 remove_watcher() i=7 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9d8c90 msgr2=0x558ace9db080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9d8c90 0x558ace9db080 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace9d47d0 msgr2=0x558ace9d6c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace9d47d0 0x558ace9d6c60 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28dc014030 msgr2=0x7f28dc01f720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f28dc014030 0x7f28dc01f720 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558ace9ddd10 msgr2=0x558ace9e0100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x558ace9ddd10 0x558ace9e0100 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27ac034650 msgr2=0x7f27ac036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27ac034650 0x7f27ac036b10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f28d40c5040 tx=0x7f28d40dc000 comp rx=0 tx=0).stop 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e1425640 1 -- 172.21.15.110:0/446774240 reap_dead start 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 msgr2=0x558ace9c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c9150 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f28dc012750 tx=0x7f28dc04a850 comp rx=0 tx=0).stop 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 shutdown_connections 2024-04-27T03:19:09.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/446774240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9c1cd0 0x558ace9c9150 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 >> 172.21.15.110:0/446774240 conn(0x558ace9be010 msgr2=0x558ace9bb670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 shutdown_connections 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/446774240 wait complete. 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27bc0346a0 msgr2=0x7f27bc036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27bc0346a0 0x7f27bc036b60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x558ace9a4f20 tx=0x7f28d4000bd0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9b4ad0 msgr2=0x558ace9b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.195+0000 7f28e410b980 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9b4ad0 0x558ace9b4ef0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00dfc0 tx=0x7f28dc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 shutdown_connections 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27bc0346a0 0x7f27bc036b60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/3668033086 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace9b4ad0 0x558ace9b4ef0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 >> 172.21.15.110:0/3668033086 conn(0x558ace9a98e0 msgr2=0x558ace9aa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 shutdown_connections 2024-04-27T03:19:09.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/3668033086 wait complete. 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9a1ef0 msgr2=0x558ace9a42e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9a1ef0 0x558ace9a42e0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace8a7de0 msgr2=0x558ace9a0b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace8a7de0 0x558ace9a0b80 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c8034650 msgr2=0x7f27c8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c8034650 0x7f27c8036b10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f28d8000f70 tx=0x7f28d80012c0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 msgr2=0x558ace999cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace999cf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f28d401bb20 tx=0x7f28d4018020 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 shutdown_connections 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x558ace9a1ef0 0x558ace9a42e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x558ace8a7de0 0x558ace9a0b80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f27c8034650 0x7f27c8036b10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 --2- 172.21.15.110:0/4256806680 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x558ace8a74c0 0x558ace999cf0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 >> 172.21.15.110:0/4256806680 conn(0x558ace81aba0 msgr2=0x558ace98d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 shutdown_connections 2024-04-27T03:19:09.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.199+0000 7f28e410b980 1 -- 172.21.15.110:0/4256806680 wait complete. 2024-04-27T03:19:09.219 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[{"user":"foo","buckets":[{"bucket":"","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[]},{"bucket":"myfoo","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"delete_bucket","bytes_sent":236,"bytes_received":0,"ops":2,"successful_ops":1}]},{"bucket":"myfoo2","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo3","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]},{"bucket":"myfoo4","time":"2024-04-27T03:00:00.000000Z","epoch":1714186800,"owner":"foo","categories":[{"category":"delete_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"foo","categories":[{"category":"delete_bucket","bytes_sent":236,"bytes_received":0,"ops":5,"successful_ops":4}],"total":{"bytes_sent":236,"bytes_received":0,"ops":5,"successful_ops":4}}]} 2024-04-27T03:19:09.219 DEBUG:tasks.util.rgw: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 236, 'bytes_received': 0, 'ops': 2, 'successful_ops': 1}]}, {'bucket': 'myfoo2', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo3', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'myfoo4', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 236, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}], 'total': {'bytes_sent': 236, 'bytes_received': 0, 'ops': 5, 'successful_ops': 4}}]} 2024-04-27T03:19:09.219 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'bar', '--display-name', 'Bar', '--access-key', 'NX5QOQKC6BH2IDN8HC7A', '--secret', 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn', '--max-buckets', '4'] 2024-04-27T03:19:09.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'bar', '--display-name', 'Bar', '--access-key', 'NX5QOQKC6BH2IDN8HC7A', '--secret', 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn', '--max-buckets', '4'] 2024-04-27T03:19:09.219 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid bar --display-name Bar --access-key NX5QOQKC6BH2IDN8HC7A --secret LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn --max-buckets 4 2024-04-27T03:19:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:09.307 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f48151a6980 1 Processor -- start 2024-04-27T03:19:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f48151a6980 1 -- start start 2024-04-27T03:19:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f48151a6980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a9fa8a24f0 0x55a9fa8a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f48151a6980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a9fa75f0d0 con 0x55a9fa8a2e10 2024-04-27T03:19:09.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f48151a6980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a9fa754430 con 0x55a9fa8a24f0 2024-04-27T03:19:09.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f4812cc1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a9fa8a24f0 0x55a9fa8a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f48124c0640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55a9fa8a2e10 0x55a9fa8a37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36236/0 (socket says 172.21.15.110:36236) 2024-04-27T03:19:09.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f48124c0640 1 -- 172.21.15.110:0/3316907302 learned_addr learned my addr 172.21.15.110:0/3316907302 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:09.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f4812cc1640 1 -- 172.21.15.110:0/3316907302 >> v1:172.21.15.77:6789/0 conn(0x55a9fa8a2e10 legacy=0x55a9fa8a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.299+0000 7f4812cc1640 1 -- 172.21.15.110:0/3316907302 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fa74d510 con 0x55a9fa8a24f0 2024-04-27T03:19:09.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f4812cc1640 1 --2- 172.21.15.110:0/3316907302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa8a28d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f480400a480 tx=0x7f480401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8baef17e6627e4e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f4811cbf640 1 -- 172.21.15.110:0/3316907302 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48040b0070 con 0x55a9fa8a24f0 2024-04-27T03:19:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f4811cbf640 1 -- 172.21.15.110:0/3316907302 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48040ab540 con 0x55a9fa8a24f0 2024-04-27T03:19:09.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f4811cbf640 1 -- 172.21.15.110:0/3316907302 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48040ab840 con 0x55a9fa8a24f0 2024-04-27T03:19:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f48151a6980 1 -- 172.21.15.110:0/3316907302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 msgr2=0x55a9fa8a28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f48151a6980 1 --2- 172.21.15.110:0/3316907302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa8a28d0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f480400a480 tx=0x7f480401b280 comp rx=0 tx=0).stop 2024-04-27T03:19:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f48151a6980 1 -- 172.21.15.110:0/3316907302 shutdown_connections 2024-04-27T03:19:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f48151a6980 1 --2- 172.21.15.110:0/3316907302 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa8a28d0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f48151a6980 1 -- 172.21.15.110:0/3316907302 >> 172.21.15.110:0/3316907302 conn(0x55a9fa815bd0 msgr2=0x55a9fa815fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f48151a6980 1 -- 172.21.15.110:0/3316907302 shutdown_connections 2024-04-27T03:19:09.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.303+0000 7f48151a6980 1 -- 172.21.15.110:0/3316907302 wait complete. 2024-04-27T03:19:09.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 Processor -- start 2024-04-27T03:19:09.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 -- start start 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa98de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9fa7a32d0 con 0x55a9fa8a24f0 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4812cc1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa98de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4812cc1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa98de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40202/0 (socket says 172.21.15.110:40202) 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4812cc1640 1 -- 172.21.15.110:0/611363999 learned_addr learned my addr 172.21.15.110:0/611363999 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4812cc1640 1 -- 172.21.15.110:0/611363999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fa729ae0 con 0x55a9fa8a24f0 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4812cc1640 1 --2- 172.21.15.110:0/611363999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa98de70 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4804017e90 tx=0x7f4804003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22ead1d24174977 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4700e09640 1 -- 172.21.15.110:0/611363999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48040b0070 con 0x55a9fa8a24f0 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4700e09640 1 -- 172.21.15.110:0/611363999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48040b1040 con 0x55a9fa8a24f0 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f4700e09640 1 -- 172.21.15.110:0/611363999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48040abc30 con 0x55a9fa8a24f0 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 -- 172.21.15.110:0/611363999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 msgr2=0x55a9fa98de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 --2- 172.21.15.110:0/611363999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa98de70 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f4804017e90 tx=0x7f4804003030 comp rx=0 tx=0).stop 2024-04-27T03:19:09.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 -- 172.21.15.110:0/611363999 shutdown_connections 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 --2- 172.21.15.110:0/611363999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa98de70 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 -- 172.21.15.110:0/611363999 >> 172.21.15.110:0/611363999 conn(0x55a9fa815bd0 msgr2=0x55a9fa989a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 -- 172.21.15.110:0/611363999 shutdown_connections 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.319+0000 7f48151a6980 1 -- 172.21.15.110:0/611363999 wait complete. 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 Processor -- start 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 -- start start 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa994d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9fa8ac540 con 0x55a9fa8a24f0 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f4812cc1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa994d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f4812cc1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa994d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40212/0 (socket says 172.21.15.110:40212) 2024-04-27T03:19:09.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f4812cc1640 1 -- 172.21.15.110:0/1475823033 learned_addr learned my addr 172.21.15.110:0/1475823033 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f4812cc1640 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fa9952d0 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f4812cc1640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa994d90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4804017e90 tx=0x7f4804018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f46ff606640 1 -- 172.21.15.110:0/1475823033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48040c6070 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a9fa995560 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f46ff606640 1 -- 172.21.15.110:0/1475823033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48040b78c0 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f46ff606640 1 -- 172.21.15.110:0/1475823033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48040b7ba0 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9fa998190 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f46ff606640 1 -- 172.21.15.110:0/1475823033 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f48040c9070 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f46ff606640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46f8034650 0x7f46f8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f46ff606640 1 -- 172.21.15.110:0/1475823033 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f48040bc070 con 0x55a9fa8a24f0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48124c0640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46f8034650 0x7f46f8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa8a2e10 0x55a9fa99bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.323+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa99c1b0 con 0x55a9fa8a2e10 2024-04-27T03:19:09.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48134c2640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa8a2e10 0x55a9fa99bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48124c0640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46f8034650 0x7f46f8036b10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f47fc000f70 tx=0x7f47fc0012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48134c2640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa8a2e10 0x55a9fa99bc70 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48134c2640 1 -- 172.21.15.110:0/1475823033 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f480c00a4d0 con 0x55a9fa8a2e10 2024-04-27T03:19:09.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa99cfa0 0x55a9fa99f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa99f9c0 con 0x55a9fa99cfa0 2024-04-27T03:19:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f4812cc1640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa99cfa0 0x55a9fa99f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f4812cc1640 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa99cfa0 0x55a9fa99f390 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f4812cc1640 1 -- 172.21.15.110:0/1475823033 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa99cfa0 2024-04-27T03:19:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9a1270 con 0x55a9fa8a2e10 2024-04-27T03:19:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48134c2640 1 -- 172.21.15.110:0/1475823033 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f480c00a4d0 con 0x55a9fa8a2e10 2024-04-27T03:19:09.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9a04a0 con 0x55a9fa99cfa0 2024-04-27T03:19:09.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f4812cc1640 1 -- 172.21.15.110:0/1475823033 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa99cfa0 2024-04-27T03:19:09.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.327+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9a08c0 con 0x55a9fa8a2e10 2024-04-27T03:19:09.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48134c2640 1 -- 172.21.15.110:0/1475823033 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f480c00a4d0 con 0x55a9fa8a2e10 2024-04-27T03:19:09.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 Processor -- start 2024-04-27T03:19:09.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 -- start start 2024-04-27T03:19:09.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9afb80 0x55a9fa9affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9fa7a7310 con 0x55a9fa9afb80 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48134c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9afb80 0x55a9fa9affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48134c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9afb80 0x55a9fa9affa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40226/0 (socket says 172.21.15.110:40226) 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48134c2640 1 -- 172.21.15.110:0/1186201797 learned_addr learned my addr 172.21.15.110:0/1186201797 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48134c2640 1 -- 172.21.15.110:0/1186201797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fa9b04e0 con 0x55a9fa9afb80 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48134c2640 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9afb80 0x55a9fa9affa0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f480c00df90 tx=0x7f480c0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f46fce01640 1 -- 172.21.15.110:0/1186201797 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f480c0157d0 con 0x55a9fa9afb80 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a9fa9b0f80 con 0x55a9fa9afb80 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f46fce01640 1 -- 172.21.15.110:0/1186201797 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480c015970 con 0x55a9fa9afb80 2024-04-27T03:19:09.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f46fce01640 1 -- 172.21.15.110:0/1186201797 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f480c015c50 con 0x55a9fa9afb80 2024-04-27T03:19:09.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9fa9b1450 con 0x55a9fa9afb80 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f46fce01640 1 -- 172.21.15.110:0/1186201797 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f480c029440 con 0x55a9fa9afb80 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f46fce01640 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46ec034650 0x7f46ec036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f46fce01640 1 -- 172.21.15.110:0/1186201797 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f480c00e040 con 0x55a9fa9afb80 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f4812cc1640 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46ec034650 0x7f46ec036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 Processor -- start 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.331+0000 7f48151a6980 1 -- start start 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9bd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9fa98e500 con 0x55a9fa9bcd80 2024-04-27T03:19:09.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f4812cc1640 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46ec034650 0x7f46ec036b10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55a9fa99c760 tx=0x7f48040bf460 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9bd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9bd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40242/0 (socket says 172.21.15.110:40242) 2024-04-27T03:19:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 -- 172.21.15.110:0/3219726002 learned_addr learned my addr 172.21.15.110:0/3219726002 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 -- 172.21.15.110:0/3219726002 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fa9bd770 con 0x55a9fa9bcd80 2024-04-27T03:19:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 --2- 172.21.15.110:0/3219726002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9bd1a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f480c01e390 tx=0x7f480c01e480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918b2b3ec95e1e0c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f46f67fc640 1 -- 172.21.15.110:0/3219726002 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f480c03c250 con 0x55a9fa9bcd80 2024-04-27T03:19:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f46f67fc640 1 -- 172.21.15.110:0/3219726002 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480c041070 con 0x55a9fa9bcd80 2024-04-27T03:19:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f46f67fc640 1 -- 172.21.15.110:0/3219726002 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f480c012440 con 0x55a9fa9bcd80 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- 172.21.15.110:0/3219726002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 msgr2=0x55a9fa9bd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 --2- 172.21.15.110:0/3219726002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9bd1a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f480c01e390 tx=0x7f480c01e480 comp rx=0 tx=0).stop 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- 172.21.15.110:0/3219726002 shutdown_connections 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 --2- 172.21.15.110:0/3219726002 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9bd1a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- 172.21.15.110:0/3219726002 >> 172.21.15.110:0/3219726002 conn(0x55a9fa9b90c0 msgr2=0x55a9fa9b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- 172.21.15.110:0/3219726002 shutdown_connections 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- 172.21.15.110:0/3219726002 wait complete. 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 Processor -- start 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- start start 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48151a6980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9fa9bdef0 con 0x55a9fa9bcd80 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9c4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40246/0 (socket says 172.21.15.110:40246) 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 learned_addr learned my addr 172.21.15.110:0/3981692890 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9fa9c4780 con 0x55a9fa9bcd80 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.335+0000 7f48134c2640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9c4240 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f480c029b70 tx=0x7f480c04a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f46f4ff9640 1 -- 172.21.15.110:0/3981692890 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f480c0113a0 con 0x55a9fa9bcd80 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a9fa9c4a10 con 0x55a9fa9bcd80 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f46f4ff9640 1 -- 172.21.15.110:0/3981692890 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480c011540 con 0x55a9fa9bcd80 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f46f4ff9640 1 -- 172.21.15.110:0/3981692890 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f480c049320 con 0x55a9fa9bcd80 2024-04-27T03:19:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9fa9c76f0 con 0x55a9fa9bcd80 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f46f4ff9640 1 -- 172.21.15.110:0/3981692890 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f480c03a020 con 0x55a9fa9bcd80 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f46f4ff9640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46dc034650 0x7f46dc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f46f4ff9640 1 -- 172.21.15.110:0/3981692890 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f480c038020 con 0x55a9fa9bcd80 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f4812cc1640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46dc034650 0x7f46dc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa9cf720 0x55a9fa9d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9d2330 con 0x55a9fa9cf720 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48124c0640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa9cf720 0x55a9fa9d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f4812cc1640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46dc034650 0x7f46dc036b10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f48040c2ae0 tx=0x7f48040eb000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48124c0640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa9cf720 0x55a9fa9d1be0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f47fc00e5a0 con 0x55a9fa9cf720 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 realm 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9ceb70 con 0x55a9fa9cf720 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f47fc00e5a0 con 0x55a9fa9cf720 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9d2e30 con 0x55a9fa9cf720 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f47fc00e5a0 con 0x55a9fa9cf720 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa9d3c10 0x55a9fa9d6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9d66b0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.339+0000 7f48134c2640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa9d3c10 0x55a9fa9d6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48134c2640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa9d3c10 0x55a9fa9d6000 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48151a6980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9d6f00 con 0x55a9fa9cf720 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f47fc00e5a0 con 0x55a9fa9cf720 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48151a6980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48151a6980 20 searching for the correct realm 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9fa9d8c70 0x55a9fa9db060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f4812cc1640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9fa9d8c70 0x55a9fa9db060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f4812cc1640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9fa9d8c70 0x55a9fa9db060 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48134c2640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f480c033070 0x7f480c03c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x7f480c033070 2024-04-27T03:19:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48124c0640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f480c033070 0x7f480c03c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.343+0000 7f48124c0640 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f480c033070 0x7f480c03c580 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc00e5a0 con 0x7f480c033070 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc00e5a0 con 0x55a9fa9cf720 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x7f480c033070 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x7f480c033070 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x7f480c033070 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f47fc029040 con 0x7f480c033070 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.347+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x7f480c033070 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f47fc029040 con 0x7f480c033070 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x7f480c033070 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x7f480c033070 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9cf720 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x7f480c033070 2024-04-27T03:19:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x7f480c033070 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.351+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x7f480c033070 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f47fc029040 con 0x7f480c033070 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9db6e0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9d7850 con 0x55a9fa9cf720 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f47fc029040 con 0x55a9fa9cf720 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.355+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9d7460 con 0x55a9fa9d3c10 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9d8140 con 0x55a9fa9cf720 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 zone default found 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 4 Realm: () 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 zonegroup default 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dbb60 con 0x7f480c033070 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dd980 con 0x55a9fa9d8c70 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9ddd20 con 0x55a9fa9d8c70 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 started sync module instance, tier type = 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dce20 con 0x55a9fa9d3c10 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94188542480960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dca80 con 0x55a9fa9d3c10 2024-04-27T03:19:09.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.359+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dec00 con 0x55a9fa9cf720 2024-04-27T03:19:09.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94188542486352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9deb70 con 0x55a9fa9cf720 2024-04-27T03:19:09.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dfb60 con 0x55a9fa9d3c10 2024-04-27T03:19:09.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94188542480960] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94188542486352] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94188542488560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dfb60 con 0x55a9fa9d3c10 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.363+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dff90 con 0x7f480c033070 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94188542490176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9dff50 con 0x7f480c033070 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e0e50 con 0x55a9fa9d3c10 2024-04-27T03:19:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94188542488560] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94188542490176] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94188542493568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e09a0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e16f0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94188542497344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e09a0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e2430 con 0x55a9fa9d8c70 2024-04-27T03:19:09.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.367+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94188542493568] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94188542497344] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94188542484784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e1f40 con 0x55a9fa9d8c70 2024-04-27T03:19:09.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e2e40 con 0x7f480c033070 2024-04-27T03:19:09.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94188542484784] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'210 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94188542499744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e2e40 con 0x7f480c033070 2024-04-27T03:19:09.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=0 2024-04-27T03:19:09.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=1 2024-04-27T03:19:09.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=2 2024-04-27T03:19:09.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=3 2024-04-27T03:19:09.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=4 2024-04-27T03:19:09.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=5 2024-04-27T03:19:09.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=6 2024-04-27T03:19:09.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94188542499744] v29'211 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 20 add_watcher() i=7 2024-04-27T03:19:09.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:09.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e60b0 con 0x55a9fa9cf720 2024-04-27T03:19:09.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.371+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94188542503680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9fa9e33d0 con 0x55a9fa9cf720 2024-04-27T03:19:09.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94188542503680] v29'216 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f46c97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:09.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:09.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 5 note: GC not initialized 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f4643fff640 20 reqs_thread_entry: start 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 Filter name: none 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f4641ffb640 20 reqs_thread_entry: start 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 cache get: name=default.rgw.meta+users.uid+bar : miss 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.e 8:718f6a54:users.uid::bar:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa09a10 con 0x7f480c033070 2024-04-27T03:19:09.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(62 bar [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x0 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 adding default.rgw.meta+users.uid+bar to cache LRU end 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : miss 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa08050 con 0x55a9fa9cf720 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(63 NX5QOQKC6BH2IDN8HC7A [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 206+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x0 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 adding default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2024-04-27T03:19:09.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.375+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.e 8:718f6a54:users.uid::bar:head [delete,create,call version.set in=58b,writefull 0~380 in=380b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa08940 con 0x7f480c033070 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(64 bar [delete,create,call,writefull 0~380] v29'3 uv3 ondisk = 0) ==== 273+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48151a6980 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x17 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48151a6980 10 moving default.rgw.meta+users.uid+bar to cache LRU end 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48151a6980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.uid:bar, ofs0, ns] 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:65 7.9 7:93e5b521:::notify.7:head [notify cookie 94188542657680 in=568b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa0d4b0 con 0x7f480c033070 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== watch-notify(notify (1) cookie 94188542499744 notify 124554051594 ret 0) ==== 598+0+0 (crc 0 0 0) 0x55a9faa08ce0 con 0x7f480c033070 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f46f67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051594 cookie 94188542499744 notifier 4716 bl.length()=556 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f46f67fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar info.flags=0x17 2024-04-27T03:19:09.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f46f67fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar to cache LRU end 2024-04-27T03:19:09.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(65 notify.7 [notify cookie 94188542657680 out=8b] v0'0 uv210 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f46f67fc640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:66 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f46d8003eb0 con 0x7f480c033070 2024-04-27T03:19:09.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(66 notify.7 [notify-ack] v0'0 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== watch-notify(notify_complete (2) cookie 94188542657680 notify 124554051594 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a9faa08220 con 0x7f480c033070 2024-04-27T03:19:09.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.379+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa08440 con 0x55a9fa9cf720 2024-04-27T03:19:09.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(67 NX5QOQKC6BH2IDN8HC7A [delete,create,writefull 0~7] v29'1 uv1 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f48151a6980 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2024-04-27T03:19:09.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f48151a6980 10 moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-04-27T03:19:09.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f48151a6980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2024-04-27T03:19:09.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 7.3 7:c609908c:::notify.5:head [notify cookie 94188542657600 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa0d410 con 0x55a9fa9d8c70 2024-04-27T03:19:09.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== watch-notify(notify (1) cookie 94188542497344 notify 124554051604 ret 0) ==== 219+0+0 (crc 0 0 0) 0x7f48040db040 con 0x55a9fa9d8c70 2024-04-27T03:19:09.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f46f5ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051604 cookie 94188542497344 notifier 4716 bl.length()=177 2024-04-27T03:19:09.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f46f5ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2024-04-27T03:19:09.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f46f5ffb640 10 rgw watcher librados: moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-04-27T03:19:09.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(68 notify.5 [notify cookie 94188542657600 out=8b] v0'0 uv210 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f46f5ffb640 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f4648003410 con 0x55a9fa9d8c70 2024-04-27T03:19:09.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(69 notify.5 [notify-ack] v0'0 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== watch-notify(notify_complete (2) cookie 94188542657600 notify 124554051604 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a9fa998190 con 0x55a9fa9d8c70 2024-04-27T03:19:09.393 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"bar","display_name":"Bar","email":"","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"bar","access_key":"NX5QOQKC6BH2IDN8HC7A","secret_key":"LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn","active":true,"create_date":"2024-04-27T03:19:09.381256Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:09.381236Z","tags":[],"group_ids":[]} 2024-04-27T03:19:09.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f46c97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.383+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94188542503680] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x55a9fa9cf720 2024-04-27T03:19:09.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94188542503680] v29'217 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94188542480960] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94188542480960] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48151a6980 20 remove_watcher() i=0 2024-04-27T03:19:09.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48151a6980 2 removed watcher, disabling cache 2024-04-27T03:19:09.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94188542486352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x55a9fa9cf720 2024-04-27T03:19:09.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94188542486352] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x55a9fa9cf720 2024-04-27T03:19:09.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48151a6980 20 remove_watcher() i=1 2024-04-27T03:19:09.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.387+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94188542488560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.391+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94188542488560] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.391+0000 7f48151a6980 20 remove_watcher() i=2 2024-04-27T03:19:09.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.391+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94188542490176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x7f480c033070 2024-04-27T03:19:09.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.391+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94188542490176] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.391+0000 7f48151a6980 20 remove_watcher() i=3 2024-04-27T03:19:09.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.391+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94188542493568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x55a9fa9d3c10 2024-04-27T03:19:09.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.395+0000 7f48134c2640 1 -- 172.21.15.110:0/3981692890 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94188542493568] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f480c026020 con 0x55a9fa9d3c10 2024-04-27T03:19:09.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.395+0000 7f48151a6980 20 remove_watcher() i=4 2024-04-27T03:19:09.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.395+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94188542497344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.395+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94188542497344] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.395+0000 7f48151a6980 20 remove_watcher() i=5 2024-04-27T03:19:09.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.395+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94188542484784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x55a9fa9d8c70 2024-04-27T03:19:09.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f4812cc1640 1 -- 172.21.15.110:0/3981692890 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94188542484784] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f48040d2070 con 0x55a9fa9d8c70 2024-04-27T03:19:09.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 20 remove_watcher() i=6 2024-04-27T03:19:09.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94188542499744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a9faa076b0 con 0x7f480c033070 2024-04-27T03:19:09.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94188542499744] v29'212 uv210 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f47fc03c1c0 con 0x7f480c033070 2024-04-27T03:19:09.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 20 remove_watcher() i=7 2024-04-27T03:19:09.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa9d3c10 msgr2=0x55a9fa9d6000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa9d3c10 0x55a9fa9d6000 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa9cf720 msgr2=0x55a9fa9d1be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa9cf720 0x55a9fa9d1be0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f480c033070 msgr2=0x7f480c03c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f480c033070 0x7f480c03c580 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9fa9d8c70 msgr2=0x55a9fa9db060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9fa9d8c70 0x55a9fa9db060 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46dc034650 msgr2=0x7f46dc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46dc034650 0x7f46dc036b10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f48040c2ae0 tx=0x7f48040eb000 comp rx=0 tx=0).stop 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48124c0640 1 -- 172.21.15.110:0/3981692890 reap_dead start 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 msgr2=0x55a9fa9c4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9c4240 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f480c029b70 tx=0x7f480c04a820 comp rx=0 tx=0).stop 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 shutdown_connections 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/3981692890 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9bcd80 0x55a9fa9c4240 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 >> 172.21.15.110:0/3981692890 conn(0x55a9fa9b90c0 msgr2=0x55a9fa9b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 shutdown_connections 2024-04-27T03:19:09.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/3981692890 wait complete. 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46ec034650 msgr2=0x7f46ec036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46ec034650 0x7f46ec036b10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55a9fa99c760 tx=0x7f48040bf460 comp rx=0 tx=0).stop 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9afb80 msgr2=0x55a9fa9affa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9afb80 0x55a9fa9affa0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f480c00df90 tx=0x7f480c0178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 shutdown_connections 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46ec034650 0x7f46ec036b10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1186201797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa9afb80 0x55a9fa9affa0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 >> 172.21.15.110:0/1186201797 conn(0x55a9fa9a4990 msgr2=0x55a9fa9a54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 shutdown_connections 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1186201797 wait complete. 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa99cfa0 msgr2=0x55a9fa99f390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa99cfa0 0x55a9fa99f390 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa8a2e10 msgr2=0x55a9fa99bc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa8a2e10 0x55a9fa99bc70 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46f8034650 msgr2=0x7f46f8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46f8034650 0x7f46f8036b10 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f47fc000f70 tx=0x7f47fc0012c0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 msgr2=0x55a9fa994d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa994d90 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f4804017e90 tx=0x7f4804018020 comp rx=0 tx=0).stop 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 shutdown_connections 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9fa99cfa0 0x55a9fa99f390 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9fa8a2e10 0x55a9fa99bc70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f46f8034650 0x7f46f8036b10 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 --2- 172.21.15.110:0/1475823033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9fa8a24f0 0x55a9fa994d90 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 >> 172.21.15.110:0/1475823033 conn(0x55a9fa815bd0 msgr2=0x55a9fa988510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 shutdown_connections 2024-04-27T03:19:09.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.399+0000 7f48151a6980 1 -- 172.21.15.110:0/1475823033 wait complete. 2024-04-27T03:19:09.420 DEBUG:tasks.util.rgw: json result: {'user_id': 'bar', 'display_name': 'Bar', 'email': '', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'bar', 'access_key': 'NX5QOQKC6BH2IDN8HC7A', 'secret_key': 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn', 'active': True, 'create_date': '2024-04-27T03:19:09.381256Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:09.381236Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:09.436 INFO:tasks.radosgw_admin:RL: create_bucket myfoo6 bar bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:19:09.443 INFO:tasks.radosgw_admin:RL: put_obj myfoo6 bar bytes_out=8 bytes_in=0 failed=False 2024-04-27T03:19:09.443 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rename', '--uid', 'bar', '--new-uid', 'bud'] 2024-04-27T03:19:09.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rename', '--uid', 'bar', '--new-uid', 'bud'] 2024-04-27T03:19:09.444 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rename --uid bar --new-uid bud 2024-04-27T03:19:09.474 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:09.474 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d3039980 1 Processor -- start 2024-04-27T03:19:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d3039980 1 -- start start 2024-04-27T03:19:09.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d3039980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5569656bc9c0 0x5569656bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d3039980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5569655750d0 con 0x5569656b84a0 2024-04-27T03:19:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d3039980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55696556a430 con 0x5569656bc9c0 2024-04-27T03:19:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d0353640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5569656bc9c0 0x5569656bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d0b54640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5569656b84a0 0x5569656b8880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:36238/0 (socket says 172.21.15.110:36238) 2024-04-27T03:19:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d0b54640 1 -- 172.21.15.110:0/4118679285 learned_addr learned my addr 172.21.15.110:0/4118679285 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d0353640 1 -- 172.21.15.110:0/4118679285 >> v1:172.21.15.77:6789/0 conn(0x5569656b84a0 legacy=0x5569656b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d0353640 1 -- 172.21.15.110:0/4118679285 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x556965563510 con 0x5569656bc9c0 2024-04-27T03:19:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.467+0000 7f92d0353640 1 --2- 172.21.15.110:0/4118679285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656bc9c0 0x5569656bedb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f92c400a480 tx=0x7f92c401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53ad46b5db7e532 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92cfb52640 1 -- 172.21.15.110:0/4118679285 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c40b0070 con 0x5569656bc9c0 2024-04-27T03:19:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92cfb52640 1 -- 172.21.15.110:0/4118679285 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92c40ab540 con 0x5569656bc9c0 2024-04-27T03:19:09.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92cfb52640 1 -- 172.21.15.110:0/4118679285 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c40ab840 con 0x5569656bc9c0 2024-04-27T03:19:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92d3039980 1 -- 172.21.15.110:0/4118679285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656bc9c0 msgr2=0x5569656bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92d3039980 1 --2- 172.21.15.110:0/4118679285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656bc9c0 0x5569656bedb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f92c400a480 tx=0x7f92c401b280 comp rx=0 tx=0).stop 2024-04-27T03:19:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92d3039980 1 -- 172.21.15.110:0/4118679285 shutdown_connections 2024-04-27T03:19:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92d3039980 1 --2- 172.21.15.110:0/4118679285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656bc9c0 0x5569656bedb0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92d3039980 1 -- 172.21.15.110:0/4118679285 >> 172.21.15.110:0/4118679285 conn(0x55696562bba0 msgr2=0x55696562bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92d3039980 1 -- 172.21.15.110:0/4118679285 shutdown_connections 2024-04-27T03:19:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.471+0000 7f92d3039980 1 -- 172.21.15.110:0/4118679285 wait complete. 2024-04-27T03:19:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d3039980 1 Processor -- start 2024-04-27T03:19:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d3039980 1 -- start start 2024-04-27T03:19:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d3039980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d3039980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5569655b92d0 con 0x5569656b84a0 2024-04-27T03:19:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d0b54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d0b54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657a35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40264/0 (socket says 172.21.15.110:40264) 2024-04-27T03:19:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d0b54640 1 -- 172.21.15.110:0/3970215248 learned_addr learned my addr 172.21.15.110:0/3970215248 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d0b54640 1 -- 172.21.15.110:0/3970215248 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55696553fae0 con 0x5569656b84a0 2024-04-27T03:19:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f92d0b54640 1 --2- 172.21.15.110:0/3970215248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657a35f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f92c0001d90 tx=0x7f92c0001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614110a16852dd8a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f91bee0d640 1 -- 172.21.15.110:0/3970215248 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c0013030 con 0x5569656b84a0 2024-04-27T03:19:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f91bee0d640 1 -- 172.21.15.110:0/3970215248 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92c000e690 con 0x5569656b84a0 2024-04-27T03:19:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.483+0000 7f91bee0d640 1 -- 172.21.15.110:0/3970215248 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c000ea10 con 0x5569656b84a0 2024-04-27T03:19:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- 172.21.15.110:0/3970215248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 msgr2=0x5569657a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 --2- 172.21.15.110:0/3970215248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657a35f0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f92c0001d90 tx=0x7f92c0001e80 comp rx=0 tx=0).stop 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- 172.21.15.110:0/3970215248 shutdown_connections 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 --2- 172.21.15.110:0/3970215248 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657a35f0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- 172.21.15.110:0/3970215248 >> 172.21.15.110:0/3970215248 conn(0x55696562bba0 msgr2=0x55696579f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- 172.21.15.110:0/3970215248 shutdown_connections 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- 172.21.15.110:0/3970215248 wait complete. 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 Processor -- start 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- start start 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657aa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5569656c1e10 con 0x5569656b84a0 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d0b54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d0b54640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657aa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40278/0 (socket says 172.21.15.110:40278) 2024-04-27T03:19:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d0b54640 1 -- 172.21.15.110:0/3396051875 learned_addr learned my addr 172.21.15.110:0/3396051875 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d0b54640 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569657aab70 con 0x5569656b84a0 2024-04-27T03:19:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d0b54640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657aa630 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x5569657a4480 tx=0x7f92c0000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f91bd60a640 1 -- 172.21.15.110:0/3396051875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c0027070 con 0x5569656b84a0 2024-04-27T03:19:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f91bd60a640 1 -- 172.21.15.110:0/3396051875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92c0019900 con 0x5569656b84a0 2024-04-27T03:19:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5569657aae00 con 0x5569656b84a0 2024-04-27T03:19:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f91bd60a640 1 -- 172.21.15.110:0/3396051875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c0019c20 con 0x5569656b84a0 2024-04-27T03:19:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.487+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569657ada80 con 0x5569656b84a0 2024-04-27T03:19:09.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f91bd60a640 1 -- 172.21.15.110:0/3396051875 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f92c002a070 con 0x5569656b84a0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f91bd60a640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91b8034650 0x7f91b8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f91bd60a640 1 -- 172.21.15.110:0/3396051875 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f92c001f070 con 0x5569656b84a0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d0353640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91b8034650 0x7f91b8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569656bc9c0 0x5569657b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657b1b60 con 0x5569656bc9c0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d1355640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569656bc9c0 0x5569657b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d0353640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91b8034650 0x7f91b8036b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f92c40ad000 tx=0x7f92c40a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d1355640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569656bc9c0 0x5569657b1560 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d1355640 1 -- 172.21.15.110:0/3396051875 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f92c800a4d0 con 0x5569656bc9c0 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657b2930 0x5569657b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657b5360 con 0x5569657b2930 2024-04-27T03:19:09.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d0b54640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657b2930 0x5569657b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d0b54640 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657b2930 0x5569657b4d20 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d0b54640 1 -- 172.21.15.110:0/3396051875 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f92c0029030 con 0x5569657b2930 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657b6c10 con 0x5569656bc9c0 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d1355640 1 -- 172.21.15.110:0/3396051875 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f92c800a4d0 con 0x5569656bc9c0 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657b5e40 con 0x5569657b2930 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d0b54640 1 -- 172.21.15.110:0/3396051875 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f92c0029030 con 0x5569657b2930 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.491+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657b6260 con 0x5569656bc9c0 2024-04-27T03:19:09.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d1355640 1 -- 172.21.15.110:0/3396051875 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f92c800a4d0 con 0x5569656bc9c0 2024-04-27T03:19:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d3039980 1 Processor -- start 2024-04-27T03:19:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d3039980 1 -- start start 2024-04-27T03:19:09.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d3039980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657c5520 0x5569657c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d3039980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5569655bd310 con 0x5569657c5520 2024-04-27T03:19:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d1355640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657c5520 0x5569657c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d1355640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657c5520 0x5569657c5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40282/0 (socket says 172.21.15.110:40282) 2024-04-27T03:19:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d1355640 1 -- 172.21.15.110:0/1335621440 learned_addr learned my addr 172.21.15.110:0/1335621440 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d1355640 1 -- 172.21.15.110:0/1335621440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569657c5e80 con 0x5569657c5520 2024-04-27T03:19:09.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d1355640 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657c5520 0x5569657c5940 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f92c800df90 tx=0x7f92c80178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f91b67fc640 1 -- 172.21.15.110:0/1335621440 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c80157d0 con 0x5569657c5520 2024-04-27T03:19:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f91b67fc640 1 -- 172.21.15.110:0/1335621440 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92c8015970 con 0x5569657c5520 2024-04-27T03:19:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f91b67fc640 1 -- 172.21.15.110:0/1335621440 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c8015c50 con 0x5569657c5520 2024-04-27T03:19:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5569657c6920 con 0x5569657c5520 2024-04-27T03:19:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569657c6df0 con 0x5569657c5520 2024-04-27T03:19:09.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.495+0000 7f91b67fc640 1 -- 172.21.15.110:0/1335621440 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f92c8029440 con 0x5569657c5520 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f91b67fc640 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91a8034650 0x7f91a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f91b67fc640 1 -- 172.21.15.110:0/1335621440 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f92c800e040 con 0x5569657c5520 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d0b54640 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91a8034650 0x7f91a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 Processor -- start 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- start start 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d0b54640 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91a8034650 0x7f91a8036b10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5569657b20f0 tx=0x7f92c0048390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5569657a3d40 con 0x5569657d2720 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40296/0 (socket says 172.21.15.110:40296) 2024-04-27T03:19:09.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 -- 172.21.15.110:0/2603946297 learned_addr learned my addr 172.21.15.110:0/2603946297 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 -- 172.21.15.110:0/2603946297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569657d3110 con 0x5569657d2720 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 --2- 172.21.15.110:0/2603946297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d2b40 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f92c801e390 tx=0x7f92c801e480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f696ffb28eba56 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f91a7fff640 1 -- 172.21.15.110:0/2603946297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c8012440 con 0x5569657d2720 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f91a7fff640 1 -- 172.21.15.110:0/2603946297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92c8041070 con 0x5569657d2720 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f91a7fff640 1 -- 172.21.15.110:0/2603946297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c8029c90 con 0x5569657d2720 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- 172.21.15.110:0/2603946297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 msgr2=0x5569657d2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 --2- 172.21.15.110:0/2603946297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d2b40 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f92c801e390 tx=0x7f92c801e480 comp rx=0 tx=0).stop 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- 172.21.15.110:0/2603946297 shutdown_connections 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 --2- 172.21.15.110:0/2603946297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d2b40 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- 172.21.15.110:0/2603946297 >> 172.21.15.110:0/2603946297 conn(0x5569657cea60 msgr2=0x5569657ccac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- 172.21.15.110:0/2603946297 shutdown_connections 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- 172.21.15.110:0/2603946297 wait complete. 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 Processor -- start 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- start start 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d3039980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5569657d3890 con 0x5569657d2720 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:40312/0 (socket says 172.21.15.110:40312) 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 learned_addr learned my addr 172.21.15.110:0/2602831872 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.499+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569657da120 con 0x5569657d2720 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d1355640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d9be0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f92c80056d0 tx=0x7f92c804a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f91a67fc640 1 -- 172.21.15.110:0/2602831872 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c804c070 con 0x5569657d2720 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5569657da3b0 con 0x5569657d2720 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f91a67fc640 1 -- 172.21.15.110:0/2602831872 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92c8038070 con 0x5569657d2720 2024-04-27T03:19:09.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f91a67fc640 1 -- 172.21.15.110:0/2602831872 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92c804c340 con 0x5569657d2720 2024-04-27T03:19:09.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569657dd090 con 0x5569657d2720 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f91a67fc640 1 -- 172.21.15.110:0/2602831872 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f92c8032070 con 0x5569657d2720 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f91a67fc640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f919c034650 0x7f919c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f91a67fc640 1 -- 172.21.15.110:0/2602831872 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f92c80220c0 con 0x5569657d2720 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d0b54640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f919c034650 0x7f919c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569657e5280 0x5569657e7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657e7e70 con 0x5569657e5280 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d0353640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569657e5280 0x5569657e7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d0b54640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f919c034650 0x7f919c036b10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f92c000c010 tx=0x7f92c003a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d0353640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569657e5280 0x5569657e7720 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 realm 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657e82a0 con 0x5569657e5280 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657e8b70 con 0x5569657e5280 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657e9590 0x5569657eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657ec130 con 0x5569657e9590 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.503+0000 7f92d1355640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657e9590 0x5569657eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d1355640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657e9590 0x5569657eb9c0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d3039980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657ec980 con 0x5569657e5280 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d3039980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d3039980 20 searching for the correct realm 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5569657ee6f0 0x5569657f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657ee6f0 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0b54640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5569657ee6f0 0x5569657f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0b54640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5569657ee6f0 0x5569657f0ae0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d1355640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f92c8057040 0x7f92c8011730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:09.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x7f92c8057040 2024-04-27T03:19:09.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f92c8057040 0x7f92c8011730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:09.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f92c8057040 0x7f92c8011730 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:09.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.507+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657ee6f0 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x7f92c8057040 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x7f92c8057040 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657ee6f0 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.511+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657ee6f0 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x7f92c8057040 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657ee6f0 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x7f92c8057040 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e5280 2024-04-27T03:19:09.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x7f92c8057040 2024-04-27T03:19:09.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657ee6f0 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x7f92c8057040 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.515+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657ee6f0 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1160 con 0x5569657e9590 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657ed330 con 0x5569657e5280 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657edbd0 con 0x5569657e9590 2024-04-27T03:19:09.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657ecf70 con 0x5569657e5280 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 zone default found 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 4 Realm: () 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 zonegroup default 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657f1a70 con 0x7f92c8057040 2024-04-27T03:19:09.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.519+0000 7f92d3039980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657f3840 con 0x5569657ee6f0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657f3c10 con 0x5569657ee6f0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 20 started sync module instance, tier type = 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f2ce0 con 0x5569657e9590 2024-04-27T03:19:09.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93911162761424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f2940 con 0x5569657e9590 2024-04-27T03:19:09.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f4b40 con 0x5569657e5280 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93911162766992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f4f60 con 0x5569657e5280 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.523+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f59d0 con 0x5569657e9590 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93911162761424] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 93911162766992] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93911162770592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f4ab0 con 0x5569657e9590 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f6270 con 0x7f92c8057040 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93911162772928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f4ab0 con 0x7f92c8057040 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f6d40 con 0x5569657e9590 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93911162770592] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.527+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 93911162772928] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93911162773984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f6d00 con 0x5569657e9590 2024-04-27T03:19:09.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f7130 con 0x5569657ee6f0 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93911162765424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f79d0 con 0x5569657ee6f0 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f8390 con 0x5569657ee6f0 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93911162773984] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93911162765424] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93911162781408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f79d0 con 0x5569657ee6f0 2024-04-27T03:19:09.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.531+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f8380 con 0x7f92c8057040 2024-04-27T03:19:09.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'213 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93911162781408] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93911162783616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f9130 con 0x7f92c8057040 2024-04-27T03:19:09.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=0 2024-04-27T03:19:09.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=1 2024-04-27T03:19:09.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=2 2024-04-27T03:19:09.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=3 2024-04-27T03:19:09.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=4 2024-04-27T03:19:09.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=5 2024-04-27T03:19:09.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=6 2024-04-27T03:19:09.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 93911162783616] v29'214 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 add_watcher() i=7 2024-04-27T03:19:09.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:09.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569657f9130 con 0x5569657e5280 2024-04-27T03:19:09.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93911162784816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5569657f9090 con 0x5569657e5280 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93911162784816] v29'218 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f9186fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.535+0000 7f92d3039980 5 note: GC not initialized 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f9185fe3640 20 reqs_thread_entry: start 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 20 Filter name: none 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f90f77fe640 20 reqs_thread_entry: start 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 cache get: name=default.rgw.meta+users.uid+bar : miss 2024-04-27T03:19:09.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.e 8:718f6a54:users.uid::bar:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55696581f8b0 con 0x7f92c8057040 2024-04-27T03:19:09.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(62 bar [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv3 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:19:09.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 cache put: name=default.rgw.meta+users.uid+bar info.flags=0x13 2024-04-27T03:19:09.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 adding default.rgw.meta+users.uid+bar to cache LRU end 2024-04-27T03:19:09.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:09.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.4 8:267425a8:users.uid::bud:head [create,call version.set in=58b,writefull 0~206 in=206b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x556965823c30 con 0x7f92c8057040 2024-04-27T03:19:09.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(63 bud [create,call,writefull 0~206] v29'1 uv1 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2024-04-27T03:19:09.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2024-04-27T03:19:09.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:bud, ofs0, ns] 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 7.3 7:c609908c:::notify.5:head [notify cookie 93911162959648 in=394b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965824820 con 0x5569657ee6f0 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== watch-notify(notify (1) cookie 93911162765424 notify 124554051606 ret 0) ==== 424+0+0 (crc 0 0 0) 0x5569657ada80 con 0x5569657ee6f0 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f91a7fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051606 cookie 93911162765424 notifier 4723 bl.length()=382 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f91a7fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f91a7fff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+bud to cache LRU end 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(64 notify.5 [notify cookie 93911162959648 out=8b] v0'0 uv213 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f91a7fff640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:65 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f9194003eb0 con 0x5569657ee6f0 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(65 notify.5 [notify-ack] v0'0 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== watch-notify(notify_complete (2) cookie 93911162959648 notify 124554051606 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5569657aae00 con 0x5569657ee6f0 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 8.5 8:a00000ee:users.uid::bar.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569658247a0 con 0x5569657e5280 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(66 bar.buckets [call out=184b] v0'0 uv5 ondisk = 0) ==== 155+0+184 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 : miss 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 20 rados->read ofs=0 len=0 2024-04-27T03:19:09.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 8.a 8:557ee8e5:root::.bucket.meta.myfoo6%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.9:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965821c00 con 0x5569657e5280 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(67 .bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 [call out=48b,stat out=16b,read 0~380 out=380b,getxattrs out=218b] v0'0 uv2 ondisk = 0) ==== 333+0+662 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 info.flags=0x17 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 adding default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 to cache LRU end 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.539+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 8.a 8:557ee8e5:root::.bucket.meta.myfoo6%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.9:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~380 in=380b,setxattr user.rgw.acl (117) in=129b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696582b840 con 0x5569657e5280 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(68 .bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 [delete,create,call,call,writefull 0~380,setxattr (117)] v29'3 uv3 ondisk = 0) ==== 417+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 info.flags=0x17 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 10 moving default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 to cache LRU end 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 10 updating xattr: name=user.rgw.acl bl.length()=117 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9, ofs0, ns] 2024-04-27T03:19:09.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [notify cookie 93911162962512 in=760b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965825c60 con 0x5569657ee6f0 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== watch-notify(notify (1) cookie 93911162765424 notify 124554051607 ret 0) ==== 790+0+0 (crc 0 0 0) 0x5569657aae00 con 0x5569657ee6f0 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f91a77fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051607 cookie 93911162765424 notifier 4723 bl.length()=748 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(69 notify.5 [notify cookie 93911162962512 out=8b] v0'0 uv213 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f91a77fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 info.flags=0x17 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f91a77fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 to cache LRU end 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f91a77fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=117 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f91a77fe640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f9104003540 con 0x5569657ee6f0 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(70 notify.5 [notify-ack] v0'0 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== watch-notify(notify_complete (2) cookie 93911162962512 notify 124554051607 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5569657aab70 con 0x5569657ee6f0 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): orig_sources= new_sources= 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): potential sources added=[] removed=[] 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): orig_dests= new_dests= 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): potential dests added=[] removed=[] 2024-04-27T03:19:09.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.remove_bucket in=145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696582b030 con 0x5569657e9590 2024-04-27T03:19:09.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(71 bud.buckets [call] v29'3 uv0 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.543+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.set_buckets_info in=208b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696582b030 con 0x5569657e9590 2024-04-27T03:19:09.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(72 bud.buckets [call] v29'4 uv4 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:73 8.18 8:1944a257:root::myfoo6:head [delete,create,call version.inc in=38b,writefull 0~166 in=166b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696582b030 con 0x5569657e5280 2024-04-27T03:19:09.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(73 myfoo6 [delete,create,call,writefull 0~166] v29'19 uv19 ondisk = 0) ==== 276+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d3039980 10 cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x7 2024-04-27T03:19:09.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d3039980 10 adding default.rgw.meta+root+myfoo6 to cache LRU end 2024-04-27T03:19:09.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d3039980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:myfoo6, ofs0, ns] 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [notify cookie 93911162962480 in=328b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55696582b030 con 0x7f92c8057040 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== watch-notify(notify (1) cookie 93911162772928 notify 124554051596 ret 0) ==== 358+0+0 (crc 0 0 0) 0x556965821df0 con 0x7f92c8057040 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f91a77fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051596 cookie 93911162772928 notifier 4723 bl.length()=316 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f91a77fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x7 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f91a77fe640 10 rgw watcher librados: moving default.rgw.meta+root+myfoo6 to cache LRU end 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f91a77fe640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:75 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f9104003570 con 0x7f92c8057040 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.547+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(74 notify.3 [notify cookie 93911162962480 out=8b] v0'0 uv213 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(75 notify.3 [notify-ack] v0'0 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== watch-notify(notify_complete (2) cookie 93911162962480 notify 124554051596 ret 0) ==== 42+0+48 (crc 0 0 0) 0x556965824900 con 0x7f92c8057040 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 : hit (requested=0x17, cached=0x17) 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 2024-04-27T03:19:09.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 8.a 8:557ee8e5:root::.bucket.meta.myfoo6%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.9:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~380 in=380b,setxattr user.rgw.acl (117) in=129b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x556965829470 con 0x5569657e5280 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(76 .bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 [delete,create,call,call,writefull 0~380,setxattr (117)] v29'4 uv4 ondisk = 0) ==== 417+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 info.flags=0x17 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 10 moving default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 to cache LRU end 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 10 updating xattr: name=user.rgw.acl bl.length()=117 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9, ofs0, ns] 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.3 7:c609908c:::notify.5:head [notify cookie 93911162962480 in=760b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5569658293f0 con 0x5569657ee6f0 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== watch-notify(notify (1) cookie 93911162765424 notify 124554051608 ret 0) ==== 790+0+0 (crc 0 0 0) 0x5569657aab70 con 0x5569657ee6f0 2024-04-27T03:19:09.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f91a7fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051608 cookie 93911162765424 notifier 4723 bl.length()=748 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f91a7fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 info.flags=0x17 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f91a7fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 to cache LRU end 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f91a7fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=117 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(77 notify.5 [notify cookie 93911162962480 out=8b] v0'0 uv213 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f91a7fff640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:78 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f9194003eb0 con 0x5569657ee6f0 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(78 notify.5 [notify-ack] v0'0 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== watch-notify(notify_complete (2) cookie 93911162962480 notify 124554051608 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55696553fae0 con 0x5569657ee6f0 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): orig_sources= new_sources= 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): potential sources added=[] removed=[] 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): orig_dests= new_dests= 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 20 handle_bi_update(): bucket=:myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]): potential dests added=[] removed=[] 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:79 4.27 4:e5f3ed89:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657e5280 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(79 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:80 4.14 4:28eb2f22:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657ee6f0 2024-04-27T03:19:09.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(80 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:81 4.d 4:b1be13f2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657e9590 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(81 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.551+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:82 4.11 4:8931bf6f:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x7f92c8057040 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(82 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:83 4.7 4:e007ada1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657e5280 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(83 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:84 4.c 4:33587581:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657e5280 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(84 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:85 4.3b 4:df87161c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657ee6f0 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(85 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6 [call out=85b] v0'0 uv10 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:86 4.3 4:c1fac81b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657e5280 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(86 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:87 4.b 4:d31bf6c6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x7f92c8057040 2024-04-27T03:19:09.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(87 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8 [call out=85b] v0'0 uv11 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:88 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657e5280 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(88 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [call out=378b] v0'0 uv5 ondisk = 0) ==== 194+0+378 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 10 cls_bucket_list_unordered: got thirteen 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:89 4.10 4:0b987959:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657ee6f0 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(89 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 get_obj_state: octx=0x5569657d2b00 obj=myfoo6:thirteen state=0x55696582a138 s->prefetch_data=0 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:90 3.3as0 3:5d3a90bb:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x556965827f30 con 0x5569657e9590 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(90 27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen [getxattrs out=856b,stat out=16b] v0'0 uv1 ondisk = 0) ==== 238+0+872 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 10 manifest: total_size = 8 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.14928272401315155342 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.acl 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.content_type 2024-04-27T03:19:09.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.etag 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.idtag 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.manifest 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.pg_ver 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.source_zone 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 Read xattr rgw_rados: user.rgw.tail_tag 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 get_obj_state: octx=0x5569657d2b00 obj=myfoo6:thirteen state=0x55696582a138 s->prefetch_data=0 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 : hit (requested=0x17, cached=0x17) 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 2024-04-27T03:19:09.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.555+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:91 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=187b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x556965821670 con 0x5569657e5280 2024-04-27T03:19:09.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.559+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(91 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [stat,call,call] v29'6 uv6 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.559+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:92 3.3as0 3:5d3a90bb:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen:head [cmpxattr user.rgw.idtag (63) op 1 mode 1 in=77b,setxattr user.rgw.acl (117) in=129b,setxattr user.rgw.idtag (33) in=47b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x556965821670 con 0x5569657e9590 2024-04-27T03:19:09.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(92 27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen [cmpxattr (63) op 1 mode 1,setxattr (117),setxattr (33)] v29'2 uv2 ondisk = 0) ==== 280+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:93 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=267b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696582e450 con 0x5569657e5280 2024-04-27T03:19:09.572 INFO:teuthology.orchestra.run.smithi110.stderr:1 objects processed in :myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9]). Next marker thirteen 2024-04-27T03:19:09.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.4 8:267425a8:users.uid::bud:head [delete,create,call version.set in=58b,writefull 0~380 in=380b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696582e450 con 0x7f92c8057040 2024-04-27T03:19:09.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(94 bud [delete,create,call,writefull 0~380] v29'2 uv2 ondisk = 0) ==== 273+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 10 moving default.rgw.meta+users.uid+bud to cache LRU end 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:bud, ofs0, ns] 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:95 7.3 7:c609908c:::notify.5:head [notify cookie 93911163005488 in=568b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55696582e450 con 0x5569657ee6f0 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== watch-notify(notify (1) cookie 93911162765424 notify 124554051609 ret 0) ==== 598+0+0 (crc 0 0 0) 0x55696553fae0 con 0x5569657ee6f0 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051609 cookie 93911162765424 notifier 4723 bl.length()=556 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bud info.flags=0x17 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+bud to cache LRU end 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(95 notify.5 [notify cookie 93911163005488 out=8b] v0'0 uv213 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:96 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f91040035d0 con 0x5569657ee6f0 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(96 notify.5 [notify-ack] v0'0 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== watch-notify(notify_complete (2) cookie 93911163005488 notify 124554051609 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5569655750d0 con 0x5569657ee6f0 2024-04-27T03:19:09.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(93 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [stat,call,call] v29'7 uv7 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a7fff640 20 handle_completion(): completion ok for obj=thirteen 2024-04-27T03:19:09.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:97 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x556965829f50 con 0x5569657e5280 2024-04-27T03:19:09.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(97 NX5QOQKC6BH2IDN8HC7A [delete,create,writefull 0~7] v29'5 uv5 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2024-04-27T03:19:09.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 10 adding default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-04-27T03:19:09.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2024-04-27T03:19:09.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:98 7.3 7:c609908c:::notify.5:head [notify cookie 93911163005488 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55696582a400 con 0x5569657ee6f0 2024-04-27T03:19:09.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== watch-notify(notify (1) cookie 93911162765424 notify 124554051610 ret 0) ==== 219+0+0 (crc 0 0 0) 0x5569655750d0 con 0x5569657ee6f0 2024-04-27T03:19:09.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051610 cookie 93911162765424 notifier 4723 bl.length()=177 2024-04-27T03:19:09.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2024-04-27T03:19:09.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 10 rgw watcher librados: moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2024-04-27T03:19:09.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f91a77fe640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:99 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f91040035d0 con 0x5569657ee6f0 2024-04-27T03:19:09.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.563+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(98 notify.5 [notify cookie 93911163005488 out=8b] v0'0 uv213 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(99 notify.5 [notify-ack] v0'0 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== watch-notify(notify_complete (2) cookie 93911163005488 notify 124554051610 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f92c002ace0 con 0x5569657ee6f0 2024-04-27T03:19:09.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d3039980 10 removing user index: bar 2024-04-27T03:19:09.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:100 8.e 8:718f6a54:users.uid::bar:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x556965829f50 con 0x7f92c8057040 2024-04-27T03:19:09.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 22 ==== osd_op_reply(100 bar [delete] v29'4 uv4 ondisk = 0) ==== 147+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d3039980 10 removing default.rgw.meta+users.uid+bar from cache 2024-04-27T03:19:09.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d3039980 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.uid:bar, ofs0, ns] 2024-04-27T03:19:09.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:101 7.9 7:93e5b521:::notify.7:head [notify cookie 93911163005408 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55696582a400 con 0x7f92c8057040 2024-04-27T03:19:09.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 23 ==== watch-notify(notify (1) cookie 93911162783616 notify 124554051597 ret 0) ==== 194+0+0 (crc 0 0 0) 0x55696582e7f0 con 0x7f92c8057040 2024-04-27T03:19:09.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f91a7fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051597 cookie 93911162783616 notifier 4723 bl.length()=152 2024-04-27T03:19:09.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f91a7fff640 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 7.9 7:93e5b521:::notify.7:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f9194003eb0 con 0x7f92c8057040 2024-04-27T03:19:09.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 24 ==== osd_op_reply(101 notify.7 [notify cookie 93911163005408 out=8b] v0'0 uv213 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 25 ==== osd_op_reply(102 notify.7 [notify-ack] v0'0 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.567+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 26 ==== watch-notify(notify_complete (2) cookie 93911163005408 notify 124554051597 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55696582af60 con 0x7f92c8057040 2024-04-27T03:19:09.583 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"bud","display_name":"Bar","email":"","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"bud","access_key":"NX5QOQKC6BH2IDN8HC7A","secret_key":"LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn","active":true,"create_date":"2024-04-27T03:19:09.381256Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:09.381236Z","tags":[],"group_ids":[]} 2024-04-27T03:19:09.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.571+0000 7f9186fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:09.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.571+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93911162784816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x5569657e5280 2024-04-27T03:19:09.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.571+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(103 data_loggenerations_metadata [watch unwatch cookie 93911162784816] v29'219 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.571+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93911162761424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x5569657e9590 2024-04-27T03:19:09.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 93911162761424] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d3039980 20 remove_watcher() i=0 2024-04-27T03:19:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d3039980 2 removed watcher, disabling cache 2024-04-27T03:19:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93911162766992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x5569657e5280 2024-04-27T03:19:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 93911162766992] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x5569657e5280 2024-04-27T03:19:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d3039980 20 remove_watcher() i=1 2024-04-27T03:19:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93911162770592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x5569657e9590 2024-04-27T03:19:09.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 93911162770592] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d3039980 20 remove_watcher() i=2 2024-04-27T03:19:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.575+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93911162772928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x7f92c8057040 2024-04-27T03:19:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 27 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 93911162772928] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d3039980 20 remove_watcher() i=3 2024-04-27T03:19:09.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93911162773984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x5569657e9590 2024-04-27T03:19:09.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d1355640 1 -- 172.21.15.110:0/2602831872 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 93911162773984] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c8026020 con 0x5569657e9590 2024-04-27T03:19:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d3039980 20 remove_watcher() i=4 2024-04-27T03:19:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93911162765424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x5569657ee6f0 2024-04-27T03:19:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 93911162765424] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d3039980 20 remove_watcher() i=5 2024-04-27T03:19:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.579+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93911162781408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x5569657ee6f0 2024-04-27T03:19:09.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d0b54640 1 -- 172.21.15.110:0/2602831872 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 93911162781408] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c0029030 con 0x5569657ee6f0 2024-04-27T03:19:09.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 20 remove_watcher() i=6 2024-04-27T03:19:09.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93911162783616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55696581d5e0 con 0x7f92c8057040 2024-04-27T03:19:09.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 <== osd.2 v2:172.21.15.110:6808/3575017494 28 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 93911162783616] v29'215 uv213 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f92c40b3040 con 0x7f92c8057040 2024-04-27T03:19:09.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 20 remove_watcher() i=7 2024-04-27T03:19:09.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657e9590 msgr2=0x5569657eb9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657e9590 0x5569657eb9c0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569657e5280 msgr2=0x5569657e7720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569657e5280 0x5569657e7720 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f92c8057040 msgr2=0x7f92c8011730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f92c8057040 0x7f92c8011730 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5569657ee6f0 msgr2=0x5569657f0ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5569657ee6f0 0x5569657f0ae0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f919c034650 msgr2=0x7f919c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f919c034650 0x7f919c036b10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f92c000c010 tx=0x7f92c003a000 comp rx=0 tx=0).stop 2024-04-27T03:19:09.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 msgr2=0x5569657d9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d0353640 1 -- 172.21.15.110:0/2602831872 reap_dead start 2024-04-27T03:19:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d9be0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f92c80056d0 tx=0x7f92c804a820 comp rx=0 tx=0).stop 2024-04-27T03:19:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 shutdown_connections 2024-04-27T03:19:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/2602831872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657d2720 0x5569657d9be0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 >> 172.21.15.110:0/2602831872 conn(0x5569657cea60 msgr2=0x5569657cc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 shutdown_connections 2024-04-27T03:19:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/2602831872 wait complete. 2024-04-27T03:19:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91a8034650 msgr2=0x7f91a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91a8034650 0x7f91a8036b10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x5569657b20f0 tx=0x7f92c0048390 comp rx=0 tx=0).stop 2024-04-27T03:19:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657c5520 msgr2=0x5569657c5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657c5520 0x5569657c5940 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f92c800df90 tx=0x7f92c80178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 shutdown_connections 2024-04-27T03:19:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91a8034650 0x7f91a8036b10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/1335621440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569657c5520 0x5569657c5940 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 >> 172.21.15.110:0/1335621440 conn(0x5569657ba330 msgr2=0x5569657bae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 shutdown_connections 2024-04-27T03:19:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/1335621440 wait complete. 2024-04-27T03:19:09.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657b2930 msgr2=0x5569657b4d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657b2930 0x5569657b4d20 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569656bc9c0 msgr2=0x5569657b1560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569656bc9c0 0x5569657b1560 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91b8034650 msgr2=0x7f91b8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91b8034650 0x7f91b8036b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f92c40ad000 tx=0x7f92c40a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 msgr2=0x5569657aa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657aa630 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x5569657a4480 tx=0x7f92c0000bd0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 shutdown_connections 2024-04-27T03:19:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5569657b2930 0x5569657b4d20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5569656bc9c0 0x5569657b1560 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f91b8034650 0x7f91b8036b10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 --2- 172.21.15.110:0/3396051875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5569656b84a0 0x5569657aa630 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 >> 172.21.15.110:0/3396051875 conn(0x55696562bba0 msgr2=0x55696579dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 shutdown_connections 2024-04-27T03:19:09.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:09.583+0000 7f92d3039980 1 -- 172.21.15.110:0/3396051875 wait complete. 2024-04-27T03:19:09.609 DEBUG:tasks.util.rgw: json result: {'user_id': 'bud', 'display_name': 'Bar', 'email': '', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'bud', 'access_key': 'NX5QOQKC6BH2IDN8HC7A', 'secret_key': 'LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn', 'active': True, 'create_date': '2024-04-27T03:19:09.381256Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:09.381236Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:14.621 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:19:14.621 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=8 failed=False 2024-04-27T03:19:14.621 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '4'] 2024-04-27T03:19:14.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--access-key', 'p5YnriCv1nAtykxBrupQ', '--secret', 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', '--max-buckets', '4'] 2024-04-27T03:19:14.621 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid fud --display-name Fud --access-key p5YnriCv1nAtykxBrupQ --secret Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh --max-buckets 4 2024-04-27T03:19:14.668 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:14.668 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:14.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2e94c980 1 Processor -- start 2024-04-27T03:19:14.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2e94c980 1 -- start start 2024-04-27T03:19:14.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2e94c980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5561fb162000 0x5561fb1f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2e94c980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5561fb0ad0d0 con 0x5561fb1f38e0 2024-04-27T03:19:14.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2e94c980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5561fb0a2430 con 0x5561fb162000 2024-04-27T03:19:14.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2c467640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5561fb162000 0x5561fb1f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2bc66640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5561fb1f38e0 0x5561fb1f5cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51272/0 (socket says 172.21.15.110:51272) 2024-04-27T03:19:14.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2bc66640 1 -- 172.21.15.110:0/636400819 learned_addr learned my addr 172.21.15.110:0/636400819 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:14.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2b465640 1 -- 172.21.15.110:0/636400819 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107167018 0 0) 0x5561fb0ad0d0 con 0x5561fb1f38e0 2024-04-27T03:19:14.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2b465640 1 -- 172.21.15.110:0/636400819 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe18001e20 con 0x5561fb1f38e0 2024-04-27T03:19:14.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2c467640 1 -- 172.21.15.110:0/636400819 >> v1:172.21.15.77:6789/0 conn(0x5561fb1f38e0 legacy=0x5561fb1f5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.659+0000 7efe2c467640 1 -- 172.21.15.110:0/636400819 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561fb09b510 con 0x5561fb162000 2024-04-27T03:19:14.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2c467640 1 --2- 172.21.15.110:0/636400819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb1f33a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efe2400a420 tx=0x7efe2401afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b0020cfdeea15a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2b465640 1 -- 172.21.15.110:0/636400819 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe240af070 con 0x5561fb162000 2024-04-27T03:19:14.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2b465640 1 -- 172.21.15.110:0/636400819 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe240aa530 con 0x5561fb162000 2024-04-27T03:19:14.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2b465640 1 -- 172.21.15.110:0/636400819 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe240aa850 con 0x5561fb162000 2024-04-27T03:19:14.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2e94c980 1 -- 172.21.15.110:0/636400819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 msgr2=0x5561fb1f33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2e94c980 1 --2- 172.21.15.110:0/636400819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb1f33a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efe2400a420 tx=0x7efe2401afd0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2e94c980 1 -- 172.21.15.110:0/636400819 shutdown_connections 2024-04-27T03:19:14.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2e94c980 1 --2- 172.21.15.110:0/636400819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb1f33a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2e94c980 1 -- 172.21.15.110:0/636400819 >> 172.21.15.110:0/636400819 conn(0x5561fb1e8a80 msgr2=0x5561fb1eaea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2e94c980 1 -- 172.21.15.110:0/636400819 shutdown_connections 2024-04-27T03:19:14.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.663+0000 7efe2e94c980 1 -- 172.21.15.110:0/636400819 wait complete. 2024-04-27T03:19:14.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2e94c980 1 Processor -- start 2024-04-27T03:19:14.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2e94c980 1 -- start start 2024-04-27T03:19:14.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2e94c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2db540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2e94c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5561fb0f12d0 con 0x5561fb162000 2024-04-27T03:19:14.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2c467640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2db540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2c467640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2db540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45856/0 (socket says 172.21.15.110:45856) 2024-04-27T03:19:14.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2c467640 1 -- 172.21.15.110:0/183030298 learned_addr learned my addr 172.21.15.110:0/183030298 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.679+0000 7efe2c467640 1 -- 172.21.15.110:0/183030298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561fb077ae0 con 0x5561fb162000 2024-04-27T03:19:14.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2c467640 1 --2- 172.21.15.110:0/183030298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2db540 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efe2400a420 tx=0x7efe240033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73fafb00f53a8a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efd1a60c640 1 -- 172.21.15.110:0/183030298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe240af050 con 0x5561fb162000 2024-04-27T03:19:14.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efd1a60c640 1 -- 172.21.15.110:0/183030298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe240b0040 con 0x5561fb162000 2024-04-27T03:19:14.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efd1a60c640 1 -- 172.21.15.110:0/183030298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe240aab00 con 0x5561fb162000 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- 172.21.15.110:0/183030298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 msgr2=0x5561fb2db540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 --2- 172.21.15.110:0/183030298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2db540 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7efe2400a420 tx=0x7efe240033b0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- 172.21.15.110:0/183030298 shutdown_connections 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 --2- 172.21.15.110:0/183030298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2db540 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- 172.21.15.110:0/183030298 >> 172.21.15.110:0/183030298 conn(0x5561fb1e8a80 msgr2=0x5561fb2d70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- 172.21.15.110:0/183030298 shutdown_connections 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- 172.21.15.110:0/183030298 wait complete. 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 Processor -- start 2024-04-27T03:19:14.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- start start 2024-04-27T03:19:14.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2e2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5561fb1f9e50 con 0x5561fb162000 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2c467640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2e2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2c467640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2e2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45860/0 (socket says 172.21.15.110:45860) 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2c467640 1 -- 172.21.15.110:0/1957018308 learned_addr learned my addr 172.21.15.110:0/1957018308 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2c467640 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561fb2e2a60 con 0x5561fb162000 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2c467640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2e2520 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efe2401b010 tx=0x7efe24019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efd18e09640 1 -- 172.21.15.110:0/1957018308 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe240c5070 con 0x5561fb162000 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5561fb2e2cf0 con 0x5561fb162000 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efd18e09640 1 -- 172.21.15.110:0/1957018308 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe240b6b80 con 0x5561fb162000 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efd18e09640 1 -- 172.21.15.110:0/1957018308 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe240c1480 con 0x5561fb162000 2024-04-27T03:19:14.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.683+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561fb2e5920 con 0x5561fb162000 2024-04-27T03:19:14.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efd18e09640 1 -- 172.21.15.110:0/1957018308 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7efe240c8070 con 0x5561fb162000 2024-04-27T03:19:14.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efd18e09640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd14034650 0x7efd14036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efd18e09640 1 -- 172.21.15.110:0/1957018308 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7efe240bb070 con 0x5561fb162000 2024-04-27T03:19:14.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2bc66640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd14034650 0x7efd14036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb1f38e0 0x5561fb2e9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb2e9ab0 con 0x5561fb1f38e0 2024-04-27T03:19:14.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2cc68640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb1f38e0 0x5561fb2e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2bc66640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd14034650 0x7efd14036b10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efe14001dc0 tx=0x7efe14001360 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2cc68640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb1f38e0 0x5561fb2e9570 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2cc68640 1 -- 172.21.15.110:0/1957018308 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7efe1c0083f0 con 0x5561fb1f38e0 2024-04-27T03:19:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb2ea560 0x5561fb2eca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb2ed190 con 0x5561fb2ea560 2024-04-27T03:19:14.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2c467640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb2ea560 0x5561fb2eca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.687+0000 7efe2c467640 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb2ea560 0x5561fb2eca20 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2c467640 1 -- 172.21.15.110:0/1957018308 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb2ea560 2024-04-27T03:19:14.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb2eea40 con 0x5561fb1f38e0 2024-04-27T03:19:14.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2cc68640 1 -- 172.21.15.110:0/1957018308 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7efe1c0083f0 con 0x5561fb1f38e0 2024-04-27T03:19:14.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb2edc70 con 0x5561fb2ea560 2024-04-27T03:19:14.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2c467640 1 -- 172.21.15.110:0/1957018308 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb2ea560 2024-04-27T03:19:14.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb2ee090 con 0x5561fb1f38e0 2024-04-27T03:19:14.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2cc68640 1 -- 172.21.15.110:0/1957018308 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7efe1c0083f0 con 0x5561fb1f38e0 2024-04-27T03:19:14.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 Processor -- start 2024-04-27T03:19:14.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 -- start start 2024-04-27T03:19:14.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb2fd350 0x5561fb2fd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5561fb0f5310 con 0x5561fb2fd350 2024-04-27T03:19:14.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2cc68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb2fd350 0x5561fb2fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2cc68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb2fd350 0x5561fb2fd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45868/0 (socket says 172.21.15.110:45868) 2024-04-27T03:19:14.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2cc68640 1 -- 172.21.15.110:0/1817330015 learned_addr learned my addr 172.21.15.110:0/1817330015 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2cc68640 1 -- 172.21.15.110:0/1817330015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561fb2fdcb0 con 0x5561fb2fd350 2024-04-27T03:19:14.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2cc68640 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb2fd350 0x5561fb2fd770 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00aab0 tx=0x7efe1c00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efd11ffb640 1 -- 172.21.15.110:0/1817330015 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe1c007d30 con 0x5561fb2fd350 2024-04-27T03:19:14.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efd11ffb640 1 -- 172.21.15.110:0/1817330015 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe1c01d070 con 0x5561fb2fd350 2024-04-27T03:19:14.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5561fb2fe750 con 0x5561fb2fd350 2024-04-27T03:19:14.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efd11ffb640 1 -- 172.21.15.110:0/1817330015 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe1c018440 con 0x5561fb2fd350 2024-04-27T03:19:14.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.691+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561fb2fec20 con 0x5561fb2fd350 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efd11ffb640 1 -- 172.21.15.110:0/1817330015 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7efe1c013070 con 0x5561fb2fd350 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efd11ffb640 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd04034650 0x7efd04036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efd11ffb640 1 -- 172.21.15.110:0/1817330015 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7efe1c04b6e0 con 0x5561fb2fd350 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2c467640 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd04034650 0x7efd04036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 Processor -- start 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 -- start start 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2c467640 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd04034650 0x7efd04036b10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5561fb2e6bb0 tx=0x7efe240e3340 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb30a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5561fb2dbbd0 con 0x5561fb30a530 2024-04-27T03:19:14.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2cc68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb30a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2cc68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb30a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45884/0 (socket says 172.21.15.110:45884) 2024-04-27T03:19:14.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2cc68640 1 -- 172.21.15.110:0/2360137643 learned_addr learned my addr 172.21.15.110:0/2360137643 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2cc68640 1 -- 172.21.15.110:0/2360137643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561fb30af20 con 0x5561fb30a530 2024-04-27T03:19:14.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2cc68640 1 --2- 172.21.15.110:0/2360137643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb30a950 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efe1c049930 tx=0x7efe1c049be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3f9745f4ce611d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efd0b7fe640 1 -- 172.21.15.110:0/2360137643 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe1c047070 con 0x5561fb30a530 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efd0b7fe640 1 -- 172.21.15.110:0/2360137643 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe1c0225e0 con 0x5561fb30a530 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efd0b7fe640 1 -- 172.21.15.110:0/2360137643 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe1c018d30 con 0x5561fb30a530 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 -- 172.21.15.110:0/2360137643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 msgr2=0x5561fb30a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 --2- 172.21.15.110:0/2360137643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb30a950 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7efe1c049930 tx=0x7efe1c049be0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 -- 172.21.15.110:0/2360137643 shutdown_connections 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 --2- 172.21.15.110:0/2360137643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb30a950 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 -- 172.21.15.110:0/2360137643 >> 172.21.15.110:0/2360137643 conn(0x5561fb306870 msgr2=0x5561fb3048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 -- 172.21.15.110:0/2360137643 shutdown_connections 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.695+0000 7efe2e94c980 1 -- 172.21.15.110:0/2360137643 wait complete. 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 Processor -- start 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 -- start start 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb311a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5561fb30b6a0 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2cc68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb311a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2cc68640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb311a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45890/0 (socket says 172.21.15.110:45890) 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 learned_addr learned my addr 172.21.15.110:0/1105604383 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561fb311f50 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2cc68640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb311a10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7efe1c049c20 tx=0x7efe1c0498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efd09ffb640 1 -- 172.21.15.110:0/1105604383 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe1c031070 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5561fb3121e0 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efd09ffb640 1 -- 172.21.15.110:0/1105604383 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe1c017a40 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efd09ffb640 1 -- 172.21.15.110:0/1105604383 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe1c020a50 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561fb314ec0 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efd09ffb640 1 -- 172.21.15.110:0/1105604383 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7efe1c034070 con 0x5561fb30a530 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efd09ffb640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efcfc034650 0x7efcfc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efd09ffb640 1 -- 172.21.15.110:0/1105604383 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7efe1c037db0 con 0x5561fb30a530 2024-04-27T03:19:14.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2c467640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efcfc034650 0x7efcfc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb31d0c0 0x5561fb31f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb31fcd0 con 0x5561fb31d0c0 2024-04-27T03:19:14.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.699+0000 7efe2bc66640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb31d0c0 0x5561fb31f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2c467640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efcfc034650 0x7efcfc036b10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efe240c86b0 tx=0x7efe240dc000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2bc66640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb31d0c0 0x5561fb31f580 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7efe1400e660 con 0x5561fb31d0c0 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 realm 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb31c510 con 0x5561fb31d0c0 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7efe1400e660 con 0x5561fb31d0c0 2024-04-27T03:19:14.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb3207a0 con 0x5561fb31d0c0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7efe1400e660 con 0x5561fb31d0c0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb321580 0x5561fb323970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb323ff0 con 0x5561fb321580 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2cc68640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb321580 0x5561fb323970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2cc68640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb321580 0x5561fb323970 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb324840 con 0x5561fb31d0c0 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7efe1400e660 con 0x5561fb31d0c0 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 20 searching for the correct realm 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5561fb3265b0 0x5561fb3289a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb3265b0 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2c467640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5561fb3265b0 0x5561fb3289a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.703+0000 7efe2c467640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5561fb3265b0 0x5561fb3289a0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb321580 2024-04-27T03:19:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2cc68640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7efe1c036070 0x7efe1c0218a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x7efe1c036070 2024-04-27T03:19:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7efe1c036070 0x7efe1c0218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7efe1c036070 0x7efe1c0218a0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1400e660 con 0x7efe1c036070 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb31d0c0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1400e660 con 0x5561fb31d0c0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb31d0c0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb3265b0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb31d0c0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb321580 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.707+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb321580 2024-04-27T03:19:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb31d0c0 2024-04-27T03:19:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x7efe1c036070 2024-04-27T03:19:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x7efe1c036070 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb31d0c0 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x7efe1c036070 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7efe14029040 con 0x7efe1c036070 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x5561fb329020 con 0x5561fb321580 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb3265b0 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb31d0c0 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.711+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb321580 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb3265b0 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb31d0c0 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x7efe1c036070 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7efe14029040 con 0x7efe1c036070 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb31d0c0 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb3265b0 2024-04-27T03:19:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x7efe1c036070 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x7efe1c036070 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb31d0c0 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x7efe1c036070 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x7efe1c036070 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb3265b0 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x7efe1c036070 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe14029040 con 0x7efe1c036070 2024-04-27T03:19:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb321580 2024-04-27T03:19:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.715+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb321580 2024-04-27T03:19:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb3265b0 2024-04-27T03:19:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb321580 2024-04-27T03:19:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x7efe1c063040 con 0x5561fb321580 2024-04-27T03:19:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb3251c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7efe14029040 con 0x5561fb31d0c0 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb324e00 con 0x5561fb321580 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb325b20 con 0x5561fb31d0c0 2024-04-27T03:19:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7efe1403c1c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 zone default found 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 4 Realm: () 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 zonegroup default 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb3294a0 con 0x7efe1c036070 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.719+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32b2c0 con 0x5561fb3265b0 2024-04-27T03:19:14.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32b690 con 0x5561fb3265b0 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 20 started sync module instance, tier type = 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32a760 con 0x5561fb321580 2024-04-27T03:19:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93879314590848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32a760 con 0x5561fb321580 2024-04-27T03:19:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.723+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32c1d0 con 0x5561fb31d0c0 2024-04-27T03:19:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93879314594976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32c190 con 0x5561fb31d0c0 2024-04-27T03:19:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32d0e0 con 0x5561fb321580 2024-04-27T03:19:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93879314590848] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 93879314594976] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93879314597312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32d0e0 con 0x5561fb321580 2024-04-27T03:19:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32d530 con 0x7efe1c036070 2024-04-27T03:19:14.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93879314588336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32d4f0 con 0x7efe1c036070 2024-04-27T03:19:14.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32e410 con 0x5561fb321580 2024-04-27T03:19:14.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.727+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93879314597312] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 93879314588336] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93879314602272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32e410 con 0x5561fb321580 2024-04-27T03:19:14.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32e800 con 0x5561fb3265b0 2024-04-27T03:19:14.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93879314598928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32f050 con 0x5561fb3265b0 2024-04-27T03:19:14.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32fa50 con 0x5561fb3265b0 2024-04-27T03:19:14.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93879314602272] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93879314598928] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93879314603840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb32eba0 con 0x5561fb3265b0 2024-04-27T03:19:14.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.731+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb3302f0 con 0x7efe1c036070 2024-04-27T03:19:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'216 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93879314611776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb330710 con 0x7efe1c036070 2024-04-27T03:19:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93879314603840] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=0 2024-04-27T03:19:14.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=1 2024-04-27T03:19:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=2 2024-04-27T03:19:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=3 2024-04-27T03:19:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=4 2024-04-27T03:19:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=5 2024-04-27T03:19:14.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=6 2024-04-27T03:19:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 93879314611776] v29'217 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 20 add_watcher() i=7 2024-04-27T03:19:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb333360 con 0x5561fb31d0c0 2024-04-27T03:19:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7efe1403c1c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.735+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93879314609504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb330710 con 0x5561fb31d0c0 2024-04-27T03:19:14.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93879314609504] v29'220 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efce27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 5 note: GC not initialized 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efce17e2640 20 reqs_thread_entry: start 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 Filter name: none 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efc52ffd640 20 reqs_thread_entry: start 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb356cb0 con 0x7efe1c036070 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(62 fud [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x0 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : miss 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb3552f0 con 0x5561fb3265b0 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 p5YnriCv1nAtykxBrupQ [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 206+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x0 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 adding default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 10 cache get: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ : hit (negative entry) 2024-04-27T03:19:14.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.739+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:64 8.e 8:75f48285:users.uid::fud:head [delete,create,call version.set in=58b,writefull 0~380 in=380b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb355bb0 con 0x7efe1c036070 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(64 fud [delete,create,call,writefull 0~380] v29'5 uv5 ondisk = 0) ==== 273+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 10 moving default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:65 7.3 7:c609908c:::notify.5:head [notify cookie 93879314765568 in=568b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb35a750 con 0x5561fb3265b0 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== watch-notify(notify (1) cookie 93879314598928 notify 124554051611 ret 0) ==== 598+0+0 (crc 0 0 0) 0x5561fb2e5920 con 0x5561fb3265b0 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efd0b7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051611 cookie 93879314598928 notifier 4730 bl.length()=556 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efd0b7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+fud info.flags=0x17 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efd0b7fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(65 notify.5 [notify cookie 93879314765568 out=8b] v0'0 uv216 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efd0b7fe640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:66 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7efcf0003eb0 con 0x5561fb3265b0 2024-04-27T03:19:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(66 notify.5 [notify-ack] v0'0 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== watch-notify(notify_complete (2) cookie 93879314765568 notify 124554051611 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5561fb2e2cf0 con 0x5561fb3265b0 2024-04-27T03:19:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb355700 con 0x5561fb3265b0 2024-04-27T03:19:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(67 p5YnriCv1nAtykxBrupQ [delete,create,writefull 0~7] v29'11 uv11 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 10 cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:19:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 10 moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:19:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:19:14.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.743+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify cookie 93879314780816 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x5561fb35ab70 con 0x5561fb321580 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 93879314597312 notify 124554051627 ret 0) ==== 219+0+0 (crc 0 0 0) 0x5561fb314ec0 con 0x5561fb321580 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [notify cookie 93879314780816 out=8b] v0'0 uv216 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efd0affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051627 cookie 93879314597312 notifier 4730 bl.length()=177 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efd0affd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ info.flags=0x7 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efd0affd640 10 rgw watcher librados: moving default.rgw.meta+users.keys+p5YnriCv1nAtykxBrupQ to cache LRU end 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efd0affd640 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7efc60003410 con 0x5561fb321580 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.2 [notify-ack] v0'0 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 93879314780816 notify 124554051627 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7efe1c06a020 con 0x5561fb321580 2024-04-27T03:19:14.757 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"fud","display_name":"Fud","email":"","suspended":0,"max_buckets":4,"subusers":[],"keys":[{"user":"fud","access_key":"p5YnriCv1nAtykxBrupQ","secret_key":"Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh","active":true,"create_date":"2024-04-27T03:19:14.744159Z"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:14.744151Z","tags":[],"group_ids":[]} 2024-04-27T03:19:14.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efce27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:14.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.747+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93879314609504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x5561fb31d0c0 2024-04-27T03:19:14.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.751+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 93879314609504] v29'221 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.751+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93879314590848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x5561fb321580 2024-04-27T03:19:14.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.751+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 93879314590848] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.751+0000 7efe2e94c980 20 remove_watcher() i=0 2024-04-27T03:19:14.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.751+0000 7efe2e94c980 2 removed watcher, disabling cache 2024-04-27T03:19:14.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.751+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93879314594976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x5561fb31d0c0 2024-04-27T03:19:14.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 93879314594976] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x5561fb31d0c0 2024-04-27T03:19:14.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2e94c980 20 remove_watcher() i=1 2024-04-27T03:19:14.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93879314597312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x5561fb321580 2024-04-27T03:19:14.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 93879314597312] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2e94c980 20 remove_watcher() i=2 2024-04-27T03:19:14.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93879314588336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x7efe1c036070 2024-04-27T03:19:14.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 93879314588336] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2e94c980 20 remove_watcher() i=3 2024-04-27T03:19:14.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.755+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93879314602272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x5561fb321580 2024-04-27T03:19:14.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2cc68640 1 -- 172.21.15.110:0/1105604383 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 93879314602272] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1c033030 con 0x5561fb321580 2024-04-27T03:19:14.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2e94c980 20 remove_watcher() i=4 2024-04-27T03:19:14.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93879314598928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x5561fb3265b0 2024-04-27T03:19:14.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 93879314598928] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2e94c980 20 remove_watcher() i=5 2024-04-27T03:19:14.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93879314603840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x5561fb3265b0 2024-04-27T03:19:14.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2c467640 1 -- 172.21.15.110:0/1105604383 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 93879314603840] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe240d1070 con 0x5561fb3265b0 2024-04-27T03:19:14.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2e94c980 20 remove_watcher() i=6 2024-04-27T03:19:14.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.759+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93879314611776] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x5561fb354950 con 0x7efe1c036070 2024-04-27T03:19:14.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 93879314611776] v29'218 uv216 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7efe1403c1c0 con 0x7efe1c036070 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 20 remove_watcher() i=7 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb321580 msgr2=0x5561fb323970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb321580 0x5561fb323970 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb31d0c0 msgr2=0x5561fb31f580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb31d0c0 0x5561fb31f580 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7efe1c036070 msgr2=0x7efe1c0218a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7efe1c036070 0x7efe1c0218a0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5561fb3265b0 msgr2=0x5561fb3289a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5561fb3265b0 0x5561fb3289a0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efcfc034650 msgr2=0x7efcfc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efcfc034650 0x7efcfc036b10 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efe240c86b0 tx=0x7efe240dc000 comp rx=0 tx=0).stop 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 msgr2=0x5561fb311a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2bc66640 1 -- 172.21.15.110:0/1105604383 reap_dead start 2024-04-27T03:19:14.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb311a10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7efe1c049c20 tx=0x7efe1c0498c0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 shutdown_connections 2024-04-27T03:19:14.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1105604383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb30a530 0x5561fb311a10 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 >> 172.21.15.110:0/1105604383 conn(0x5561fb306870 msgr2=0x5561fb303ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 shutdown_connections 2024-04-27T03:19:14.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1105604383 wait complete. 2024-04-27T03:19:14.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd04034650 msgr2=0x7efd04036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd04034650 0x7efd04036b10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5561fb2e6bb0 tx=0x7efe240e3340 comp rx=0 tx=0).stop 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb2fd350 msgr2=0x5561fb2fd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb2fd350 0x5561fb2fd770 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00aab0 tx=0x7efe1c00af70 comp rx=0 tx=0).stop 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 shutdown_connections 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd04034650 0x7efd04036b10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1817330015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb2fd350 0x5561fb2fd770 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 >> 172.21.15.110:0/1817330015 conn(0x5561fb2f2160 msgr2=0x5561fb2f2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 shutdown_connections 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1817330015 wait complete. 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb2ea560 msgr2=0x5561fb2eca20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb2ea560 0x5561fb2eca20 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb1f38e0 msgr2=0x5561fb2e9570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb1f38e0 0x5561fb2e9570 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd14034650 msgr2=0x7efd14036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd14034650 0x7efd14036b10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efe14001dc0 tx=0x7efe14001360 comp rx=0 tx=0).stop 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 msgr2=0x5561fb2e2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2e2520 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7efe2401b010 tx=0x7efe24019810 comp rx=0 tx=0).stop 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 shutdown_connections 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5561fb2ea560 0x5561fb2eca20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5561fb1f38e0 0x5561fb2e9570 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7efd14034650 0x7efd14036b10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 --2- 172.21.15.110:0/1957018308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5561fb162000 0x5561fb2e2520 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 >> 172.21.15.110:0/1957018308 conn(0x5561fb1e8a80 msgr2=0x5561fb2d5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 shutdown_connections 2024-04-27T03:19:14.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.763+0000 7efe2e94c980 1 -- 172.21.15.110:0/1957018308 wait complete. 2024-04-27T03:19:14.785 DEBUG:tasks.util.rgw: json result: {'user_id': 'fud', 'display_name': 'Fud', 'email': '', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'fud', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True, 'create_date': '2024-04-27T03:19:14.744159Z'}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:14.744151Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:14.804 INFO:tasks.radosgw_admin:RL: create_bucket myfoo7 fud bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:19:14.811 INFO:tasks.radosgw_admin:RL: put_obj myfoo7 fud bytes_out=8 bytes_in=0 failed=False 2024-04-27T03:19:14.811 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rename', '--uid', 'bud', '--new-uid', 'fud'] 2024-04-27T03:19:14.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rename', '--uid', 'bud', '--new-uid', 'fud'] 2024-04-27T03:19:14.811 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rename --uid bud --new-uid fud 2024-04-27T03:19:14.843 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:14.843 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:14.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f8352980 1 Processor -- start 2024-04-27T03:19:14.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f8352980 1 -- start start 2024-04-27T03:19:14.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f8352980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56413b097670 0x56413b099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f8352980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x56413af520d0 con 0x56413b007860 2024-04-27T03:19:14.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f8352980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x56413af47430 con 0x56413b097670 2024-04-27T03:19:14.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f566c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56413b097670 0x56413b099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f5e6d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x56413b007860 0x56413b095f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51280/0 (socket says 172.21.15.110:51280) 2024-04-27T03:19:14.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/3474179559 learned_addr learned my addr 172.21.15.110:0/3474179559 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:14.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f4e6b640 1 -- 172.21.15.110:0/3474179559 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944766592 0 0) 0x56413af520d0 con 0x56413b007860 2024-04-27T03:19:14.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f4e6b640 1 -- 172.21.15.110:0/3474179559 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6e4001e20 con 0x56413b007860 2024-04-27T03:19:14.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f566c640 1 -- 172.21.15.110:0/3474179559 >> v1:172.21.15.77:6789/0 conn(0x56413b007860 legacy=0x56413b095f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.835+0000 7ff6f566c640 1 -- 172.21.15.110:0/3474179559 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413af40510 con 0x56413b097670 2024-04-27T03:19:14.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f566c640 1 --2- 172.21.15.110:0/3474179559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b097670 0x56413b099a60 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000a480 tx=0x7ff6e001b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5e601e1e7cf48 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f4e6b640 1 -- 172.21.15.110:0/3474179559 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e00b0070 con 0x56413b097670 2024-04-27T03:19:14.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f4e6b640 1 -- 172.21.15.110:0/3474179559 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e00ab540 con 0x56413b097670 2024-04-27T03:19:14.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f4e6b640 1 -- 172.21.15.110:0/3474179559 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e00ab820 con 0x56413b097670 2024-04-27T03:19:14.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f8352980 1 -- 172.21.15.110:0/3474179559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b097670 msgr2=0x56413b099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f8352980 1 --2- 172.21.15.110:0/3474179559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b097670 0x56413b099a60 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000a480 tx=0x7ff6e001b280 comp rx=0 tx=0).stop 2024-04-27T03:19:14.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f8352980 1 -- 172.21.15.110:0/3474179559 shutdown_connections 2024-04-27T03:19:14.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f8352980 1 --2- 172.21.15.110:0/3474179559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b097670 0x56413b099a60 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f8352980 1 -- 172.21.15.110:0/3474179559 >> 172.21.15.110:0/3474179559 conn(0x56413b08da30 msgr2=0x56413b08fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f8352980 1 -- 172.21.15.110:0/3474179559 shutdown_connections 2024-04-27T03:19:14.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.839+0000 7ff6f8352980 1 -- 172.21.15.110:0/3474179559 wait complete. 2024-04-27T03:19:14.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f8352980 1 Processor -- start 2024-04-27T03:19:14.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f8352980 1 -- start start 2024-04-27T03:19:14.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f8352980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b17e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f8352980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56413af962d0 con 0x56413b007860 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f5e6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b17e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f5e6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b17e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45908/0 (socket says 172.21.15.110:45908) 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/3142023996 learned_addr learned my addr 172.21.15.110:0/3142023996 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/3142023996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413af1cae0 con 0x56413b007860 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/3142023996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b17e270 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0001dc0 tx=0x7ff6f000b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90db42e3129ddebf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff5e3e07640 1 -- 172.21.15.110:0/3142023996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6f0012020 con 0x56413b007860 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff5e3e07640 1 -- 172.21.15.110:0/3142023996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6f000e7c0 con 0x56413b007860 2024-04-27T03:19:14.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.855+0000 7ff5e3e07640 1 -- 172.21.15.110:0/3142023996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6f000eb40 con 0x56413b007860 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- 172.21.15.110:0/3142023996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 msgr2=0x56413b17e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 --2- 172.21.15.110:0/3142023996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b17e270 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0001dc0 tx=0x7ff6f000b6d0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- 172.21.15.110:0/3142023996 shutdown_connections 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 --2- 172.21.15.110:0/3142023996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b17e270 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- 172.21.15.110:0/3142023996 >> 172.21.15.110:0/3142023996 conn(0x56413b08da30 msgr2=0x56413b179e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- 172.21.15.110:0/3142023996 shutdown_connections 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- 172.21.15.110:0/3142023996 wait complete. 2024-04-27T03:19:14.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 Processor -- start 2024-04-27T03:19:14.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- start start 2024-04-27T03:19:14.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b1852e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56413b09ab40 con 0x56413b007860 2024-04-27T03:19:14.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f5e6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b1852e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f5e6d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b1852e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45924/0 (socket says 172.21.15.110:45924) 2024-04-27T03:19:14.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/710591933 learned_addr learned my addr 172.21.15.110:0/710591933 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413b185820 con 0x56413b007860 2024-04-27T03:19:14.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b1852e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x56413b17f100 tx=0x7ff6f0000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff5e2604640 1 -- 172.21.15.110:0/710591933 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6f0012030 con 0x56413b007860 2024-04-27T03:19:14.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff5e2604640 1 -- 172.21.15.110:0/710591933 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6f0001440 con 0x56413b007860 2024-04-27T03:19:14.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff5e2604640 1 -- 172.21.15.110:0/710591933 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6f001aaf0 con 0x56413b007860 2024-04-27T03:19:14.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56413b185ab0 con 0x56413b007860 2024-04-27T03:19:14.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.859+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413b188730 con 0x56413b007860 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff5e2604640 1 -- 172.21.15.110:0/710591933 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff6f0015020 con 0x56413b007860 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff5e2604640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5dc034650 0x7ff5dc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff5e2604640 1 -- 172.21.15.110:0/710591933 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7ff6f0020030 con 0x56413b007860 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f566c640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5dc034650 0x7ff5dc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b097670 0x56413b18c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b18c810 con 0x56413b097670 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f666e640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b097670 0x56413b18c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f566c640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5dc034650 0x7ff5dc036b10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00ad000 tx=0x7ff6e00a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f666e640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b097670 0x56413b18c210 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f666e640 1 -- 172.21.15.110:0/710591933 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff6e800a4d0 con 0x56413b097670 2024-04-27T03:19:14.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b18d600 0x56413b18f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b190020 con 0x56413b18d600 2024-04-27T03:19:14.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b18d600 0x56413b18f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.863+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b18d600 0x56413b18f9f0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/710591933 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b18d600 2024-04-27T03:19:14.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1918d0 con 0x56413b097670 2024-04-27T03:19:14.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f666e640 1 -- 172.21.15.110:0/710591933 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7ff6e800a4d0 con 0x56413b097670 2024-04-27T03:19:14.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b190b00 con 0x56413b18d600 2024-04-27T03:19:14.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/710591933 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b18d600 2024-04-27T03:19:14.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b190ef0 con 0x56413b097670 2024-04-27T03:19:14.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f666e640 1 -- 172.21.15.110:0/710591933 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7ff6e800a4d0 con 0x56413b097670 2024-04-27T03:19:14.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 Processor -- start 2024-04-27T03:19:14.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 -- start start 2024-04-27T03:19:14.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1a0210 0x56413b1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56413af9a310 con 0x56413b1a0210 2024-04-27T03:19:14.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f666e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1a0210 0x56413b1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f666e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1a0210 0x56413b1a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45940/0 (socket says 172.21.15.110:45940) 2024-04-27T03:19:14.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f666e640 1 -- 172.21.15.110:0/321213376 learned_addr learned my addr 172.21.15.110:0/321213376 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f666e640 1 -- 172.21.15.110:0/321213376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413b1a0b70 con 0x56413b1a0210 2024-04-27T03:19:14.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f666e640 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1a0210 0x56413b1a0630 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800dfc0 tx=0x7ff6e80178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff5db7fe640 1 -- 172.21.15.110:0/321213376 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e80157d0 con 0x56413b1a0210 2024-04-27T03:19:14.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff5db7fe640 1 -- 172.21.15.110:0/321213376 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e8015970 con 0x56413b1a0210 2024-04-27T03:19:14.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56413b1a1610 con 0x56413b1a0210 2024-04-27T03:19:14.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff5db7fe640 1 -- 172.21.15.110:0/321213376 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e8015c70 con 0x56413b1a0210 2024-04-27T03:19:14.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.867+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413b1a1ae0 con 0x56413b1a0210 2024-04-27T03:19:14.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff5db7fe640 1 -- 172.21.15.110:0/321213376 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff6e8029440 con 0x56413b1a0210 2024-04-27T03:19:14.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff5db7fe640 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5d00346a0 0x7ff5d0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff5db7fe640 1 -- 172.21.15.110:0/321213376 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7ff6e800e040 con 0x56413b1a0210 2024-04-27T03:19:14.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5d00346a0 0x7ff5d0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 Processor -- start 2024-04-27T03:19:14.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 -- start start 2024-04-27T03:19:14.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5d00346a0 0x7ff5d0036b60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x56413b190620 tx=0x7ff6f00484c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56413b17e9c0 con 0x56413b1ad3b0 2024-04-27T03:19:14.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f666e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f666e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1ad7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45954/0 (socket says 172.21.15.110:45954) 2024-04-27T03:19:14.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f666e640 1 -- 172.21.15.110:0/379239381 learned_addr learned my addr 172.21.15.110:0/379239381 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f666e640 1 -- 172.21.15.110:0/379239381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413b1adda0 con 0x56413b1ad3b0 2024-04-27T03:19:14.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f666e640 1 --2- 172.21.15.110:0/379239381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1ad7d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8015fa0 tx=0x7ff6e8015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29c1f59852fe91 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff5d97fa640 1 -- 172.21.15.110:0/379239381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e80113a0 con 0x56413b1ad3b0 2024-04-27T03:19:14.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff5d97fa640 1 -- 172.21.15.110:0/379239381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e8011540 con 0x56413b1ad3b0 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff5d97fa640 1 -- 172.21.15.110:0/379239381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e80284d0 con 0x56413b1ad3b0 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 -- 172.21.15.110:0/379239381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 msgr2=0x56413b1ad7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 --2- 172.21.15.110:0/379239381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1ad7d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8015fa0 tx=0x7ff6e8015fd0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 -- 172.21.15.110:0/379239381 shutdown_connections 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 --2- 172.21.15.110:0/379239381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1ad7d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 -- 172.21.15.110:0/379239381 >> 172.21.15.110:0/379239381 conn(0x56413b1a96f0 msgr2=0x56413b1a7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 -- 172.21.15.110:0/379239381 shutdown_connections 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 -- 172.21.15.110:0/379239381 wait complete. 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.871+0000 7ff6f8352980 1 Processor -- start 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 1 -- start start 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56413b1ae500 con 0x56413b1ad3b0 2024-04-27T03:19:14.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f666e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f666e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1b4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45966/0 (socket says 172.21.15.110:45966) 2024-04-27T03:19:14.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 learned_addr learned my addr 172.21.15.110:0/183652093 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:14.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413b1b4d60 con 0x56413b1ad3b0 2024-04-27T03:19:14.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f666e640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1b4820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff6e801eb40 tx=0x7ff6e804a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff5c77fe640 1 -- 172.21.15.110:0/183652093 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e8038070 con 0x56413b1ad3b0 2024-04-27T03:19:14.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56413b1b4ff0 con 0x56413b1ad3b0 2024-04-27T03:19:14.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff5c77fe640 1 -- 172.21.15.110:0/183652093 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6e8049450 con 0x56413b1ad3b0 2024-04-27T03:19:14.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff5c77fe640 1 -- 172.21.15.110:0/183652093 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6e80113a0 con 0x56413b1ad3b0 2024-04-27T03:19:14.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413b1b7cd0 con 0x56413b1ad3b0 2024-04-27T03:19:14.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff5c77fe640 1 -- 172.21.15.110:0/183652093 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff6e804c0d0 con 0x56413b1ad3b0 2024-04-27T03:19:14.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff5c77fe640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5c0034650 0x7ff5c0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff5c77fe640 1 -- 172.21.15.110:0/183652093 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7ff6e8030070 con 0x56413b1ad3b0 2024-04-27T03:19:14.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5c0034650 0x7ff5c0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b1bfea0 0x56413b1c2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c2ab0 con 0x56413b1bfea0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.875+0000 7ff6f566c640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b1bfea0 0x56413b1c2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5c0034650 0x7ff5c0036b10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff6f003b040 tx=0x7ff6f0039000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f566c640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b1bfea0 0x56413b1c2360 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 realm 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c2e80 con 0x56413b1bfea0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c3750 con 0x56413b1bfea0 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b1c4170 0x56413b1c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c6cf0 con 0x56413b1c4170 2024-04-27T03:19:14.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f666e640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b1c4170 0x56413b1c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f666e640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b1c4170 0x56413b1c6580 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:14.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c7540 con 0x56413b1bfea0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 20 searching for the correct realm 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56413b1c92b0 0x56413b1cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c92b0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56413b1c92b0 0x56413b1cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.879+0000 7ff6f5e6d640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56413b1c92b0 0x56413b1cb6a0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f666e640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff6e8056040 0x7ff6e80227f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x7ff6e8056040 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff6e8056040 0x7ff6e80227f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:14.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff6e8056040 0x7ff6e80227f0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:14.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c92b0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.883+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x7ff6e8056040 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x7ff6e8056040 2024-04-27T03:19:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c92b0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c92b0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.887+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x7ff6e8056040 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c92b0 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x7ff6e8056040 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1bfea0 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x7ff6e8056040 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c92b0 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x7ff6e8056040 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.891+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c92b0 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cbcb0 con 0x56413b1c4170 2024-04-27T03:19:14.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:14.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:14.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:14.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:14.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c7ec0 con 0x56413b1bfea0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c8760 con 0x56413b1c4170 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1c7b00 con 0x56413b1bfea0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:14.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 zone default found 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 4 Realm: () 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 zonegroup default 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cc5e0 con 0x7ff6e8056040 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1ce3b0 con 0x56413b1c92b0 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1ce770 con 0x56413b1c92b0 2024-04-27T03:19:14.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:14.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 started sync module instance, tier type = 2024-04-27T03:19:14.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:14.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.895+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cd850 con 0x56413b1c4170 2024-04-27T03:19:14.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.899+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.899+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94838164610416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cd850 con 0x56413b1c4170 2024-04-27T03:19:14.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.899+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cf250 con 0x56413b1bfea0 2024-04-27T03:19:14.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.899+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.899+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94838164614464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1cf210 con 0x56413b1bfea0 2024-04-27T03:19:14.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.899+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d0170 con 0x56413b1c4170 2024-04-27T03:19:14.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.899+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94838164610416] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94838164614464] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94838164616768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d0130 con 0x56413b1c4170 2024-04-27T03:19:14.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d05c0 con 0x7ff6e8056040 2024-04-27T03:19:14.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94838164607904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d0580 con 0x7ff6e8056040 2024-04-27T03:19:14.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d14c0 con 0x56413b1c4170 2024-04-27T03:19:14.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94838164616768] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.903+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94838164607904] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94838164621744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d14c0 con 0x56413b1c4170 2024-04-27T03:19:14.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d18b0 con 0x56413b1c92b0 2024-04-27T03:19:14.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94838164618336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d2100 con 0x56413b1c92b0 2024-04-27T03:19:14.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d2b00 con 0x56413b1c92b0 2024-04-27T03:19:14.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94838164621744] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94838164618336] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94838164626512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d2610 con 0x56413b1c92b0 2024-04-27T03:19:14.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.907+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d3510 con 0x7ff6e8056040 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'219 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94838164626512] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94838164631648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d3480 con 0x7ff6e8056040 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=0 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=1 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=2 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=3 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=4 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=5 2024-04-27T03:19:14.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=6 2024-04-27T03:19:14.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94838164631648] v29'220 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 20 add_watcher() i=7 2024-04-27T03:19:14.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:14.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d6880 con 0x56413b1bfea0 2024-04-27T03:19:14.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.911+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94838164623312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1d3da0 con 0x56413b1bfea0 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94838164623312] v29'222 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff5a77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 5 note: GC not initialized 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff5a67fc640 20 reqs_thread_entry: start 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 20 Filter name: none 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff517fff640 20 reqs_thread_entry: start 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 10 cache get: name=default.rgw.meta+users.uid+bud : miss 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 20 rados->read ofs=0 len=0 2024-04-27T03:19:14.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:62 8.4 8:267425a8:users.uid::bud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x56413b1fa040 con 0x7ff6e8056040 2024-04-27T03:19:14.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(62 bud [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv2 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:19:14.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x13 2024-04-27T03:19:14.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2024-04-27T03:19:14.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:14.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.915+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 8.e 8:75f48285:users.uid::fud:head [create,call version.set in=58b,writefull 0~206 in=206b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1fe3d0 con 0x7ff6e8056040 2024-04-27T03:19:14.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.919+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(63 fud [create,call,writefull 0~206] v29'6 uv5 ondisk = -17 ((17) File exists)) ==== 231+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.927 INFO:teuthology.orchestra.run.smithi110.stderr:could not rename user: unable to rename user, user name given by --new-uid already exists. to overwrite this user, add --yes-i-really-mean-it 2024-04-27T03:19:14.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.919+0000 7ff5a77fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:14.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.919+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94838164623312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x56413b1bfea0 2024-04-27T03:19:14.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.919+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(64 data_loggenerations_metadata [watch unwatch cookie 94838164623312] v29'223 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.919+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94838164610416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x56413b1c4170 2024-04-27T03:19:14.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.923+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(65 notify.0 [watch unwatch cookie 94838164610416] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.923+0000 7ff6f8352980 20 remove_watcher() i=0 2024-04-27T03:19:14.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.923+0000 7ff6f8352980 2 removed watcher, disabling cache 2024-04-27T03:19:14.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.923+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94838164614464] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x56413b1bfea0 2024-04-27T03:19:14.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.923+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(66 notify.1 [watch unwatch cookie 94838164614464] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x56413b1bfea0 2024-04-27T03:19:14.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.923+0000 7ff6f8352980 20 remove_watcher() i=1 2024-04-27T03:19:14.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.923+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94838164616768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x56413b1c4170 2024-04-27T03:19:14.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.927+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [watch unwatch cookie 94838164616768] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.927+0000 7ff6f8352980 20 remove_watcher() i=2 2024-04-27T03:19:14.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.927+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:68 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94838164607904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x7ff6e8056040 2024-04-27T03:19:14.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.927+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(68 notify.3 [watch unwatch cookie 94838164607904] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.927+0000 7ff6f8352980 20 remove_watcher() i=3 2024-04-27T03:19:14.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.927+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94838164621744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x56413b1c4170 2024-04-27T03:19:14.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f666e640 1 -- 172.21.15.110:0/183652093 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.4 [watch unwatch cookie 94838164621744] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e8026020 con 0x56413b1c4170 2024-04-27T03:19:14.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 20 remove_watcher() i=4 2024-04-27T03:19:14.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94838164618336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x56413b1c92b0 2024-04-27T03:19:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(70 notify.5 [watch unwatch cookie 94838164618336] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 20 remove_watcher() i=5 2024-04-27T03:19:14.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94838164626512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x56413b1c92b0 2024-04-27T03:19:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f5e6d640 1 -- 172.21.15.110:0/183652093 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(71 notify.6 [watch unwatch cookie 94838164626512] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6f0008030 con 0x56413b1c92b0 2024-04-27T03:19:14.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 20 remove_watcher() i=6 2024-04-27T03:19:14.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94838164631648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x56413b1f7d70 con 0x7ff6e8056040 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(72 notify.7 [watch unwatch cookie 94838164631648] v29'221 uv219 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ff6e00b3040 con 0x7ff6e8056040 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 20 remove_watcher() i=7 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b1c4170 msgr2=0x56413b1c6580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b1c4170 0x56413b1c6580 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b1bfea0 msgr2=0x56413b1c2360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.931+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b1bfea0 0x56413b1c2360 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff6e8056040 msgr2=0x7ff6e80227f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ff6e8056040 0x7ff6e80227f0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56413b1c92b0 msgr2=0x56413b1cb6a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56413b1c92b0 0x56413b1cb6a0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5c0034650 msgr2=0x7ff5c0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5c0034650 0x7ff5c0036b10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff6f003b040 tx=0x7ff6f0039000 comp rx=0 tx=0).stop 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 msgr2=0x56413b1b4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/183652093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1ad3b0 0x56413b1b4820 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff6e801eb40 tx=0x7ff6e804a880 comp rx=0 tx=0).stop 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 reap_dead start 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f566c640 1 -- 172.21.15.110:0/183652093 reap_dead start 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 shutdown_connections 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 >> 172.21.15.110:0/183652093 conn(0x56413b1a96f0 msgr2=0x56413b1a6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 shutdown_connections 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/183652093 wait complete. 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5d00346a0 msgr2=0x7ff5d0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5d00346a0 0x7ff5d0036b60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x56413b190620 tx=0x7ff6f00484c0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1a0210 msgr2=0x56413b1a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1a0210 0x56413b1a0630 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800dfc0 tx=0x7ff6e80178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 shutdown_connections 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5d00346a0 0x7ff5d0036b60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/321213376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b1a0210 0x56413b1a0630 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 >> 172.21.15.110:0/321213376 conn(0x56413b195020 msgr2=0x56413b195b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 shutdown_connections 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/321213376 wait complete. 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b18d600 msgr2=0x56413b18f9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b18d600 0x56413b18f9f0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b097670 msgr2=0x56413b18c210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b097670 0x56413b18c210 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5dc034650 msgr2=0x7ff5dc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5dc034650 0x7ff5dc036b10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00ad000 tx=0x7ff6e00a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 msgr2=0x56413b1852e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b1852e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x56413b17f100 tx=0x7ff6f0000bd0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 shutdown_connections 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56413b18d600 0x56413b18f9f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56413b097670 0x56413b18c210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff5dc034650 0x7ff5dc036b10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 --2- 172.21.15.110:0/710591933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56413b007860 0x56413b1852e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:14.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 >> 172.21.15.110:0/710591933 conn(0x56413b08da30 msgr2=0x56413b178910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:14.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 shutdown_connections 2024-04-27T03:19:14.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:14.935+0000 7ff6f8352980 1 -- 172.21.15.110:0/710591933 wait complete. 2024-04-27T03:19:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 17 2024-04-27T03:19:14.962 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:19:14.962 INFO:tasks.radosgw_admin:RL: get_obj myfoo6 bud bytes_out=0 bytes_in=8 failed=False 2024-04-27T03:19:14.967 INFO:tasks.radosgw_admin:RL: get_obj myfoo7 fud bytes_out=0 bytes_in=0 failed=False 2024-04-27T03:19:14.967 INFO:tasks.radosgw_admin:RL: get_obj myfoo7 fud bytes_out=0 bytes_in=8 failed=False 2024-04-27T03:19:14.967 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'bud', '--purge-data'] 2024-04-27T03:19:14.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'bud', '--purge-data'] 2024-04-27T03:19:14.968 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid bud --purge-data 2024-04-27T03:19:15.039 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:15.039 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:15.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835de6b980 1 Processor -- start 2024-04-27T03:19:15.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835de6b980 1 -- start start 2024-04-27T03:19:15.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835de6b980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55fcce1ac4a0 0x55fcce1ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835de6b980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55fcce0690d0 con 0x55fcce1acdc0 2024-04-27T03:19:15.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835de6b980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55fcce05e430 con 0x55fcce1ac4a0 2024-04-27T03:19:15.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835b986640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55fcce1ac4a0 0x55fcce1ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f8353fff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55fcce1acdc0 0x55fcce1ad790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51282/0 (socket says 172.21.15.110:51282) 2024-04-27T03:19:15.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f8353fff640 1 -- 172.21.15.110:0/4220977033 learned_addr learned my addr 172.21.15.110:0/4220977033 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:15.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835b986640 1 -- 172.21.15.110:0/4220977033 >> v1:172.21.15.77:6789/0 conn(0x55fcce1acdc0 legacy=0x55fcce1ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.031+0000 7f835b986640 1 -- 172.21.15.110:0/4220977033 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcce057510 con 0x55fcce1ac4a0 2024-04-27T03:19:15.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835b986640 1 --2- 172.21.15.110:0/4220977033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce1ac880 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f834c00a480 tx=0x7f834c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5346556a100b4aa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835b185640 1 -- 172.21.15.110:0/4220977033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f834c0b0070 con 0x55fcce1ac4a0 2024-04-27T03:19:15.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835b185640 1 -- 172.21.15.110:0/4220977033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c0ab540 con 0x55fcce1ac4a0 2024-04-27T03:19:15.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835b185640 1 -- 172.21.15.110:0/4220977033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f834c0ab840 con 0x55fcce1ac4a0 2024-04-27T03:19:15.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835de6b980 1 -- 172.21.15.110:0/4220977033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 msgr2=0x55fcce1ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835de6b980 1 --2- 172.21.15.110:0/4220977033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce1ac880 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f834c00a480 tx=0x7f834c01b280 comp rx=0 tx=0).stop 2024-04-27T03:19:15.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835de6b980 1 -- 172.21.15.110:0/4220977033 shutdown_connections 2024-04-27T03:19:15.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835de6b980 1 --2- 172.21.15.110:0/4220977033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce1ac880 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835de6b980 1 -- 172.21.15.110:0/4220977033 >> 172.21.15.110:0/4220977033 conn(0x55fcce11fba0 msgr2=0x55fcce11ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835de6b980 1 -- 172.21.15.110:0/4220977033 shutdown_connections 2024-04-27T03:19:15.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.035+0000 7f835de6b980 1 -- 172.21.15.110:0/4220977033 wait complete. 2024-04-27T03:19:15.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835de6b980 1 Processor -- start 2024-04-27T03:19:15.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835de6b980 1 -- start start 2024-04-27T03:19:15.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835de6b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce297d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835de6b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fcce0ad2d0 con 0x55fcce1ac4a0 2024-04-27T03:19:15.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835b986640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce297d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835b986640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce297d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:45988/0 (socket says 172.21.15.110:45988) 2024-04-27T03:19:15.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835b986640 1 -- 172.21.15.110:0/3620962069 learned_addr learned my addr 172.21.15.110:0/3620962069 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835b986640 1 -- 172.21.15.110:0/3620962069 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcce033ae0 con 0x55fcce1ac4a0 2024-04-27T03:19:15.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f835b986640 1 --2- 172.21.15.110:0/3620962069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce297d40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f834c0ae040 tx=0x7f834c003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16748cb121af96d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.051+0000 7f8249e0b640 1 -- 172.21.15.110:0/3620962069 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f834c0b0040 con 0x55fcce1ac4a0 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f8249e0b640 1 -- 172.21.15.110:0/3620962069 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c0ab9b0 con 0x55fcce1ac4a0 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f8249e0b640 1 -- 172.21.15.110:0/3620962069 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f834c0abc90 con 0x55fcce1ac4a0 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- 172.21.15.110:0/3620962069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 msgr2=0x55fcce297d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 --2- 172.21.15.110:0/3620962069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce297d40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f834c0ae040 tx=0x7f834c003030 comp rx=0 tx=0).stop 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- 172.21.15.110:0/3620962069 shutdown_connections 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 --2- 172.21.15.110:0/3620962069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce297d40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- 172.21.15.110:0/3620962069 >> 172.21.15.110:0/3620962069 conn(0x55fcce11fba0 msgr2=0x55fcce2938d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- 172.21.15.110:0/3620962069 shutdown_connections 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- 172.21.15.110:0/3620962069 wait complete. 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 Processor -- start 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- start start 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce29ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fcce1b6500 con 0x55fcce1ac4a0 2024-04-27T03:19:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835b986640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce29ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835b986640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce29ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46002/0 (socket says 172.21.15.110:46002) 2024-04-27T03:19:15.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835b986640 1 -- 172.21.15.110:0/4182675584 learned_addr learned my addr 172.21.15.110:0/4182675584 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835b986640 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcce29f200 con 0x55fcce1ac4a0 2024-04-27T03:19:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835b986640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce29ecc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f834c01bb20 tx=0x7f834c018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f8243fff640 1 -- 172.21.15.110:0/4182675584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f834c0c5070 con 0x55fcce1ac4a0 2024-04-27T03:19:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fcce29f490 con 0x55fcce1ac4a0 2024-04-27T03:19:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f8243fff640 1 -- 172.21.15.110:0/4182675584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f834c0b6970 con 0x55fcce1ac4a0 2024-04-27T03:19:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f8243fff640 1 -- 172.21.15.110:0/4182675584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f834c0b0050 con 0x55fcce1ac4a0 2024-04-27T03:19:15.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.055+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcce2a2070 con 0x55fcce1ac4a0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f8243fff640 1 -- 172.21.15.110:0/4182675584 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f834c0c8070 con 0x55fcce1ac4a0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f8243fff640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8244034650 0x7f8244036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f8243fff640 1 -- 172.21.15.110:0/4182675584 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f834c0bc070 con 0x55fcce1ac4a0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f8353fff640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8244034650 0x7f8244036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce1acdc0 0x55fcce2a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2a60d0 con 0x55fcce1acdc0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835c187640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce1acdc0 0x55fcce2a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f8353fff640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8244034650 0x7f8244036b10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8344000f70 tx=0x7f83440012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835c187640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce1acdc0 0x55fcce2a5b50 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835c187640 1 -- 172.21.15.110:0/4182675584 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f835400a4d0 con 0x55fcce1acdc0 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2a6ec0 0x55fcce2a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2a98f0 con 0x55fcce2a6ec0 2024-04-27T03:19:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835b986640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2a6ec0 0x55fcce2a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835b986640 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2a6ec0 0x55fcce2a92b0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835b986640 1 -- 172.21.15.110:0/4182675584 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2a6ec0 2024-04-27T03:19:15.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.059+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ab1a0 con 0x55fcce1acdc0 2024-04-27T03:19:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835c187640 1 -- 172.21.15.110:0/4182675584 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f835400a4d0 con 0x55fcce1acdc0 2024-04-27T03:19:15.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2aa3d0 con 0x55fcce2a6ec0 2024-04-27T03:19:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835b986640 1 -- 172.21.15.110:0/4182675584 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2a6ec0 2024-04-27T03:19:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2aa7c0 con 0x55fcce1acdc0 2024-04-27T03:19:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835c187640 1 -- 172.21.15.110:0/4182675584 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f835400a4d0 con 0x55fcce1acdc0 2024-04-27T03:19:15.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835de6b980 1 Processor -- start 2024-04-27T03:19:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835de6b980 1 -- start start 2024-04-27T03:19:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835de6b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2b9aa0 0x55fcce2b9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835de6b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fcce0b1310 con 0x55fcce2b9aa0 2024-04-27T03:19:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835c187640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2b9aa0 0x55fcce2b9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835c187640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2b9aa0 0x55fcce2b9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46012/0 (socket says 172.21.15.110:46012) 2024-04-27T03:19:15.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835c187640 1 -- 172.21.15.110:0/2091563663 learned_addr learned my addr 172.21.15.110:0/2091563663 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835c187640 1 -- 172.21.15.110:0/2091563663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcce2ba400 con 0x55fcce2b9aa0 2024-04-27T03:19:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835c187640 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2b9aa0 0x55fcce2b9ec0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f835400dfc0 tx=0x7f83540178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f82417fa640 1 -- 172.21.15.110:0/2091563663 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f83540157d0 con 0x55fcce2b9aa0 2024-04-27T03:19:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fcce2baea0 con 0x55fcce2b9aa0 2024-04-27T03:19:15.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.063+0000 7f82417fa640 1 -- 172.21.15.110:0/2091563663 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8354015970 con 0x55fcce2b9aa0 2024-04-27T03:19:15.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f82417fa640 1 -- 172.21.15.110:0/2091563663 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354015c50 con 0x55fcce2b9aa0 2024-04-27T03:19:15.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcce2bb370 con 0x55fcce2b9aa0 2024-04-27T03:19:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f82417fa640 1 -- 172.21.15.110:0/2091563663 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8354020440 con 0x55fcce2b9aa0 2024-04-27T03:19:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f82417fa640 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8234034650 0x7f8234036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f82417fa640 1 -- 172.21.15.110:0/2091563663 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f835400e040 con 0x55fcce2b9aa0 2024-04-27T03:19:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835b986640 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8234034650 0x7f8234036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 Processor -- start 2024-04-27T03:19:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- start start 2024-04-27T03:19:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835b986640 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8234034650 0x7f8234036b10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55fcce2a9ef0 tx=0x7f834c000bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2c7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fcce2983d0 con 0x55fcce2c6c50 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835c187640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835c187640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46026/0 (socket says 172.21.15.110:46026) 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835c187640 1 -- 172.21.15.110:0/297338582 learned_addr learned my addr 172.21.15.110:0/297338582 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835c187640 1 -- 172.21.15.110:0/297338582 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcce2c7640 con 0x55fcce2c6c50 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835c187640 1 --2- 172.21.15.110:0/297338582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2c7070 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f835404aaa0 tx=0x7f8354013870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd64d2f6171c9257 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f823affd640 1 -- 172.21.15.110:0/297338582 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354041070 con 0x55fcce2c6c50 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f823affd640 1 -- 172.21.15.110:0/297338582 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8354020b50 con 0x55fcce2c6c50 2024-04-27T03:19:15.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f823affd640 1 -- 172.21.15.110:0/297338582 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354011cc0 con 0x55fcce2c6c50 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- 172.21.15.110:0/297338582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 msgr2=0x55fcce2c7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 --2- 172.21.15.110:0/297338582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2c7070 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f835404aaa0 tx=0x7f8354013870 comp rx=0 tx=0).stop 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- 172.21.15.110:0/297338582 shutdown_connections 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 --2- 172.21.15.110:0/297338582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2c7070 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- 172.21.15.110:0/297338582 >> 172.21.15.110:0/297338582 conn(0x55fcce2c2fe0 msgr2=0x55fcce2c1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- 172.21.15.110:0/297338582 shutdown_connections 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.067+0000 7f835de6b980 1 -- 172.21.15.110:0/297338582 wait complete. 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 Processor -- start 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 -- start start 2024-04-27T03:19:15.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55fcce2c7d00 con 0x55fcce2c6c50 2024-04-27T03:19:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835c187640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835c187640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46042/0 (socket says 172.21.15.110:46042) 2024-04-27T03:19:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 learned_addr learned my addr 172.21.15.110:0/3263085039 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcce2ce5a0 con 0x55fcce2c6c50 2024-04-27T03:19:15.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835c187640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2ce060 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f835401e7d0 tx=0x7f835404aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f82397fa640 1 -- 172.21.15.110:0/3263085039 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354038070 con 0x55fcce2c6c50 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fcce2ce830 con 0x55fcce2c6c50 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f82397fa640 1 -- 172.21.15.110:0/3263085039 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8354011730 con 0x55fcce2c6c50 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcce2d1510 con 0x55fcce2c6c50 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f82397fa640 1 -- 172.21.15.110:0/3263085039 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8354041040 con 0x55fcce2c6c50 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f82397fa640 1 -- 172.21.15.110:0/3263085039 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f835403b070 con 0x55fcce2c6c50 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f82397fa640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8224034650 0x7f8224036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f82397fa640 1 -- 172.21.15.110:0/3263085039 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f8354034da0 con 0x55fcce2c6c50 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835b986640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8224034650 0x7f8224036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce2d9720 0x55fcce2dbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2dc2e0 con 0x55fcce2d9720 2024-04-27T03:19:15.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.071+0000 7f8353fff640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce2d9720 0x55fcce2dbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835b986640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8224034650 0x7f8224036b10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f834c0c5040 tx=0x7f834c0dc000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f8353fff640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce2d9720 0x55fcce2dbbe0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f834400e5a0 con 0x55fcce2d9720 2024-04-27T03:19:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 realm 2024-04-27T03:19:15.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2d8bc0 con 0x55fcce2d9720 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f834400e5a0 con 0x55fcce2d9720 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2dcde0 con 0x55fcce2d9720 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f834400e5a0 con 0x55fcce2d9720 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2ddbc0 0x55fcce2dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e0660 con 0x55fcce2ddbc0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835c187640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2ddbc0 0x55fcce2dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835c187640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2ddbc0 0x55fcce2dffb0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e0eb0 con 0x55fcce2d9720 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f834400e5a0 con 0x55fcce2d9720 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 20 searching for the correct realm 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fcce2e2c20 0x55fcce2e5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2e2c20 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835b986640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fcce2e2c20 0x55fcce2e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.075+0000 7f835b986640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fcce2e2c20 0x55fcce2e5010 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835c187640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f83540133b0 0x7f83540491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x7f83540133b0 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f83540133b0 0x7f83540491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f83540133b0 0x7f83540491f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834400e5a0 con 0x7f83540133b0 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834400e5a0 con 0x55fcce2d9720 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2e2c20 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.079+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x7f83540133b0 2024-04-27T03:19:15.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x7f83540133b0 2024-04-27T03:19:15.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x7f83540133b0 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f8344029040 con 0x7f83540133b0 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2e2c20 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2e2c20 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x7f83540133b0 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f8344029040 con 0x7f83540133b0 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2e2c20 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.083+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x7f83540133b0 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x7f83540133b0 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2d9720 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x7f83540133b0 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x7f83540133b0 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2e2c20 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x7f83540133b0 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8344029040 con 0x7f83540133b0 2024-04-27T03:19:15.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2e2c20 2024-04-27T03:19:15.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5690 con 0x55fcce2ddbc0 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.087+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e1830 con 0x55fcce2d9720 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8344029040 con 0x55fcce2d9720 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e1440 con 0x55fcce2ddbc0 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e2120 con 0x55fcce2d9720 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 zone default found 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 4 Realm: () 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 zonegroup default 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e5b50 con 0x7f83540133b0 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e7970 con 0x55fcce2e2c20 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e7d40 con 0x55fcce2e2c20 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 started sync module instance, tier type = 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:15.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.091+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e6e10 con 0x55fcce2ddbc0 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94544279269888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8650 con 0x55fcce2ddbc0 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8ea0 con 0x55fcce2d9720 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94544279275088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8fc0 con 0x55fcce2d9720 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e9ad0 con 0x55fcce2ddbc0 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94544279269888] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94544279275088] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94544279278976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e9a90 con 0x55fcce2ddbc0 2024-04-27T03:19:15.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.095+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ea370 con 0x7f83540133b0 2024-04-27T03:19:15.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94544279281344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e9a90 con 0x7f83540133b0 2024-04-27T03:19:15.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2eae00 con 0x55fcce2ddbc0 2024-04-27T03:19:15.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94544279281344] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94544279278976] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94544279273888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2eae00 con 0x55fcce2ddbc0 2024-04-27T03:19:15.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2eb1f0 con 0x55fcce2e2c20 2024-04-27T03:19:15.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94544279282400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2eba90 con 0x55fcce2e2c20 2024-04-27T03:19:15.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.099+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ec450 con 0x55fcce2e2c20 2024-04-27T03:19:15.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94544279273888] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94544279282400] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94544279284016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ec450 con 0x55fcce2e2c20 2024-04-27T03:19:15.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2eccf0 con 0x7f83540133b0 2024-04-27T03:19:15.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94544279284016] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'222 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94544279292448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ec450 con 0x7f83540133b0 2024-04-27T03:19:15.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=0 2024-04-27T03:19:15.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=1 2024-04-27T03:19:15.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=2 2024-04-27T03:19:15.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=3 2024-04-27T03:19:15.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=4 2024-04-27T03:19:15.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=5 2024-04-27T03:19:15.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=6 2024-04-27T03:19:15.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94544279292448] v29'223 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 20 add_watcher() i=7 2024-04-27T03:19:15.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:15.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2f01a0 con 0x55fcce2d9720 2024-04-27T03:19:15.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.103+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94544279290224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ed660 con 0x55fcce2d9720 2024-04-27T03:19:15.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.107+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94544279290224] v29'224 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.107+0000 7f8211fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:15.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.107+0000 7f835de6b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:15.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.107+0000 7f835de6b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:15.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.107+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2effa0 con 0x55fcce2ddbc0 2024-04-27T03:19:15.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.107+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v29'56 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.107+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2f0060 con 0x7f83540133b0 2024-04-27T03:19:15.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.111+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 gc.1 [create,call,call,call] v29'33 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.111+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2f94e0 con 0x55fcce2e2c20 2024-04-27T03:19:15.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.111+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v29'33 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.111+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2f9570 con 0x55fcce2d9720 2024-04-27T03:19:15.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.115+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(65 gc.3 [create,call,call,call] v29'225 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.115+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2f9ae0 con 0x55fcce2ddbc0 2024-04-27T03:19:15.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.115+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v29'76 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.115+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2f9ae0 con 0x55fcce2e2c20 2024-04-27T03:19:15.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.115+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v29'57 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.115+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa070 con 0x55fcce2ddbc0 2024-04-27T03:19:15.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.119+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v29'36 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.119+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa070 con 0x55fcce2d9720 2024-04-27T03:19:15.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.119+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 gc.7 [create,call,call,call] v29'52 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.119+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa520 con 0x55fcce2ddbc0 2024-04-27T03:19:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.123+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v29'26 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.123+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa520 con 0x55fcce2ddbc0 2024-04-27T03:19:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.127+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v29'32 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.127+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa5b0 con 0x55fcce2ddbc0 2024-04-27T03:19:15.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.127+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v29'77 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.127+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa5b0 con 0x55fcce2e2c20 2024-04-27T03:19:15.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.127+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v29'30 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.127+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa5b0 con 0x55fcce2ddbc0 2024-04-27T03:19:15.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.131+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v29'32 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.131+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2fa5b0 con 0x55fcce2d9720 2024-04-27T03:19:15.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.131+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(75 gc.13 [create,call,call,call] v29'63 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.131+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2d9720 2024-04-27T03:19:15.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.135+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(76 gc.14 [create,call,call,call] v29'226 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.135+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2ddbc0 2024-04-27T03:19:15.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.135+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v29'68 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.135+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2ddbc0 2024-04-27T03:19:15.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.139+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v29'78 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.139+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2e2c20 2024-04-27T03:19:15.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.139+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v29'39 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.139+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2ddbc0 2024-04-27T03:19:15.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.143+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v29'34 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.143+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2d9720 2024-04-27T03:19:15.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.143+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(81 gc.19 [create,call,call,call] v29'227 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.143+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2ddbc0 2024-04-27T03:19:15.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.147+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v29'69 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.147+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2ddbc0 2024-04-27T03:19:15.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.147+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v29'42 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.147+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2e2c20 2024-04-27T03:19:15.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.151+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v29'34 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.151+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2d9720 2024-04-27T03:19:15.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.151+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 gc.23 [create,call,call,call] v29'44 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.151+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2e2c20 2024-04-27T03:19:15.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.155+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v29'28 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.155+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2e2c20 2024-04-27T03:19:15.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.155+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v29'58 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.155+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2e2c20 2024-04-27T03:19:15.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.155+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v29'35 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.155+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2ddbc0 2024-04-27T03:19:15.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.159+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(89 gc.27 [create,call,call,call] v29'38 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.159+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2d9720 2024-04-27T03:19:15.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.159+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v29'53 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.159+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2ddbc0 2024-04-27T03:19:15.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.163+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(91 gc.29 [create,call,call,call] v29'70 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.163+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x55fcce2d9720 2024-04-27T03:19:15.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.163+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v29'64 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.163+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2faa60 con 0x7f83540133b0 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(93 gc.31 [create,call,call,call] v29'32 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f8210fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 20 Filter name: none 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f818a7fc640 20 reqs_thread_entry: start 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 cache get: name=default.rgw.meta+users.uid+bud : miss 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.4 8:267425a8:users.uid::bud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2f8c70 con 0x7f83540133b0 2024-04-27T03:19:15.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(94 bud [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv2 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:19:15.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 cache put: name=default.rgw.meta+users.uid+bud info.flags=0x13 2024-04-27T03:19:15.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 adding default.rgw.meta+users.uid+bud to cache LRU end 2024-04-27T03:19:15.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:15.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318440 con 0x55fcce2ddbc0 2024-04-27T03:19:15.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(95 bud.buckets [call out=184b] v0'0 uv4 ondisk = 0) ==== 155+0+184 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 : miss 2024-04-27T03:19:15.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:96 8.a 8:557ee8e5:root::.bucket.meta.myfoo6%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.9:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318230 con 0x55fcce2d9720 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(96 .bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 [call out=48b,stat out=16b,read 0~380 out=380b,getxattrs out=212b] v0'0 uv4 ondisk = 0) ==== 333+0+656 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 info.flags=0x17 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 adding default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 to cache LRU end 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 updating xattr: name=user.rgw.acl bl.length()=117 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:97 4.27 4:e5f3ed89:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce395760 con 0x55fcce2d9720 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(97 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:98 4.14 4:28eb2f22:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce395760 con 0x55fcce2e2c20 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(98 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.167+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 4.d 4:b1be13f2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce395760 con 0x55fcce2ddbc0 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(99 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:100 4.11 4:8931bf6f:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3957a0 con 0x7f83540133b0 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(100 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:101 4.7 4:e007ada1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3957a0 con 0x55fcce2d9720 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(101 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:102 4.c 4:33587581:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3957a0 con 0x55fcce2d9720 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(102 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:103 4.3b 4:df87161c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3957a0 con 0x55fcce2e2c20 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(103 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6 [call out=85b] v0'0 uv10 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:104 4.3 4:c1fac81b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3957a0 con 0x55fcce2d9720 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(104 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:105 4.b 4:d31bf6c6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce395760 con 0x7f83540133b0 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(105 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8 [call out=85b] v0'0 uv11 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834403c1c0 con 0x7f83540133b0 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:106 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce395760 con 0x55fcce2d9720 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(106 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [call out=289b] v0'0 uv7 ondisk = 0) ==== 194+0+289 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 10 cls_bucket_list_unordered: got thirteen 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:107 4.10 4:0b987959:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce395760 con 0x55fcce2e2c20 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(107 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 20 get_obj_state: octx=0x7f834400e520 obj=myfoo6:thirteen state=0x55fcce393808 s->prefetch_data=0 2024-04-27T03:19:15.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.171+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 3.3as0 3:5d3a90bb:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce394630 con 0x55fcce2ddbc0 2024-04-27T03:19:15.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(108 27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen [getxattrs out=820b,stat out=16b] v0'0 uv2 ondisk = 0) ==== 238+0+836 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835de6b980 10 manifest: total_size = 8 2024-04-27T03:19:15.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835de6b980 20 get_obj_state: setting s->obj_tag to _t-rV96CCcgJGL3c6G0w_IDrCJxscoun 2024-04-27T03:19:15.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835de6b980 20 get_obj_state: octx=0x7f834400e520 obj=myfoo6:thirteen state=0x55fcce393808 s->prefetch_data=0 2024-04-27T03:19:15.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835de6b980 20 prepare_atomic_modification: state is not atomic. state=0x55fcce393808 2024-04-27T03:19:15.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835de6b980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 2024-04-27T03:19:15.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:109 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=187b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce394ef0 con 0x55fcce2d9720 2024-04-27T03:19:15.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(109 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [stat,call,call] v29'8 uv8 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f834403c1c0 con 0x55fcce2d9720 2024-04-27T03:19:15.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.175+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:110 3.3as0 3:5d3a90bb:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce394ec0 con 0x55fcce2ddbc0 2024-04-27T03:19:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(110 27a51273-3ccc-49e2-8682-1122abc3d898.4220.9_thirteen [call] v29'3 uv3 ondisk = 0) ==== 196+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:111 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=264b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce394e40 con 0x55fcce2d9720 2024-04-27T03:19:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 20 list_objects_ordered: starting attempt 1 2024-04-27T03:19:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2024-04-27T03:19:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:112 4.27 4:e5f3ed89:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce394e40 con 0x55fcce2d9720 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:113 4.14 4:28eb2f22:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce320d20 con 0x55fcce2e2c20 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:114 4.d 4:b1be13f2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce394e40 con 0x55fcce2ddbc0 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:115 4.11 4:8931bf6f:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce320d20 con 0x7f83540133b0 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:116 4.7 4:e007ada1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce394e40 con 0x55fcce2d9720 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:117 4.c 4:33587581:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce32a560 con 0x55fcce2d9720 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:118 4.3b 4:df87161c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce32d3c0 con 0x55fcce2e2c20 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(112 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(113 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(115 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:119 4.3 4:c1fac81b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce32d3c0 con 0x55fcce2d9720 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(118 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6 [call out=85b] v0'0 uv10 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(116 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(117 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:120 4.b 4:d31bf6c6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x7f83540133b0 2024-04-27T03:19:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:121 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce329a10 con 0x55fcce2d9720 2024-04-27T03:19:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:122 4.10 4:0b987959:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce32a000 con 0x55fcce2e2c20 2024-04-27T03:19:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(114 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(119 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(120 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8 [call out=85b] v0'0 uv11 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.179+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(122 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(111 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [stat,call,call] v29'9 uv9 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f823a7fc640 20 handle_completion(): completion ok for obj=thirteen 2024-04-27T03:19:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(121 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [call out=124b] v0'0 uv9 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-04-27T03:19:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-04-27T03:19:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-04-27T03:19:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo6[27a51273-3ccc-49e2-8682-1122abc3d898.4220.9])"; uploads.size()=0; is_truncated=0 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:123 4.27 4:e5f3ed89:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:124 4.14 4:28eb2f22:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce329450 con 0x55fcce2e2c20 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:125 4.d 4:b1be13f2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2ddbc0 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:126 4.11 4:8931bf6f:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce320870 con 0x7f83540133b0 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:127 4.7 4:e007ada1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce329450 con 0x55fcce2d9720 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:128 4.c 4:33587581:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:129 4.3b 4:df87161c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce329450 con 0x55fcce2e2c20 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:130 4.3 4:c1fac81b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:131 4.b 4:d31bf6c6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce329450 con 0x7f83540133b0 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:132 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(123 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:133 4.10 4:0b987959:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2e2c20 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(124 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(129 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6 [call out=85b] v0'0 uv10 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(126 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 42 ==== osd_op_reply(128 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 43 ==== osd_op_reply(130 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(131 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8 [call out=85b] v0'0 uv11 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(133 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 44 ==== osd_op_reply(127 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 45 ==== osd_op_reply(132 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [call out=124b] v0'0 uv9 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(125 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:134 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.set_buckets_info in=208b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2ddbc0 2024-04-27T03:19:15.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(134 bud.buckets [call] v29'5 uv5 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 10 cache get: name=default.rgw.meta+root+myfoo6 : miss 2024-04-27T03:19:15.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.183+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:135 8.18 8:1944a257:root::myfoo6:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 46 ==== osd_op_reply(135 myfoo6 [call out=48b,read 0~166 out=166b] v0'0 uv19 ondisk = 0) ==== 192+0+214 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 20 rados_obj.operate() r=0 bl.length=166 2024-04-27T03:19:15.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 10 cache put: name=default.rgw.meta+root+myfoo6 info.flags=0x11 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 10 adding default.rgw.meta+root+myfoo6 to cache LRU end 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:136 8.18 8:1944a257:root::myfoo6:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 47 ==== osd_op_reply(136 myfoo6 [call,call,delete] v29'20 uv20 ondisk = 0) ==== 234+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 10 removing default.rgw.meta+root+myfoo6 from cache 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:myfoo6, ofs0, ns] 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:137 7.e 7:7759931f:::notify.3:head [notify cookie 94544279543536 in=162b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x7f83540133b0 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 22 ==== watch-notify(notify (1) cookie 94544279281344 notify 124554051599 ret 0) ==== 192+0+0 (crc 0 0 0) 0x55fcce317740 con 0x7f83540133b0 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f823affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051599 cookie 94544279281344 notifier 4744 bl.length()=150 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 23 ==== osd_op_reply(137 notify.3 [notify cookie 94544279543536 out=8b] v0'0 uv222 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f823affd640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:138 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f8220003d70 con 0x7f83540133b0 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 24 ==== osd_op_reply(138 notify.3 [notify-ack] v0'0 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 25 ==== watch-notify(notify_complete (2) cookie 94544279543536 notify 124554051599 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55fcce2efaf0 con 0x7f83540133b0 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:139 8.a 8:557ee8e5:root::.bucket.meta.myfoo6%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.9:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 48 ==== osd_op_reply(139 .bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 [call,call,delete] v29'6 uv6 ondisk = 0) ==== 291+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 10 removing default.rgw.meta+root+.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 from cache 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo6:27a51273-3ccc-49e2-8682-1122abc3d898.4220.9, ofs0, ns] 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:140 7.3 7:c609908c:::notify.5:head [notify cookie 94544279543536 in=219b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2e2c20 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== watch-notify(notify (1) cookie 94544279282400 notify 124554051612 ret 0) ==== 249+0+0 (crc 0 0 0) 0x55fcce2fa070 con 0x55fcce2e2c20 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f823affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051612 cookie 94544279282400 notifier 4744 bl.length()=207 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(140 notify.5 [notify cookie 94544279543536 out=8b] v0'0 uv222 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f823affd640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:141 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f8220003d70 con 0x55fcce2e2c20 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== osd_op_reply(141 notify.5 [notify-ack] v0'0 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== watch-notify(notify_complete (2) cookie 94544279543536 notify 124554051612 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f834c0e3040 con 0x55fcce2e2c20 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:142 4.27 4:e5f3ed89:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:143 4.14 4:28eb2f22:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318150 con 0x55fcce2e2c20 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:144 4.d 4:b1be13f2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3283e0 con 0x55fcce2ddbc0 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:145 4.11 4:8931bf6f:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328cb0 con 0x7f83540133b0 2024-04-27T03:19:15.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:146 4.7 4:e007ada1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2d9720 2024-04-27T03:19:15.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:147 4.c 4:33587581:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318150 con 0x55fcce2d9720 2024-04-27T03:19:15.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.187+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:148 4.3b 4:df87161c:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328cb0 con 0x55fcce2e2c20 2024-04-27T03:19:15.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:149 4.3 4:c1fac81b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3283e0 con 0x55fcce2d9720 2024-04-27T03:19:15.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:150 4.b 4:d31bf6c6:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328cb0 con 0x7f83540133b0 2024-04-27T03:19:15.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:151 4.35 4:ac431417:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318150 con 0x55fcce2d9720 2024-04-27T03:19:15.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:152 4.10 4:0b987959:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce328000 con 0x55fcce2e2c20 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 49 ==== osd_op_reply(147 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.5 [delete] v29'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 50 ==== osd_op_reply(146 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.4 [delete] v29'3 uv3 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 51 ==== osd_op_reply(142 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.0 [delete] v29'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(143 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.1 [delete] v29'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 52 ==== osd_op_reply(149 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.7 [delete] v29'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 26 ==== osd_op_reply(150 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.8 [delete] v29'12 uv12 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== osd_op_reply(152 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.10 [delete] v29'3 uv3 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(144 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.2 [delete] v29'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(148 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.6 [delete] v29'11 uv11 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 27 ==== osd_op_reply(145 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.3 [delete] v29'6 uv6 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 53 ==== osd_op_reply(151 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.9.9 [delete] v29'10 uv10 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 10 cache get: name=default.rgw.log++pubsub..bucket.myfoo6/27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 : miss 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:153 6.18 6:1a8918e8:::pubsub..bucket.myfoo6%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.9:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318150 con 0x7f83540133b0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 28 ==== osd_op_reply(153 pubsub..bucket.myfoo6/27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 209+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 10 cache put: name=default.rgw.log++pubsub..bucket.myfoo6/27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 info.flags=0x0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 10 adding default.rgw.log++pubsub..bucket.myfoo6/27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 to cache LRU end 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.191+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:154 6.18 6:1a8918e8:::pubsub..bucket.myfoo6%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318150 con 0x7f83540133b0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 29 ==== osd_op_reply(154 pubsub..bucket.myfoo6/27a51273-3ccc-49e2-8682-1122abc3d898.4220.9 [delete] v29'34 uv28 ondisk = -2 ((2) No such file or directory)) ==== 209+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:155 8.19 8:99adfd5d:users.uid::bud.buckets:head [call user.remove_bucket in=145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318150 con 0x55fcce2ddbc0 2024-04-27T03:19:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(155 bud.buckets [call] v29'6 uv6 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835de6b980 10 removing key index: NX5QOQKC6BH2IDN8HC7A 2024-04-27T03:19:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:156 8.a 8:50c8eddb:users.keys::NX5QOQKC6BH2IDN8HC7A:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318120 con 0x55fcce2d9720 2024-04-27T03:19:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 54 ==== osd_op_reply(156 NX5QOQKC6BH2IDN8HC7A [delete] v29'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835de6b980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2024-04-27T03:19:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:157 7.3 7:c609908c:::notify.5:head [notify cookie 94544279470368 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce32cb90 con 0x55fcce2e2c20 2024-04-27T03:19:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== watch-notify(notify (1) cookie 94544279282400 notify 124554051613 ret 0) ==== 212+0+0 (crc 0 0 0) 0x7f834c0e3040 con 0x55fcce2e2c20 2024-04-27T03:19:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f823a7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051613 cookie 94544279282400 notifier 4744 bl.length()=170 2024-04-27T03:19:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== osd_op_reply(157 notify.5 [notify cookie 94544279470368 out=8b] v0'0 uv222 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.195+0000 7f823a7fc640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:158 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f81900032d0 con 0x55fcce2e2c20 2024-04-27T03:19:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== watch-notify(notify_complete (2) cookie 94544279470368 notify 124554051613 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55fcce2a2070 con 0x55fcce2e2c20 2024-04-27T03:19:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== osd_op_reply(158 notify.5 [notify-ack] v0'0 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 10 removing email index: 2024-04-27T03:19:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 10 removing user buckets index 2024-04-27T03:19:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:159 8.19 8:99adfd5d:users.uid::bud.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318120 con 0x55fcce2ddbc0 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(159 bud.buckets [delete] v29'7 uv7 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:bud.buckets, ofs0, ns] 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:160 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94544279470368 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce32cb90 con 0x55fcce2e2c20 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== watch-notify(notify (1) cookie 94544279284016 notify 124554051614 ret 0) ==== 202+0+0 (crc 0 0 0) 0x55fcce2a2070 con 0x55fcce2e2c20 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f823affd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051614 cookie 94544279284016 notifier 4744 bl.length()=160 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== osd_op_reply(160 notify.6 [notify cookie 94544279470368 out=8b] v0'0 uv222 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f823affd640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:161 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f8220003d70 con 0x55fcce2e2c20 2024-04-27T03:19:15.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(161 notify.6 [notify-ack] v0'0 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== watch-notify(notify_complete (2) cookie 94544279470368 notify 124554051614 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55fcce29f490 con 0x55fcce2e2c20 2024-04-27T03:19:15.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 10 removing user index: bud 2024-04-27T03:19:15.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:162 8.4 8:267425a8:users.uid::bud:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce318120 con 0x7f83540133b0 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 30 ==== osd_op_reply(162 bud [call,call,delete] v29'3 uv3 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.199+0000 7f835de6b980 10 removing default.rgw.meta+users.uid+bud from cache 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835de6b980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:bud, ofs0, ns] 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:163 7.3 7:c609908c:::notify.5:head [notify cookie 94544279470368 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55fcce32cb90 con 0x55fcce2e2c20 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 46 ==== watch-notify(notify (1) cookie 94544279282400 notify 124554051615 ret 0) ==== 194+0+0 (crc 0 0 0) 0x55fcce29f490 con 0x55fcce2e2c20 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f823a7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051615 cookie 94544279282400 notifier 4744 bl.length()=152 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 47 ==== osd_op_reply(163 notify.5 [notify cookie 94544279470368 out=8b] v0'0 uv222 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f823a7fc640 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:164 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f81900032d0 con 0x55fcce2e2c20 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 48 ==== osd_op_reply(164 notify.5 [notify-ack] v0'0 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 49 ==== watch-notify(notify_complete (2) cookie 94544279470368 notify 124554051615 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55fcce29f200 con 0x55fcce2e2c20 2024-04-27T03:19:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f8211fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:165 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94544279290224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce3180f0 con 0x55fcce2d9720 2024-04-27T03:19:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 55 ==== osd_op_reply(165 data_loggenerations_metadata [watch unwatch cookie 94544279290224] v29'228 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.203+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:166 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94544279269888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ec140 con 0x55fcce2ddbc0 2024-04-27T03:19:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.207+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(166 notify.0 [watch unwatch cookie 94544279269888] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.207+0000 7f835de6b980 20 remove_watcher() i=0 2024-04-27T03:19:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.207+0000 7f835de6b980 2 removed watcher, disabling cache 2024-04-27T03:19:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.207+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:167 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94544279275088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ec140 con 0x55fcce2d9720 2024-04-27T03:19:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.207+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.1 v2:172.21.15.110:6800/1860462708 56 ==== osd_op_reply(167 notify.1 [watch unwatch cookie 94544279275088] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8344051040 con 0x55fcce2d9720 2024-04-27T03:19:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.207+0000 7f835de6b980 20 remove_watcher() i=1 2024-04-27T03:19:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.207+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:168 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94544279278976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8650 con 0x55fcce2ddbc0 2024-04-27T03:19:15.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.211+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(168 notify.2 [watch unwatch cookie 94544279278976] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.211+0000 7f835de6b980 20 remove_watcher() i=2 2024-04-27T03:19:15.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.211+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:169 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94544279281344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8650 con 0x7f83540133b0 2024-04-27T03:19:15.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.211+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 31 ==== osd_op_reply(169 notify.3 [watch unwatch cookie 94544279281344] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.211+0000 7f835de6b980 20 remove_watcher() i=3 2024-04-27T03:19:15.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.211+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:170 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94544279273888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8650 con 0x55fcce2ddbc0 2024-04-27T03:19:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835c187640 1 -- 172.21.15.110:0/3263085039 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== osd_op_reply(170 notify.4 [watch unwatch cookie 94544279273888] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f835403a030 con 0x55fcce2ddbc0 2024-04-27T03:19:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 20 remove_watcher() i=4 2024-04-27T03:19:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:171 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94544279282400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8650 con 0x55fcce2e2c20 2024-04-27T03:19:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 50 ==== osd_op_reply(171 notify.5 [watch unwatch cookie 94544279282400] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 20 remove_watcher() i=5 2024-04-27T03:19:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:172 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94544279284016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2e8650 con 0x55fcce2e2c20 2024-04-27T03:19:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835b986640 1 -- 172.21.15.110:0/3263085039 <== osd.3 v2:172.21.15.110:6816/3916647200 51 ==== osd_op_reply(172 notify.6 [watch unwatch cookie 94544279284016] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f834c0c7030 con 0x55fcce2e2c20 2024-04-27T03:19:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 20 remove_watcher() i=6 2024-04-27T03:19:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:173 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94544279292448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55fcce2ea930 con 0x7f83540133b0 2024-04-27T03:19:15.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 <== osd.2 v2:172.21.15.110:6808/3575017494 32 ==== osd_op_reply(173 notify.7 [watch unwatch cookie 94544279292448] v29'224 uv222 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8344051040 con 0x7f83540133b0 2024-04-27T03:19:15.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 20 remove_watcher() i=7 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2ddbc0 msgr2=0x55fcce2dffb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2ddbc0 0x55fcce2dffb0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce2d9720 msgr2=0x55fcce2dbbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce2d9720 0x55fcce2dbbe0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f83540133b0 msgr2=0x7f83540491f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f83540133b0 0x7f83540491f0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fcce2e2c20 msgr2=0x55fcce2e5010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55fcce2e2c20 0x55fcce2e5010 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8224034650 msgr2=0x7f8224036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8224034650 0x7f8224036b10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f834c0c5040 tx=0x7f834c0dc000 comp rx=0 tx=0).stop 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f8353fff640 1 -- 172.21.15.110:0/3263085039 reap_dead start 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 msgr2=0x55fcce2ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2ce060 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f835401e7d0 tx=0x7f835404aa30 comp rx=0 tx=0).stop 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 shutdown_connections 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/3263085039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2c6c50 0x55fcce2ce060 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 >> 172.21.15.110:0/3263085039 conn(0x55fcce2c2fe0 msgr2=0x55fcce2c0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 shutdown_connections 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/3263085039 wait complete. 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8234034650 msgr2=0x7f8234036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8234034650 0x7f8234036b10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55fcce2a9ef0 tx=0x7f834c000bd0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2b9aa0 msgr2=0x55fcce2b9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.215+0000 7f835de6b980 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2b9aa0 0x55fcce2b9ec0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f835400dfc0 tx=0x7f83540178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 shutdown_connections 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8234034650 0x7f8234036b10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/2091563663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce2b9aa0 0x55fcce2b9ec0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 >> 172.21.15.110:0/2091563663 conn(0x55fcce2ae8b0 msgr2=0x55fcce2af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 shutdown_connections 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/2091563663 wait complete. 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2a6ec0 msgr2=0x55fcce2a92b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2a6ec0 0x55fcce2a92b0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce1acdc0 msgr2=0x55fcce2a5b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce1acdc0 0x55fcce2a5b50 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8244034650 msgr2=0x7f8244036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8244034650 0x7f8244036b10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8344000f70 tx=0x7f83440012c0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 msgr2=0x55fcce29ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce29ecc0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f834c01bb20 tx=0x7f834c018020 comp rx=0 tx=0).stop 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 shutdown_connections 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55fcce2a6ec0 0x55fcce2a92b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55fcce1acdc0 0x55fcce2a5b50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8244034650 0x7f8244036b10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 --2- 172.21.15.110:0/4182675584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55fcce1ac4a0 0x55fcce29ecc0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 >> 172.21.15.110:0/4182675584 conn(0x55fcce11fba0 msgr2=0x55fcce2923e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 shutdown_connections 2024-04-27T03:19:15.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.219+0000 7f835de6b980 1 -- 172.21.15.110:0/4182675584 wait complete. 2024-04-27T03:19:15.237 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'fud', '--purge-data'] 2024-04-27T03:19:15.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'fud', '--purge-data'] 2024-04-27T03:19:15.237 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid fud --purge-data 2024-04-27T03:19:15.314 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:15.314 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:15.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.303+0000 7f16bd456980 1 Processor -- start 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16bd456980 1 -- start start 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16bd456980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a86b4809d0 0x55a86b482dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16bd456980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a86b3390d0 con 0x55a86b47c4c0 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16bd456980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a86b32e430 con 0x55a86b4809d0 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16ba770640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a86b4809d0 0x55a86b482dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16baf71640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55a86b47c4c0 0x55a86b47c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:51298/0 (socket says 172.21.15.110:51298) 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16baf71640 1 -- 172.21.15.110:0/3589407014 learned_addr learned my addr 172.21.15.110:0/3589407014 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:15.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16ba770640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a86b4809d0 0x55a86b482dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46054/0 (socket says 172.21.15.110:46054) 2024-04-27T03:19:15.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16ba770640 1 -- 172.21.15.110:0/3589407014 >> v1:172.21.15.77:6789/0 conn(0x55a86b47c4c0 legacy=0x55a86b47c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16ba770640 1 -- 172.21.15.110:0/3589407014 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b327510 con 0x55a86b4809d0 2024-04-27T03:19:15.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16ba770640 1 --2- 172.21.15.110:0/3589407014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b4809d0 0x55a86b482dc0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f16b000a480 tx=0x7f16b001b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df890567bb633dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.307+0000 7f16b9f6f640 1 -- 172.21.15.110:0/3589407014 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b00b0070 con 0x55a86b4809d0 2024-04-27T03:19:15.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16b9f6f640 1 -- 172.21.15.110:0/3589407014 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b00ab540 con 0x55a86b4809d0 2024-04-27T03:19:15.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16b9f6f640 1 -- 172.21.15.110:0/3589407014 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b00ab840 con 0x55a86b4809d0 2024-04-27T03:19:15.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16bd456980 1 -- 172.21.15.110:0/3589407014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b4809d0 msgr2=0x55a86b482dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16bd456980 1 --2- 172.21.15.110:0/3589407014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b4809d0 0x55a86b482dc0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f16b000a480 tx=0x7f16b001b280 comp rx=0 tx=0).stop 2024-04-27T03:19:15.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16bd456980 1 -- 172.21.15.110:0/3589407014 shutdown_connections 2024-04-27T03:19:15.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16bd456980 1 --2- 172.21.15.110:0/3589407014 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b4809d0 0x55a86b482dc0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16bd456980 1 -- 172.21.15.110:0/3589407014 >> 172.21.15.110:0/3589407014 conn(0x55a86b3efba0 msgr2=0x55a86b3effb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16bd456980 1 -- 172.21.15.110:0/3589407014 shutdown_connections 2024-04-27T03:19:15.320 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.311+0000 7f16bd456980 1 -- 172.21.15.110:0/3589407014 wait complete. 2024-04-27T03:19:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16bd456980 1 Processor -- start 2024-04-27T03:19:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16bd456980 1 -- start start 2024-04-27T03:19:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16bd456980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b5675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16bd456980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86b37d2d0 con 0x55a86b47c4c0 2024-04-27T03:19:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16baf71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b5675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16baf71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b5675e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46062/0 (socket says 172.21.15.110:46062) 2024-04-27T03:19:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16baf71640 1 -- 172.21.15.110:0/3702492959 learned_addr learned my addr 172.21.15.110:0/3702492959 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16baf71640 1 -- 172.21.15.110:0/3702492959 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b303ae0 con 0x55a86b47c4c0 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.323+0000 7f16baf71640 1 --2- 172.21.15.110:0/3702492959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b5675e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0015f0 tx=0x7f16ac001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ce2bc9b603d22c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f15a8e09640 1 -- 172.21.15.110:0/3702492959 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ac004530 con 0x55a86b47c4c0 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f15a8e09640 1 -- 172.21.15.110:0/3702492959 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ac0046f0 con 0x55a86b47c4c0 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f15a8e09640 1 -- 172.21.15.110:0/3702492959 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ac004a90 con 0x55a86b47c4c0 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- 172.21.15.110:0/3702492959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 msgr2=0x55a86b5675e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 --2- 172.21.15.110:0/3702492959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b5675e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0015f0 tx=0x7f16ac001ab0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.337 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- 172.21.15.110:0/3702492959 shutdown_connections 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 --2- 172.21.15.110:0/3702492959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b5675e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- 172.21.15.110:0/3702492959 >> 172.21.15.110:0/3702492959 conn(0x55a86b3efba0 msgr2=0x55a86b563170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- 172.21.15.110:0/3702492959 shutdown_connections 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- 172.21.15.110:0/3702492959 wait complete. 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 Processor -- start 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- start start 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b56e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86b485df0 con 0x55a86b47c4c0 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16baf71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b56e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16baf71640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b56e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46068/0 (socket says 172.21.15.110:46068) 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16baf71640 1 -- 172.21.15.110:0/258453509 learned_addr learned my addr 172.21.15.110:0/258453509 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16baf71640 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b56eb60 con 0x55a86b47c4c0 2024-04-27T03:19:15.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16baf71640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b56e620 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x55a86b568470 tx=0x7f16ac000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f15a7606640 1 -- 172.21.15.110:0/258453509 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ac013760 con 0x55a86b47c4c0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a86b56edf0 con 0x55a86b47c4c0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86b571a70 con 0x55a86b47c4c0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f15a7606640 1 -- 172.21.15.110:0/258453509 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ac013900 con 0x55a86b47c4c0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.327+0000 7f15a7606640 1 -- 172.21.15.110:0/258453509 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16ac013be0 con 0x55a86b47c4c0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f15a7606640 1 -- 172.21.15.110:0/258453509 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f16ac010070 con 0x55a86b47c4c0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f15a7606640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15a0034650 0x7f15a0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f15a7606640 1 -- 172.21.15.110:0/258453509 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f16ac018070 con 0x55a86b47c4c0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16ba770640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15a0034650 0x7f15a0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b4809d0 0x55a86b575550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b575b20 con 0x55a86b4809d0 2024-04-27T03:19:15.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bb772640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b4809d0 0x55a86b575550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16ba770640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15a0034650 0x7f15a0036b10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f16b00ad000 tx=0x7f16b00a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bb772640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b4809d0 0x55a86b575550 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bb772640 1 -- 172.21.15.110:0/258453509 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b4809d0 2024-04-27T03:19:15.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b576910 0x55a86b578d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b579340 con 0x55a86b576910 2024-04-27T03:19:15.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16baf71640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b576910 0x55a86b578d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16baf71640 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b576910 0x55a86b578d00 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16baf71640 1 -- 172.21.15.110:0/258453509 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f16ac01b030 con 0x55a86b576910 2024-04-27T03:19:15.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b57abf0 con 0x55a86b4809d0 2024-04-27T03:19:15.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bb772640 1 -- 172.21.15.110:0/258453509 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b4809d0 2024-04-27T03:19:15.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.331+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b579e20 con 0x55a86b576910 2024-04-27T03:19:15.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16baf71640 1 -- 172.21.15.110:0/258453509 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f16ac01b030 con 0x55a86b576910 2024-04-27T03:19:15.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b57a240 con 0x55a86b4809d0 2024-04-27T03:19:15.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bb772640 1 -- 172.21.15.110:0/258453509 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b4809d0 2024-04-27T03:19:15.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bd456980 1 Processor -- start 2024-04-27T03:19:15.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bd456980 1 -- start start 2024-04-27T03:19:15.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bd456980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b589500 0x55a86b589920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bd456980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86b381310 con 0x55a86b589500 2024-04-27T03:19:15.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bb772640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b589500 0x55a86b589920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bb772640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b589500 0x55a86b589920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46080/0 (socket says 172.21.15.110:46080) 2024-04-27T03:19:15.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bb772640 1 -- 172.21.15.110:0/2544288380 learned_addr learned my addr 172.21.15.110:0/2544288380 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bb772640 1 -- 172.21.15.110:0/2544288380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b589e60 con 0x55a86b589500 2024-04-27T03:19:15.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bb772640 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b589500 0x55a86b589920 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f16b400aab0 tx=0x7f16b400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f15a4e01640 1 -- 172.21.15.110:0/2544288380 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b4007d30 con 0x55a86b589500 2024-04-27T03:19:15.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a86b58a900 con 0x55a86b589500 2024-04-27T03:19:15.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86b58add0 con 0x55a86b589500 2024-04-27T03:19:15.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f15a4e01640 1 -- 172.21.15.110:0/2544288380 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b401d070 con 0x55a86b589500 2024-04-27T03:19:15.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.335+0000 7f15a4e01640 1 -- 172.21.15.110:0/2544288380 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b4022460 con 0x55a86b589500 2024-04-27T03:19:15.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f15a4e01640 1 -- 172.21.15.110:0/2544288380 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f16b4013070 con 0x55a86b589500 2024-04-27T03:19:15.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f15a4e01640 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15940346a0 0x7f1594036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f15a4e01640 1 -- 172.21.15.110:0/2544288380 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f16b404b5b0 con 0x55a86b589500 2024-04-27T03:19:15.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16baf71640 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15940346a0 0x7f1594036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 Processor -- start 2024-04-27T03:19:15.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- start start 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16baf71640 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15940346a0 0x7f1594036b60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55a86b5760d0 tx=0x7f16ac01e470 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b596ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86b567d30 con 0x55a86b5966b0 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bb772640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b596ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bb772640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b596ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46084/0 (socket says 172.21.15.110:46084) 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bb772640 1 -- 172.21.15.110:0/3898371286 learned_addr learned my addr 172.21.15.110:0/3898371286 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bb772640 1 -- 172.21.15.110:0/3898371286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b5970a0 con 0x55a86b5966b0 2024-04-27T03:19:15.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bb772640 1 --2- 172.21.15.110:0/3898371286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b596ad0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f16b40208a0 tx=0x7f16b4021ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa09ee383bf1479 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f159e7fc640 1 -- 172.21.15.110:0/3898371286 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b403b290 con 0x55a86b5966b0 2024-04-27T03:19:15.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f159e7fc640 1 -- 172.21.15.110:0/3898371286 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b4022ad0 con 0x55a86b5966b0 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f159e7fc640 1 -- 172.21.15.110:0/3898371286 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b40213c0 con 0x55a86b5966b0 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- 172.21.15.110:0/3898371286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 msgr2=0x55a86b596ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 --2- 172.21.15.110:0/3898371286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b596ad0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f16b40208a0 tx=0x7f16b4021ce0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- 172.21.15.110:0/3898371286 shutdown_connections 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 --2- 172.21.15.110:0/3898371286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b596ad0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- 172.21.15.110:0/3898371286 >> 172.21.15.110:0/3898371286 conn(0x55a86b5929f0 msgr2=0x55a86b590a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- 172.21.15.110:0/3898371286 shutdown_connections 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- 172.21.15.110:0/3898371286 wait complete. 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 Processor -- start 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- start start 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b59dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.339+0000 7f16bd456980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a86b597800 con 0x55a86b5966b0 2024-04-27T03:19:15.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bb772640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b59dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bb772640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b59dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:46088/0 (socket says 172.21.15.110:46088) 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 learned_addr learned my addr 172.21.15.110:0/2276899801 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86b59e110 con 0x55a86b5966b0 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bb772640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b59dbd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f16b40209b0 tx=0x7f16b4049850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f159cff9640 1 -- 172.21.15.110:0/2276899801 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b4040040 con 0x55a86b5966b0 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a86b59e3a0 con 0x55a86b5966b0 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f159cff9640 1 -- 172.21.15.110:0/2276899801 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b4022600 con 0x55a86b5966b0 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f159cff9640 1 -- 172.21.15.110:0/2276899801 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b40228e0 con 0x55a86b5966b0 2024-04-27T03:19:15.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86b5a1080 con 0x55a86b5966b0 2024-04-27T03:19:15.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f159cff9640 1 -- 172.21.15.110:0/2276899801 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f16b4032020 con 0x55a86b5966b0 2024-04-27T03:19:15.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f159cff9640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1584034650 0x7f1584036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f159cff9640 1 -- 172.21.15.110:0/2276899801 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6400+0+0 (secure 0 0 0) 0x7f16b4030020 con 0x55a86b5966b0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16baf71640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1584034650 0x7f1584036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b5a91d0 0x55a86b5ab690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5abde0 con 0x55a86b5a91d0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16ba770640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b5a91d0 0x55a86b5ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16baf71640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1584034650 0x7f1584036b10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f16ac010910 tx=0x7f16ac037000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16ba770640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b5a91d0 0x55a86b5ab690 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 20 realm 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5ac210 con 0x55a86b5a91d0 2024-04-27T03:19:15.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.343+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5acae0 con 0x55a86b5a91d0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b5ad500 0x55a86b5af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b00a0 con 0x55a86b5ad500 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bb772640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b5ad500 0x55a86b5af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bb772640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b5ad500 0x55a86b5af930 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b08f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 20 searching for the correct realm 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a86b5b2660 0x55a86b5b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5b2660 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16baf71640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a86b5b2660 0x55a86b5b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16baf71640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a86b5b2660 0x55a86b5b4a50 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.347+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16bb772640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16b4037100 0x7f16b4013a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:15.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x7f16b4037100 2024-04-27T03:19:15.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16b4037100 0x7f16b4013a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:15.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16b4037100 0x7f16b4013a00 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:15.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5b2660 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 29 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x7f16b4037100 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.351+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x7f16b4037100 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 29 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5b2660 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 29 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5b2660 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x7f16b4037100 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 29 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5b2660 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.355+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x7f16b4037100 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5a91d0 2024-04-27T03:19:15.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x7f16b4037100 2024-04-27T03:19:15.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5b2660 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x7f16b4037100 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5b2660 2024-04-27T03:19:15.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 29 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.359+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 29 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5060 con 0x55a86b5ad500 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 29 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b1270 con 0x55a86b5a91d0 2024-04-27T03:19:15.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b1b10 con 0x55a86b5ad500 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b0eb0 con 0x55a86b5a91d0 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 zone default found 2024-04-27T03:19:15.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 4 Realm: () 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 zonegroup default 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b5990 con 0x7f16b4037100 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b7760 con 0x55a86b5b2660 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b7b00 con 0x55a86b5b2660 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 started sync module instance, tier type = 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.363+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b6c00 con 0x55a86b5ad500 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94181844021536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b6c00 con 0x55a86b5ad500 2024-04-27T03:19:15.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b85f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94181844025600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b8520 con 0x55a86b5a91d0 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b94e0 con 0x55a86b5ad500 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94181844025600] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.367+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94181844021536] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94181844019024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b9030 con 0x55a86b5ad500 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b9d80 con 0x7f16b4037100 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94181844031696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5b9030 con 0x7f16b4037100 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5ba820 con 0x55a86b5ad500 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94181844031696] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94181844019024] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94181844027920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5ba7e0 con 0x55a86b5ad500 2024-04-27T03:19:15.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.371+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bac40 con 0x55a86b5b2660 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94181844032752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bb0f0 con 0x55a86b5b2660 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bbab0 con 0x55a86b5b2660 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94181844027920] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94181844032752] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94181844039168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bb0f0 con 0x55a86b5b2660 2024-04-27T03:19:15.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bc350 con 0x7f16b4037100 2024-04-27T03:19:15.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94181844039168] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v29'225 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94181844036672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bb060 con 0x7f16b4037100 2024-04-27T03:19:15.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 20 add_watcher() i=0 2024-04-27T03:19:15.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 20 add_watcher() i=1 2024-04-27T03:19:15.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 20 add_watcher() i=2 2024-04-27T03:19:15.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 20 add_watcher() i=3 2024-04-27T03:19:15.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 20 add_watcher() i=4 2024-04-27T03:19:15.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 20 add_watcher() i=5 2024-04-27T03:19:15.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.375+0000 7f16bd456980 20 add_watcher() i=6 2024-04-27T03:19:15.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94181844036672] v29'226 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16bd456980 20 add_watcher() i=7 2024-04-27T03:19:15.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16bd456980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:15.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bf4c0 con 0x55a86b5a91d0 2024-04-27T03:19:15.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94181844041376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bc9e0 con 0x55a86b5a91d0 2024-04-27T03:19:15.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94181844041376] v29'229 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f15717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:15.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16bd456980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:15.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16bd456980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:15.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.379+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bf2a0 con 0x55a86b5ad500 2024-04-27T03:19:15.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.383+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v29'57 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.383+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5bf330 con 0x7f16b4037100 2024-04-27T03:19:15.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.383+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 gc.1 [create,call,call,call] v29'35 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.383+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c87e0 con 0x55a86b5b2660 2024-04-27T03:19:15.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.383+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v29'34 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.383+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c8c90 con 0x55a86b5a91d0 2024-04-27T03:19:15.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.387+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(65 gc.3 [create,call,call,call] v29'230 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.387+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c7c60 con 0x55a86b5ad500 2024-04-27T03:19:15.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.387+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v29'79 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.387+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c7c60 con 0x55a86b5b2660 2024-04-27T03:19:15.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.391+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v29'59 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.391+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c7c60 con 0x55a86b5ad500 2024-04-27T03:19:15.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.391+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v29'37 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.391+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c7e30 con 0x55a86b5a91d0 2024-04-27T03:19:15.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.395+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 gc.7 [create,call,call,call] v29'54 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.395+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c97f0 con 0x55a86b5ad500 2024-04-27T03:19:15.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.395+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v29'27 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.395+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c97f0 con 0x55a86b5ad500 2024-04-27T03:19:15.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.399+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v29'33 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.399+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c97f0 con 0x55a86b5ad500 2024-04-27T03:19:15.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.399+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v29'80 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.399+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c97f0 con 0x55a86b5b2660 2024-04-27T03:19:15.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.403+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v29'31 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.403+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c97f0 con 0x55a86b5ad500 2024-04-27T03:19:15.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.403+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v29'33 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.403+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c97f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.407+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(75 gc.13 [create,call,call,call] v29'65 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.407+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c97f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.407+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(76 gc.14 [create,call,call,call] v29'231 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.407+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5ad500 2024-04-27T03:19:15.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.411+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v29'71 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.411+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5ad500 2024-04-27T03:19:15.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.411+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v29'81 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.411+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5b2660 2024-04-27T03:19:15.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.415+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v29'40 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.415+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5ad500 2024-04-27T03:19:15.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.415+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v29'35 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.415+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5a91d0 2024-04-27T03:19:15.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.419+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(81 gc.19 [create,call,call,call] v29'232 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.419+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5ad500 2024-04-27T03:19:15.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.423+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v29'72 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.423+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5ad500 2024-04-27T03:19:15.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.423+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v29'43 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.423+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5b2660 2024-04-27T03:19:15.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.423+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v29'35 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.423+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5a91d0 2024-04-27T03:19:15.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.427+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 gc.23 [create,call,call,call] v29'45 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.427+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9d80 con 0x55a86b5b2660 2024-04-27T03:19:15.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.427+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v29'29 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.427+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9d80 con 0x55a86b5b2660 2024-04-27T03:19:15.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.431+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v29'60 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.431+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5b2660 2024-04-27T03:19:15.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.431+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v29'36 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.431+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5ad500 2024-04-27T03:19:15.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.435+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(89 gc.27 [create,call,call,call] v29'39 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.435+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5a91d0 2024-04-27T03:19:15.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.435+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v29'55 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.435+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5ad500 2024-04-27T03:19:15.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.439+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(91 gc.29 [create,call,call,call] v29'73 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.439+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x55a86b5a91d0 2024-04-27T03:19:15.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.439+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v29'66 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.439+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c9930 con 0x7f16b4037100 2024-04-27T03:19:15.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(93 gc.31 [create,call,call,call] v29'33 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f14ebfff640 20 reqs_thread_entry: start 2024-04-27T03:19:15.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:15.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 20 Filter name: none 2024-04-27T03:19:15.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f14e9ffb640 20 reqs_thread_entry: start 2024-04-27T03:19:15.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 cache get: name=default.rgw.meta+users.uid+fud : miss 2024-04-27T03:19:15.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:94 8.e 8:75f48285:users.uid::fud:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5c73f0 con 0x7f16b4037100 2024-04-27T03:19:15.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(94 fud [call out=48b,read 0~380 out=380b,getxattrs out=75b] v0'0 uv5 ondisk = 0) ==== 231+0+503 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 20 rados_obj.operate() r=0 bl.length=380 2024-04-27T03:19:15.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 cache put: name=default.rgw.meta+users.uid+fud info.flags=0x13 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 adding default.rgw.meta+users.uid+fud to cache LRU end 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:95 8.1 8:810902b1:users.uid::fud.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5e76f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(95 fud.buckets [call out=186b] v0'0 uv5 ondisk = 0) ==== 155+0+186 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 : miss 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:96 8.b 8:d4f365da:root::.bucket.meta.myfoo7%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.10:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5e75a0 con 0x55a86b5b2660 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(96 .bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 [call out=48b,stat out=16b,read 0~382 out=382b,getxattrs out=218b] v0'0 uv1 ondisk = 0) ==== 334+0+664 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 20 rados_obj.operate() r=0 bl.length=382 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 info.flags=0x17 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 adding default.rgw.meta+root+.bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 to cache LRU end 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 2024-04-27T03:19:15.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:97 4.16 4:6abbee4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664ab0 con 0x55a86b5b2660 2024-04-27T03:19:15.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(97 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0 [call out=85b] v0'0 uv5 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:98 4.10 4:0b667e8b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664af0 con 0x55a86b5b2660 2024-04-27T03:19:15.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(98 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1 [call out=85b] v0'0 uv2 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.443+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 4.3e 4:7ee8f60e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664af0 con 0x55a86b5ad500 2024-04-27T03:19:15.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(99 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2 [call out=85b] v0'0 uv10 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:100 4.13 4:cb3019b8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664af0 con 0x7f16b4037100 2024-04-27T03:19:15.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(100 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3 [call out=85b] v0'0 uv4 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 4.2b 4:d500f2d0:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664af0 con 0x55a86b5ad500 2024-04-27T03:19:15.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(101 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:102 4.8 4:103aa1ab:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664af0 con 0x7f16b4037100 2024-04-27T03:19:15.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(102 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5 [call out=378b] v0'0 uv11 ondisk = 0) ==== 195+0+378 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bd456980 10 cls_bucket_list_unordered: got fourteen 2024-04-27T03:19:15.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 4.1e 4:781b0f7d:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664ab0 con 0x55a86b5a91d0 2024-04-27T03:19:15.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(103 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6 [call out=85b] v0'0 uv3 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:104 4.3a 4:5c08cc25:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664b10 con 0x55a86b5b2660 2024-04-27T03:19:15.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(104 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7 [call out=85b] v0'0 uv6 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.447+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:105 4.2b 4:d57152f7:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664b10 con 0x55a86b5ad500 2024-04-27T03:19:15.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(105 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8 [call out=85b] v0'0 uv2 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b40083f0 con 0x55a86b5ad500 2024-04-27T03:19:15.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 4.28 4:17785049:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664b10 con 0x55a86b5ad500 2024-04-27T03:19:15.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(106 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b406d060 con 0x55a86b5ad500 2024-04-27T03:19:15.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:107 4.4 4:201a8e98:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664b10 con 0x55a86b5a91d0 2024-04-27T03:19:15.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(107 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10 [call out=85b] v0'0 uv7 ondisk = 0) ==== 196+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 20 get_obj_state: octx=0x55a86b596a90 obj=myfoo7:fourteen state=0x55a86b662b58 s->prefetch_data=0 2024-04-27T03:19:15.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:108 3.23s0 3:c6460fef:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.10_fourteen:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b663980 con 0x7f16b4037100 2024-04-27T03:19:15.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(108 27a51273-3ccc-49e2-8682-1122abc3d898.4220.10_fourteen [getxattrs out=858b,stat out=16b] v0'0 uv3 ondisk = 0) ==== 239+0+874 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 10 manifest: total_size = 8 2024-04-27T03:19:15.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.12965510782039256061 2024-04-27T03:19:15.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 20 get_obj_state: octx=0x55a86b596a90 obj=myfoo7:fourteen state=0x55a86b662b58 s->prefetch_data=0 2024-04-27T03:19:15.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 20 prepare_atomic_modification: state is not atomic. state=0x55a86b662b58 2024-04-27T03:19:15.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5 2024-04-27T03:19:15.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.451+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:109 4.8 4:103aa1ab:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=188b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664210 con 0x7f16b4037100 2024-04-27T03:19:15.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.455+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(109 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5 [stat,call,call] v29'12 uv12 ondisk = 0) ==== 279+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.455+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:110 3.23s0 3:c6460fef:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.10_fourteen:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664210 con 0x7f16b4037100 2024-04-27T03:19:15.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.455+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(110 27a51273-3ccc-49e2-8682-1122abc3d898.4220.10_fourteen [call] v29'4 uv4 ondisk = 0) ==== 197+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.455+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:111 4.8 4:103aa1ab:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=265b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664180 con 0x7f16b4037100 2024-04-27T03:19:15.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 20 list_objects_ordered: starting attempt 1 2024-04-27T03:19:15.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2024-04-27T03:19:15.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:112 4.16 4:6abbee4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b664180 con 0x55a86b5b2660 2024-04-27T03:19:15.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:113 4.10 4:0b667e8b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f0030 con 0x55a86b5b2660 2024-04-27T03:19:15.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:114 4.3e 4:7ee8f60e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f2990 con 0x55a86b5ad500 2024-04-27T03:19:15.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:115 4.13 4:cb3019b8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f5440 con 0x7f16b4037100 2024-04-27T03:19:15.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:116 4.2b 4:d500f2d0:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f8140 con 0x55a86b5ad500 2024-04-27T03:19:15.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:117 4.8 4:103aa1ab:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fab30 con 0x7f16b4037100 2024-04-27T03:19:15.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:118 4.1e 4:781b0f7d:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f8180 con 0x55a86b5a91d0 2024-04-27T03:19:15.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(112 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0 [call out=85b] v0'0 uv5 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(113 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1 [call out=85b] v0'0 uv2 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(115 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3 [call out=85b] v0'0 uv4 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(116 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b406d060 con 0x55a86b5ad500 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(118 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6 [call out=85b] v0'0 uv3 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:119 4.3a 4:5c08cc25:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f8180 con 0x55a86b5b2660 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:120 4.2b 4:d57152f7:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f8180 con 0x55a86b5ad500 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(114 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2 [call out=85b] v0'0 uv10 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b406d060 con 0x55a86b5ad500 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:121 4.28 4:17785049:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fc380 con 0x55a86b5ad500 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:122 4.4 4:201a8e98:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fc900 con 0x55a86b5a91d0 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(119 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7 [call out=85b] v0'0 uv6 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(122 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10 [call out=85b] v0'0 uv7 ondisk = 0) ==== 196+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(120 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8 [call out=85b] v0'0 uv2 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b406d060 con 0x55a86b5ad500 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(121 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b406d060 con 0x55a86b5ad500 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 22 ==== osd_op_reply(111 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5 [stat,call,call] v29'13 uv13 ondisk = 0) ==== 279+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f159dffb640 20 handle_completion(): completion ok for obj=fourteen 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 23 ==== osd_op_reply(117 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5 [call out=124b] v0'0 uv13 ondisk = 0) ==== 195+0+124 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-04-27T03:19:15.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo7[27a51273-3ccc-49e2-8682-1122abc3d898.4220.10])"; uploads.size()=0; is_truncated=0 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:123 4.16 4:6abbee4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f29f0 con 0x55a86b5b2660 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:124 4.10 4:0b667e8b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fcd50 con 0x55a86b5b2660 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:125 4.3e 4:7ee8f60e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fbdc0 con 0x55a86b5ad500 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:126 4.13 4:cb3019b8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa560 con 0x7f16b4037100 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:127 4.2b 4:d500f2d0:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5ec6a0 con 0x55a86b5ad500 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:128 4.8 4:103aa1ab:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f29f0 con 0x7f16b4037100 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:129 4.1e 4:781b0f7d:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5e8c70 con 0x55a86b5a91d0 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:130 4.3a 4:5c08cc25:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa560 con 0x55a86b5b2660 2024-04-27T03:19:15.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:131 4.2b 4:d57152f7:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f29f0 con 0x55a86b5ad500 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:132 4.28 4:17785049:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fbdc0 con 0x55a86b5ad500 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:133 4.4 4:201a8e98:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5ec6e0 con 0x55a86b5a91d0 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 24 ==== osd_op_reply(128 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5 [call out=124b] v0'0 uv13 ondisk = 0) ==== 195+0+124 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 25 ==== osd_op_reply(126 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3 [call out=85b] v0'0 uv4 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(123 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0 [call out=85b] v0'0 uv5 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(129 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6 [call out=85b] v0'0 uv3 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(130 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7 [call out=85b] v0'0 uv6 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(124 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1 [call out=85b] v0'0 uv2 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(127 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(125 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2 [call out=85b] v0'0 uv10 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.459+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(132 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9 [call out=85b] v0'0 uv1 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(133 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10 [call out=85b] v0'0 uv7 ondisk = 0) ==== 196+0+85 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(131 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8 [call out=85b] v0'0 uv2 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:134 8.1 8:810902b1:users.uid::fud.buckets:head [call user.set_buckets_info in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(134 fud.buckets [call] v29'6 uv6 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 10 cache get: name=default.rgw.meta+root+myfoo7 : miss 2024-04-27T03:19:15.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:135 8.9 8:938c738d:root::myfoo7:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x7f16b4037100 2024-04-27T03:19:15.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 26 ==== osd_op_reply(135 myfoo7 [call out=48b,read 0~168 out=168b] v0'0 uv44 ondisk = 0) ==== 192+0+216 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 20 rados_obj.operate() r=0 bl.length=168 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 10 cache put: name=default.rgw.meta+root+myfoo7 info.flags=0x11 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 10 adding default.rgw.meta+root+myfoo7 to cache LRU end 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:136 8.9 8:938c738d:root::myfoo7:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x7f16b4037100 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 27 ==== osd_op_reply(136 myfoo7 [call,call,delete] v29'45 uv45 ondisk = 0) ==== 234+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 10 removing default.rgw.meta+root+myfoo7 from cache 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:myfoo7, ofs0, ns] 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:137 7.e 7:7759931f:::notify.3:head [notify cookie 94181844305808 in=162b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x7f16b4037100 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 28 ==== watch-notify(notify (1) cookie 94181844031696 notify 124554051600 ret 0) ==== 192+0+0 (crc 0 0 0) 0x55a86b5c9f50 con 0x7f16b4037100 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f159dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051600 cookie 94181844031696 notifier 4751 bl.length()=150 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f159dffb640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:138 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f14f00032d0 con 0x7f16b4037100 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.463+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 29 ==== osd_op_reply(137 notify.3 [notify cookie 94181844305808 out=8b] v0'0 uv225 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 30 ==== osd_op_reply(138 notify.3 [notify-ack] v0'0 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 31 ==== watch-notify(notify_complete (2) cookie 94181844305808 notify 124554051600 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a86b5bf010 con 0x7f16b4037100 2024-04-27T03:19:15.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:139 8.b 8:d4f365da:root::.bucket.meta.myfoo7%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.10:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5b2660 2024-04-27T03:19:15.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(139 .bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 [call,call,delete] v29'2 uv2 ondisk = 0) ==== 292+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16bd456980 10 removing default.rgw.meta+root+.bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 from cache 2024-04-27T03:19:15.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16bd456980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo7:27a51273-3ccc-49e2-8682-1122abc3d898.4220.10, ofs0, ns] 2024-04-27T03:19:15.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:140 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94181844219152 in=220b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5ad500 2024-04-27T03:19:15.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== watch-notify(notify (1) cookie 94181844021536 notify 124554051629 ret 0) ==== 250+0+0 (crc 0 0 0) 0x55a86b5bf500 con 0x55a86b5ad500 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f159e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051629 cookie 94181844021536 notifier 4751 bl.length()=208 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f159e7fc640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:141 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f1580003d70 con 0x55a86b5ad500 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 45 ==== osd_op_reply(140 notify.0 [notify cookie 94181844219152 out=8b] v0'0 uv225 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.467+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 46 ==== osd_op_reply(141 notify.0 [notify-ack] v0'0 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 47 ==== watch-notify(notify_complete (2) cookie 94181844219152 notify 124554051629 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a86b59e3a0 con 0x55a86b5ad500 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:142 4.16 4:6abbee4e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5b2660 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:143 4.10 4:0b667e8b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f79e0 con 0x55a86b5b2660 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:144 4.3e 4:7ee8f60e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5e5d30 con 0x55a86b5ad500 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:145 4.13 4:cb3019b8:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fab30 con 0x7f16b4037100 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:146 4.2b 4:d500f2d0:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fb3c0 con 0x55a86b5ad500 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:147 4.8 4:103aa1ab:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x7f16b4037100 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:148 4.1e 4:781b0f7d:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5e5d30 con 0x55a86b5a91d0 2024-04-27T03:19:15.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:149 4.3a 4:5c08cc25:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fb3c0 con 0x55a86b5b2660 2024-04-27T03:19:15.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:150 4.2b 4:d57152f7:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fab30 con 0x55a86b5ad500 2024-04-27T03:19:15.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:151 4.28 4:17785049:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5f79e0 con 0x55a86b5ad500 2024-04-27T03:19:15.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:152 4.4 4:201a8e98:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== osd_op_reply(143 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.1 [delete] v29'4 uv4 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(142 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.0 [delete] v29'6 uv6 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 48 ==== osd_op_reply(144 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.2 [delete] v29'11 uv11 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 49 ==== osd_op_reply(146 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.4 [delete] v29'3 uv3 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 50 ==== osd_op_reply(151 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.9 [delete] v29'2 uv2 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(152 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.10 [delete] v29'8 uv8 ondisk = 0) ==== 196+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(148 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.6 [delete] v29'4 uv4 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 51 ==== osd_op_reply(150 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.8 [delete] v29'4 uv4 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 32 ==== osd_op_reply(145 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.3 [delete] v29'5 uv5 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.471+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(149 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.7 [delete] v29'7 uv7 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 33 ==== osd_op_reply(147 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.10.5 [delete] v29'14 uv14 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 10 cache get: name=default.rgw.log++pubsub..bucket.myfoo7/27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 : miss 2024-04-27T03:19:15.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 20 rados->read ofs=0 len=0 2024-04-27T03:19:15.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:153 6.6 6:6609d728:::pubsub..bucket.myfoo7%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.10:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fb3c0 con 0x55a86b5b2660 2024-04-27T03:19:15.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== osd_op_reply(153 pubsub..bucket.myfoo7/27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 210+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:15.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 10 cache put: name=default.rgw.log++pubsub..bucket.myfoo7/27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 info.flags=0x0 2024-04-27T03:19:15.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 10 adding default.rgw.log++pubsub..bucket.myfoo7/27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 to cache LRU end 2024-04-27T03:19:15.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:154 6.6 6:6609d728:::pubsub..bucket.myfoo7%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5e5d30 con 0x55a86b5b2660 2024-04-27T03:19:15.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(154 pubsub..bucket.myfoo7/27a51273-3ccc-49e2-8682-1122abc3d898.4220.10 [delete] v29'13 uv12 ondisk = -2 ((2) No such file or directory)) ==== 210+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.475+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:155 8.1 8:810902b1:users.uid::fud.buckets:head [call user.remove_bucket in=147b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(155 fud.buckets [call] v29'7 uv7 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bd456980 10 removing key index: p5YnriCv1nAtykxBrupQ 2024-04-27T03:19:15.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:156 8.13 8:cc695d75:users.keys::p5YnriCv1nAtykxBrupQ:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5b2660 2024-04-27T03:19:15.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== osd_op_reply(156 p5YnriCv1nAtykxBrupQ [delete] v29'12 uv12 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bd456980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:p5YnriCv1nAtykxBrupQ, ofs0, ns] 2024-04-27T03:19:15.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:157 7.15 7:a93a5511:::notify.2:head [notify cookie 94181844305808 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5ad500 2024-04-27T03:19:15.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 52 ==== watch-notify(notify (1) cookie 94181844019024 notify 124554051630 ret 0) ==== 212+0+0 (crc 0 0 0) 0x55a86b59e3a0 con 0x55a86b5ad500 2024-04-27T03:19:15.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f159e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051630 cookie 94181844019024 notifier 4751 bl.length()=170 2024-04-27T03:19:15.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f159e7fc640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:158 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f1580003d70 con 0x55a86b5ad500 2024-04-27T03:19:15.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 53 ==== osd_op_reply(157 notify.2 [notify cookie 94181844305808 out=8b] v0'0 uv225 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 54 ==== watch-notify(notify_complete (2) cookie 94181844305808 notify 124554051630 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a86b59e110 con 0x55a86b5ad500 2024-04-27T03:19:15.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bd456980 10 removing email index: 2024-04-27T03:19:15.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bd456980 10 removing user buckets index 2024-04-27T03:19:15.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 55 ==== osd_op_reply(158 notify.2 [notify-ack] v0'0 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.479+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:159 8.1 8:810902b1:users.uid::fud.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(159 fud.buckets [delete] v29'8 uv8 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16bd456980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:fud.buckets, ofs0, ns] 2024-04-27T03:19:15.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:160 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94181844305808 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5b2660 2024-04-27T03:19:15.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== watch-notify(notify (1) cookie 94181844039168 notify 124554051616 ret 0) ==== 202+0+0 (crc 0 0 0) 0x55a86b5e6a90 con 0x55a86b5b2660 2024-04-27T03:19:15.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f159dffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051616 cookie 94181844039168 notifier 4751 bl.length()=160 2024-04-27T03:19:15.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f159dffb640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:161 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f14f00032d0 con 0x55a86b5b2660 2024-04-27T03:19:15.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== osd_op_reply(160 notify.6 [notify cookie 94181844305808 out=8b] v0'0 uv225 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== osd_op_reply(161 notify.6 [notify-ack] v0'0 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== watch-notify(notify_complete (2) cookie 94181844305808 notify 124554051616 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a86b5bf330 con 0x55a86b5b2660 2024-04-27T03:19:15.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16bd456980 10 removing user index: fud 2024-04-27T03:19:15.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:162 8.e 8:75f48285:users.uid::fud:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x7f16b4037100 2024-04-27T03:19:15.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 34 ==== osd_op_reply(162 fud [call,call,delete] v29'7 uv7 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16bd456980 10 removing default.rgw.meta+users.uid+fud from cache 2024-04-27T03:19:15.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16bd456980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:fud, ofs0, ns] 2024-04-27T03:19:15.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.483+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:163 7.3 7:c609908c:::notify.5:head [notify cookie 94181844305808 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5b2660 2024-04-27T03:19:15.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== watch-notify(notify (1) cookie 94181844032752 notify 124554051617 ret 0) ==== 194+0+0 (crc 0 0 0) 0x55a86b5bf330 con 0x55a86b5b2660 2024-04-27T03:19:15.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f159e7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 124554051617 cookie 94181844032752 notifier 4751 bl.length()=152 2024-04-27T03:19:15.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(163 notify.5 [notify cookie 94181844305808 out=8b] v0'0 uv225 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f159e7fc640 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:164 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29) -- 0x7f1580003d70 con 0x55a86b5b2660 2024-04-27T03:19:15.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== osd_op_reply(164 notify.5 [notify-ack] v0'0 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 46 ==== watch-notify(notify_complete (2) cookie 94181844305808 notify 124554051617 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a86b571a70 con 0x55a86b5b2660 2024-04-27T03:19:15.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f15717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:15.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.487+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:165 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94181844041376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.491+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(165 data_loggenerations_metadata [watch unwatch cookie 94181844041376] v29'233 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.491+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:166 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94181844021536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5ad500 2024-04-27T03:19:15.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.491+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 56 ==== osd_op_reply(166 notify.0 [watch unwatch cookie 94181844021536] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.491+0000 7f16bd456980 20 remove_watcher() i=0 2024-04-27T03:19:15.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.491+0000 7f16bd456980 2 removed watcher, disabling cache 2024-04-27T03:19:15.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.491+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:167 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94181844025600] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5a91d0 2024-04-27T03:19:15.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.495+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(167 notify.1 [watch unwatch cookie 94181844025600] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x55a86b5a91d0 2024-04-27T03:19:15.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.495+0000 7f16bd456980 20 remove_watcher() i=1 2024-04-27T03:19:15.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.495+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:168 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94181844019024] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5ad500 2024-04-27T03:19:15.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.495+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 57 ==== osd_op_reply(168 notify.2 [watch unwatch cookie 94181844019024] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.495+0000 7f16bd456980 20 remove_watcher() i=2 2024-04-27T03:19:15.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.495+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:169 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94181844031696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x7f16b4037100 2024-04-27T03:19:15.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.499+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 35 ==== osd_op_reply(169 notify.3 [watch unwatch cookie 94181844031696] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.499+0000 7f16bd456980 20 remove_watcher() i=3 2024-04-27T03:19:15.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.499+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:170 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94181844027920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5ad500 2024-04-27T03:19:15.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.499+0000 7f16bb772640 1 -- 172.21.15.110:0/2276899801 <== osd.0 v2:172.21.15.77:6800/3496324319 58 ==== osd_op_reply(170 notify.4 [watch unwatch cookie 94181844027920] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b4076040 con 0x55a86b5ad500 2024-04-27T03:19:15.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.499+0000 7f16bd456980 20 remove_watcher() i=4 2024-04-27T03:19:15.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.499+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:171 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94181844032752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5b2660 2024-04-27T03:19:15.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 47 ==== osd_op_reply(171 notify.5 [watch unwatch cookie 94181844032752] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 20 remove_watcher() i=5 2024-04-27T03:19:15.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:172 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94181844039168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x55a86b5b2660 2024-04-27T03:19:15.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16baf71640 1 -- 172.21.15.110:0/2276899801 <== osd.3 v2:172.21.15.110:6816/3916647200 48 ==== osd_op_reply(172 notify.6 [watch unwatch cookie 94181844039168] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16ac03d030 con 0x55a86b5b2660 2024-04-27T03:19:15.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 20 remove_watcher() i=6 2024-04-27T03:19:15.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:173 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94181844036672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) -- 0x55a86b5fa5f0 con 0x7f16b4037100 2024-04-27T03:19:15.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 <== osd.2 v2:172.21.15.110:6808/3575017494 36 ==== osd_op_reply(173 notify.7 [watch unwatch cookie 94181844036672] v29'227 uv225 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f16b00b3040 con 0x7f16b4037100 2024-04-27T03:19:15.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 20 remove_watcher() i=7 2024-04-27T03:19:15.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b5ad500 msgr2=0x55a86b5af930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b5ad500 0x55a86b5af930 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b5a91d0 msgr2=0x55a86b5ab690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b5a91d0 0x55a86b5ab690 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16b4037100 msgr2=0x7f16b4013a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f16b4037100 0x7f16b4013a00 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a86b5b2660 msgr2=0x55a86b5b4a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.503+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a86b5b2660 0x55a86b5b4a50 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1584034650 msgr2=0x7f1584036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1584034650 0x7f1584036b10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f16ac010910 tx=0x7f16ac037000 comp rx=0 tx=0).stop 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16ba770640 1 -- 172.21.15.110:0/2276899801 reap_dead start 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 msgr2=0x55a86b59dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b59dbd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f16b40209b0 tx=0x7f16b4049850 comp rx=0 tx=0).stop 2024-04-27T03:19:15.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 shutdown_connections 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/2276899801 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b5966b0 0x55a86b59dbd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 >> 172.21.15.110:0/2276899801 conn(0x55a86b5929f0 msgr2=0x55a86b590050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 shutdown_connections 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2276899801 wait complete. 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15940346a0 msgr2=0x7f1594036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15940346a0 0x7f1594036b60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55a86b5760d0 tx=0x7f16ac01e470 comp rx=0 tx=0).stop 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b589500 msgr2=0x55a86b589920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b589500 0x55a86b589920 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f16b400aab0 tx=0x7f16b400af70 comp rx=0 tx=0).stop 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 shutdown_connections 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15940346a0 0x7f1594036b60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/2544288380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b589500 0x55a86b589920 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 >> 172.21.15.110:0/2544288380 conn(0x55a86b57e310 msgr2=0x55a86b57ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 shutdown_connections 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/2544288380 wait complete. 2024-04-27T03:19:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b576910 msgr2=0x55a86b578d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b576910 0x55a86b578d00 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b4809d0 msgr2=0x55a86b575550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b4809d0 0x55a86b575550 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15a0034650 msgr2=0x7f15a0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15a0034650 0x7f15a0036b10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f16b00ad000 tx=0x7f16b00a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 msgr2=0x55a86b56e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b56e620 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x55a86b568470 tx=0x7f16ac000f50 comp rx=0 tx=0).stop 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 shutdown_connections 2024-04-27T03:19:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a86b576910 0x55a86b578d00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a86b4809d0 0x55a86b575550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f15a0034650 0x7f15a0036b10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 --2- 172.21.15.110:0/258453509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a86b47c4c0 0x55a86b56e620 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:15.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 >> 172.21.15.110:0/258453509 conn(0x55a86b3efba0 msgr2=0x55a86b561c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:15.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 shutdown_connections 2024-04-27T03:19:15.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:15.507+0000 7f16bd456980 1 -- 172.21.15.110:0/258453509 wait complete. 2024-04-27T03:19:55.529 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'trim', '--uid', 'foo'] 2024-04-27T03:19:55.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'trim', '--uid', 'foo'] 2024-04-27T03:19:55.530 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage trim --uid foo 2024-04-27T03:19:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:55.570 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.558+0000 7f1f88bd8980 1 Processor -- start 2024-04-27T03:19:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.558+0000 7f1f88bd8980 1 -- start start 2024-04-27T03:19:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f88bd8980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56159d7119c0 0x56159d713db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f88bd8980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x56159d5ca0d0 con 0x56159d70d4b0 2024-04-27T03:19:55.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f88bd8980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x56159d5bf430 con 0x56159d7119c0 2024-04-27T03:19:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f85ef2640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x56159d7119c0 0x56159d713db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f866f3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x56159d70d4b0 0x56159d70d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44456/0 (socket says 172.21.15.110:44456) 2024-04-27T03:19:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f866f3640 1 -- 172.21.15.110:0/1196633829 learned_addr learned my addr 172.21.15.110:0/1196633829 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f85ef2640 1 -- 172.21.15.110:0/1196633829 >> v1:172.21.15.77:6789/0 conn(0x56159d70d4b0 legacy=0x56159d70d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f85ef2640 1 -- 172.21.15.110:0/1196633829 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159d5b8510 con 0x56159d7119c0 2024-04-27T03:19:55.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f85ef2640 1 --2- 172.21.15.110:0/1196633829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d7119c0 0x56159d713db0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00a480 tx=0x7f1f7c01b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec776849750d045 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f856f1640 1 -- 172.21.15.110:0/1196633829 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f7c0b0070 con 0x56159d7119c0 2024-04-27T03:19:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f856f1640 1 -- 172.21.15.110:0/1196633829 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0ab540 con 0x56159d7119c0 2024-04-27T03:19:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.562+0000 7f1f856f1640 1 -- 172.21.15.110:0/1196633829 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f7c0ab840 con 0x56159d7119c0 2024-04-27T03:19:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.566+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1196633829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d7119c0 msgr2=0x56159d713db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.566+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/1196633829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d7119c0 0x56159d713db0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00a480 tx=0x7f1f7c01b280 comp rx=0 tx=0).stop 2024-04-27T03:19:55.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.566+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1196633829 shutdown_connections 2024-04-27T03:19:55.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.566+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/1196633829 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d7119c0 0x56159d713db0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.566+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1196633829 >> 172.21.15.110:0/1196633829 conn(0x56159d680b90 msgr2=0x56159d680fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.566+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1196633829 shutdown_connections 2024-04-27T03:19:55.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.566+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1196633829 wait complete. 2024-04-27T03:19:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 Processor -- start 2024-04-27T03:19:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 -- start start 2024-04-27T03:19:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7f8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159d60e2d0 con 0x56159d70d4b0 2024-04-27T03:19:55.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f866f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f866f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7f8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50934/0 (socket says 172.21.15.110:50934) 2024-04-27T03:19:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f866f3640 1 -- 172.21.15.110:0/285603919 learned_addr learned my addr 172.21.15.110:0/285603919 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f866f3640 1 -- 172.21.15.110:0/285603919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159d594ae0 con 0x56159d70d4b0 2024-04-27T03:19:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f866f3640 1 --2- 172.21.15.110:0/285603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7f8630 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1f800017b0 tx=0x7f1f80001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845c86e99bcacf4f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1e74608640 1 -- 172.21.15.110:0/285603919 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f80016070 con 0x56159d70d4b0 2024-04-27T03:19:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1e74608640 1 -- 172.21.15.110:0/285603919 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f80004530 con 0x56159d70d4b0 2024-04-27T03:19:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1e74608640 1 -- 172.21.15.110:0/285603919 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f80004870 con 0x56159d70d4b0 2024-04-27T03:19:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 -- 172.21.15.110:0/285603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 msgr2=0x56159d7f8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/285603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7f8630 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1f800017b0 tx=0x7f1f80001c70 comp rx=0 tx=0).stop 2024-04-27T03:19:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 -- 172.21.15.110:0/285603919 shutdown_connections 2024-04-27T03:19:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/285603919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7f8630 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 -- 172.21.15.110:0/285603919 >> 172.21.15.110:0/285603919 conn(0x56159d680b90 msgr2=0x56159d7f41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 -- 172.21.15.110:0/285603919 shutdown_connections 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.582+0000 7f1f88bd8980 1 -- 172.21.15.110:0/285603919 wait complete. 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 Processor -- start 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 -- start start 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159d716de0 con 0x56159d70d4b0 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f866f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f866f3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50946/0 (socket says 172.21.15.110:50946) 2024-04-27T03:19:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f866f3640 1 -- 172.21.15.110:0/935682837 learned_addr learned my addr 172.21.15.110:0/935682837 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f866f3640 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159d7ffbb0 con 0x56159d70d4b0 2024-04-27T03:19:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f866f3640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7ff670 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000f90 tx=0x7f1f80001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1e72e05640 1 -- 172.21.15.110:0/935682837 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f80020070 con 0x56159d70d4b0 2024-04-27T03:19:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1e72e05640 1 -- 172.21.15.110:0/935682837 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f80011400 con 0x56159d70d4b0 2024-04-27T03:19:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1e72e05640 1 -- 172.21.15.110:0/935682837 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f80016040 con 0x56159d70d4b0 2024-04-27T03:19:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56159d7ffe40 con 0x56159d70d4b0 2024-04-27T03:19:55.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56159d802ac0 con 0x56159d70d4b0 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1e72e05640 1 -- 172.21.15.110:0/935682837 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f1f80023070 con 0x56159d70d4b0 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1e72e05640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e6c0346f0 0x7f1e6c036bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1e72e05640 1 -- 172.21.15.110:0/935682837 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f1f800491f0 con 0x56159d70d4b0 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f85ef2640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e6c0346f0 0x7f1e6c036bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d7119c0 0x56159d806710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d806c50 con 0x56159d7119c0 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.586+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d7119c0 0x56159d806710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f85ef2640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e6c0346f0 0x7f1e6c036bb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0ad000 tx=0x7f1f7c0a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.599 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d7119c0 0x56159d806710 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f86ef4640 1 -- 172.21.15.110:0/935682837 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f1f780083f0 con 0x56159d7119c0 2024-04-27T03:19:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d807670 0x56159d809b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d80a280 con 0x56159d807670 2024-04-27T03:19:55.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f866f3640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d807670 0x56159d809b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f866f3640 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d807670 0x56159d809b10 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f866f3640 1 -- 172.21.15.110:0/935682837 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d807670 2024-04-27T03:19:55.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d80bb30 con 0x56159d7119c0 2024-04-27T03:19:55.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f86ef4640 1 -- 172.21.15.110:0/935682837 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f1f780083f0 con 0x56159d7119c0 2024-04-27T03:19:55.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d80ad60 con 0x56159d807670 2024-04-27T03:19:55.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f866f3640 1 -- 172.21.15.110:0/935682837 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d807670 2024-04-27T03:19:55.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.590+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d80b150 con 0x56159d7119c0 2024-04-27T03:19:55.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f86ef4640 1 -- 172.21.15.110:0/935682837 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f1f780083f0 con 0x56159d7119c0 2024-04-27T03:19:55.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f88bd8980 1 Processor -- start 2024-04-27T03:19:55.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f88bd8980 1 -- start start 2024-04-27T03:19:55.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f88bd8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d81a430 0x56159d81a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f88bd8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159d612310 con 0x56159d81a430 2024-04-27T03:19:55.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f86ef4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d81a430 0x56159d81a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f86ef4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d81a430 0x56159d81a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50958/0 (socket says 172.21.15.110:50958) 2024-04-27T03:19:55.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f86ef4640 1 -- 172.21.15.110:0/1601423969 learned_addr learned my addr 172.21.15.110:0/1601423969 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f86ef4640 1 -- 172.21.15.110:0/1601423969 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159d81ad90 con 0x56159d81a430 2024-04-27T03:19:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d81a430 0x56159d81a850 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800aab0 tx=0x7f1f7800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1e6bfff640 1 -- 172.21.15.110:0/1601423969 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f78007d30 con 0x56159d81a430 2024-04-27T03:19:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56159d81b830 con 0x56159d81a430 2024-04-27T03:19:55.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1e6bfff640 1 -- 172.21.15.110:0/1601423969 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7801d070 con 0x56159d81a430 2024-04-27T03:19:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1e6bfff640 1 -- 172.21.15.110:0/1601423969 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f78018440 con 0x56159d81a430 2024-04-27T03:19:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56159d81bd00 con 0x56159d81a430 2024-04-27T03:19:55.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1e6bfff640 1 -- 172.21.15.110:0/1601423969 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f1f78013070 con 0x56159d81a430 2024-04-27T03:19:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1e6bfff640 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e60034650 0x7f1e60036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.594+0000 7f1e6bfff640 1 -- 172.21.15.110:0/1601423969 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f1f7804b730 con 0x56159d81a430 2024-04-27T03:19:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f866f3640 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e60034650 0x7f1e60036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 Processor -- start 2024-04-27T03:19:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- start start 2024-04-27T03:19:55.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f866f3640 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e60034650 0x7f1e60036b10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x56159d80a880 tx=0x7f1f80049050 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d827a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159d7f8d80 con 0x56159d8275e0 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d827a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d827a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50960/0 (socket says 172.21.15.110:50960) 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3341233857 learned_addr learned my addr 172.21.15.110:0/3341233857 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3341233857 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159d827fd0 con 0x56159d8275e0 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/3341233857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d827a00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1f78049970 tx=0x7f1f7803a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e87f63cc650c261 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1e69ffb640 1 -- 172.21.15.110:0/3341233857 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f7803f020 con 0x56159d8275e0 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1e69ffb640 1 -- 172.21.15.110:0/3341233857 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f780177c0 con 0x56159d8275e0 2024-04-27T03:19:55.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1e69ffb640 1 -- 172.21.15.110:0/3341233857 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f78022460 con 0x56159d8275e0 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3341233857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 msgr2=0x56159d827a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3341233857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d827a00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f1f78049970 tx=0x7f1f7803a110 comp rx=0 tx=0).stop 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3341233857 shutdown_connections 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3341233857 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d827a00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3341233857 >> 172.21.15.110:0/3341233857 conn(0x56159d823970 msgr2=0x56159d8219d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3341233857 shutdown_connections 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3341233857 wait complete. 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 Processor -- start 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- start start 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d82eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f88bd8980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x56159d828740 con 0x56159d8275e0 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d82eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d82eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50970/0 (socket says 172.21.15.110:50970) 2024-04-27T03:19:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.598+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 learned_addr learned my addr 172.21.15.110:0/3823857949 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56159d82f050 con 0x56159d8275e0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d82eb10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1f780168b0 tx=0x7f1f780498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1e57fff640 1 -- 172.21.15.110:0/3823857949 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f7802f020 con 0x56159d8275e0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1e57fff640 1 -- 172.21.15.110:0/3823857949 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f78020360 con 0x56159d8275e0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56159d82f2e0 con 0x56159d8275e0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1e57fff640 1 -- 172.21.15.110:0/3823857949 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1f7803f040 con 0x56159d8275e0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56159d831fc0 con 0x56159d8275e0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1e57fff640 1 -- 172.21.15.110:0/3823857949 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f1f78032020 con 0x56159d8275e0 2024-04-27T03:19:55.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1e57fff640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e50034600 0x7f1e50036ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1e57fff640 1 -- 172.21.15.110:0/3823857949 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f1f78035d80 con 0x56159d8275e0 2024-04-27T03:19:55.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f866f3640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e50034600 0x7f1e50036ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d83a0e0 0x56159d83c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d83ccf0 con 0x56159d83a0e0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f85ef2640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d83a0e0 0x56159d83c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f866f3640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e50034600 0x7f1e50036ac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1f80025040 tx=0x7f1f80031000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f85ef2640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d83a0e0 0x56159d83c5a0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 20 realm 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d83d120 con 0x56159d83a0e0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.602+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d83d9f0 con 0x56159d83a0e0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d83e410 0x56159d840840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d840fb0 con 0x56159d83e410 2024-04-27T03:19:55.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d83e410 0x56159d840840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d83e410 0x56159d840840 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:55.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d841800 con 0x56159d83a0e0 2024-04-27T03:19:55.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 20 searching for the correct realm 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56159d843570 0x56159d845960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d843570 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f866f3640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56159d843570 0x56159d845960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f866f3640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56159d843570 0x56159d845960 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f86ef4640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f1f78034020 0x7f1f7803b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x7f1f78034020 2024-04-27T03:19:55.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.606+0000 7f1f85ef2640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f1f78034020 0x7f1f7803b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f1f78034020 0x7f1f7803b9c0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d843570 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x7f1f78034020 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.610+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x7f1f78034020 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d843570 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d843570 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x7f1f78034020 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d843570 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.614+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x7f1f78034020 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83a0e0 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x7f1f78034020 2024-04-27T03:19:55.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d843570 2024-04-27T03:19:55.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x7f1f78034020 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d843570 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.618+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x56159d845fe0 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d842150 con 0x56159d83a0e0 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d8429f0 con 0x56159d83e410 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:55.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d841d90 con 0x56159d83a0e0 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 zone default found 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 4 Realm: () 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 zonegroup default 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d8468f0 con 0x7f1f78034020 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d8484e0 con 0x56159d843570 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d8489d0 con 0x56159d843570 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 started sync module instance, tier type = 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:55.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.622+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8473b0 con 0x56159d83e410 2024-04-27T03:19:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.626+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.626+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94650837006240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8492e0 con 0x56159d83e410 2024-04-27T03:19:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.626+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d849b30 con 0x56159d83a0e0 2024-04-27T03:19:55.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.626+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.626+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94650837013216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d849c90 con 0x56159d83a0e0 2024-04-27T03:19:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.626+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84a7b0 con 0x56159d83e410 2024-04-27T03:19:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.626+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94650837006240] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94650837013216] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94650837017216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84a770 con 0x56159d83e410 2024-04-27T03:19:55.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84b050 con 0x7f1f78034020 2024-04-27T03:19:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94650837019552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84a770 con 0x7f1f78034020 2024-04-27T03:19:55.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84bae0 con 0x56159d83e410 2024-04-27T03:19:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94650837017216] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.630+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94650837019552] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94650837020608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84bae0 con 0x56159d83e410 2024-04-27T03:19:55.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84bed0 con 0x56159d843570 2024-04-27T03:19:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94650837012016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84c720 con 0x56159d843570 2024-04-27T03:19:55.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84d0e0 con 0x56159d843570 2024-04-27T03:19:55.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94650837020608] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94650837022224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84d980 con 0x56159d843570 2024-04-27T03:19:55.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84e340 con 0x7f1f78034020 2024-04-27T03:19:55.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.634+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94650837012016] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94650837022224] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'228 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94650837030096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84d8f0 con 0x7f1f78034020 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=0 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=1 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=2 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=3 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=4 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=5 2024-04-27T03:19:55.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=6 2024-04-27T03:19:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94650837030096] v31'229 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 20 add_watcher() i=7 2024-04-27T03:19:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d851370 con 0x56159d83a0e0 2024-04-27T03:19:55.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.638+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94650837032592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d84e790 con 0x56159d83a0e0 2024-04-27T03:19:55.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94650837032592] v31'234 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1e3cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:55.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:55.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:55.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 5 note: GC not initialized 2024-04-27T03:19:55.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1db77fe640 20 reqs_thread_entry: start 2024-04-27T03:19:55.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:55.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 20 Filter name: none 2024-04-27T03:19:55.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1db57fa640 20 reqs_thread_entry: start 2024-04-27T03:19:55.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:19:55.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x56159d874a10 con 0x56159d83a0e0 2024-04-27T03:19:55.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:55.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:19:55.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:19:55.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:55.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_trim in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d843570 2024-04-27T03:19:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 usage.10 [call] v31'32 uv32 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.642+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_trim in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d843570 2024-04-27T03:19:55.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.646+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(64 usage.10 [call] v31'33 uv32 ondisk = -61 ((61) No data available)) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.646+0000 7f1e3cfe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:55.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.646+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94650837032592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d83a0e0 2024-04-27T03:19:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.646+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(65 data_loggenerations_metadata [watch unwatch cookie 94650837032592] v31'235 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.646+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94650837006240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d83e410 2024-04-27T03:19:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.650+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(66 notify.0 [watch unwatch cookie 94650837006240] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.650+0000 7f1f88bd8980 20 remove_watcher() i=0 2024-04-27T03:19:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.650+0000 7f1f88bd8980 2 removed watcher, disabling cache 2024-04-27T03:19:55.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.650+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94650837013216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d83a0e0 2024-04-27T03:19:55.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.650+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(67 notify.1 [watch unwatch cookie 94650837013216] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x56159d83a0e0 2024-04-27T03:19:55.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.650+0000 7f1f88bd8980 20 remove_watcher() i=1 2024-04-27T03:19:55.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.650+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94650837017216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d83e410 2024-04-27T03:19:55.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.654+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(68 notify.2 [watch unwatch cookie 94650837017216] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.654+0000 7f1f88bd8980 20 remove_watcher() i=2 2024-04-27T03:19:55.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.654+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:69 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94650837019552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x7f1f78034020 2024-04-27T03:19:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.654+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(69 notify.3 [watch unwatch cookie 94650837019552] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.654+0000 7f1f88bd8980 20 remove_watcher() i=3 2024-04-27T03:19:55.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.654+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94650837020608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d83e410 2024-04-27T03:19:55.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f86ef4640 1 -- 172.21.15.110:0/3823857949 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(70 notify.4 [watch unwatch cookie 94650837020608] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f78041050 con 0x56159d83e410 2024-04-27T03:19:55.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f88bd8980 20 remove_watcher() i=4 2024-04-27T03:19:55.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94650837012016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d843570 2024-04-27T03:19:55.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(71 notify.5 [watch unwatch cookie 94650837012016] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f88bd8980 20 remove_watcher() i=5 2024-04-27T03:19:55.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94650837022224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x56159d843570 2024-04-27T03:19:55.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f866f3640 1 -- 172.21.15.110:0/3823857949 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(72 notify.6 [watch unwatch cookie 94650837022224] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f8002c070 con 0x56159d843570 2024-04-27T03:19:55.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f88bd8980 20 remove_watcher() i=6 2024-04-27T03:19:55.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.658+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:73 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94650837030096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x56159d8744f0 con 0x7f1f78034020 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(73 notify.7 [watch unwatch cookie 94650837030096] v31'230 uv228 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f1f7c0b3040 con 0x7f1f78034020 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 20 remove_watcher() i=7 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d83e410 msgr2=0x56159d840840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d83e410 0x56159d840840 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d83a0e0 msgr2=0x56159d83c5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d83a0e0 0x56159d83c5a0 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f1f78034020 msgr2=0x7f1f7803b9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f1f78034020 0x7f1f7803b9c0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56159d843570 msgr2=0x56159d845960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56159d843570 0x56159d845960 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e50034600 msgr2=0x7f1e50036ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e50034600 0x7f1e50036ac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1f80025040 tx=0x7f1f80031000 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 msgr2=0x56159d82eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d82eb10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1f780168b0 tx=0x7f1f780498d0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 shutdown_connections 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f85ef2640 1 -- 172.21.15.110:0/3823857949 reap_dead start 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x56159d843570 0x56159d845960 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d83e410 0x56159d840840 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d83a0e0 0x56159d83c5a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e50034600 0x7f1e50036ac0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f1f78034020 0x7f1f7803b9c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/3823857949 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d8275e0 0x56159d82eb10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 >> 172.21.15.110:0/3823857949 conn(0x56159d823970 msgr2=0x56159d828e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 shutdown_connections 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/3823857949 wait complete. 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e60034650 msgr2=0x7f1e60036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e60034650 0x7f1e60036b10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x56159d80a880 tx=0x7f1f80049050 comp rx=0 tx=0).stop 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d81a430 msgr2=0x56159d81a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d81a430 0x56159d81a850 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800aab0 tx=0x7f1f7800af70 comp rx=0 tx=0).stop 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 shutdown_connections 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e60034650 0x7f1e60036b10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/1601423969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d81a430 0x56159d81a850 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 >> 172.21.15.110:0/1601423969 conn(0x56159d80f240 msgr2=0x56159d80fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 shutdown_connections 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/1601423969 wait complete. 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d807670 msgr2=0x56159d809b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d807670 0x56159d809b10 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d7119c0 msgr2=0x56159d806710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d7119c0 0x56159d806710 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e6c0346f0 msgr2=0x7f1e6c036bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e6c0346f0 0x7f1e6c036bb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0ad000 tx=0x7f1f7c0a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 msgr2=0x56159d7ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7ff670 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1f80000f90 tx=0x7f1f80001260 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 shutdown_connections 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x56159d807670 0x56159d809b10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x56159d7119c0 0x56159d806710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1e6c0346f0 0x7f1e6c036bb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 --2- 172.21.15.110:0/935682837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x56159d70d4b0 0x56159d7ff670 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 >> 172.21.15.110:0/935682837 conn(0x56159d680b90 msgr2=0x56159d7f2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 shutdown_connections 2024-04-27T03:19:55.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.662+0000 7f1f88bd8980 1 -- 172.21.15.110:0/935682837 wait complete. 2024-04-27T03:19:55.684 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--uid', 'foo'] 2024-04-27T03:19:55.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'usage', 'show', '--uid', 'foo'] 2024-04-27T03:19:55.684 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph usage show --uid foo 2024-04-27T03:19:55.759 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:55.759 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:55.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fecaf9980 1 Processor -- start 2024-04-27T03:19:55.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fecaf9980 1 -- start start 2024-04-27T03:19:55.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fecaf9980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55b3ff20b6b0 0x55b3ff20daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fecaf9980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55b3ff0c70d0 con 0x55b3ff207150 2024-04-27T03:19:55.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fecaf9980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55b3ff0bc430 con 0x55b3ff20b6b0 2024-04-27T03:19:55.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fe9e13640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55b3ff20b6b0 0x55b3ff20daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fea614640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55b3ff207150 0x55b3ff207530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44460/0 (socket says 172.21.15.110:44460) 2024-04-27T03:19:55.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fea614640 1 -- 172.21.15.110:0/1583540794 learned_addr learned my addr 172.21.15.110:0/1583540794 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:55.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fe9e13640 1 -- 172.21.15.110:0/1583540794 >> v1:172.21.15.77:6789/0 conn(0x55b3ff207150 legacy=0x55b3ff207530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.750+0000 7f8fe9e13640 1 -- 172.21.15.110:0/1583540794 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ff0b5510 con 0x55b3ff20b6b0 2024-04-27T03:19:55.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fe9e13640 1 --2- 172.21.15.110:0/1583540794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff20b6b0 0x55b3ff20daa0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000a480 tx=0x7f8fe001b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16667a70ea47904b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fe9612640 1 -- 172.21.15.110:0/1583540794 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe00b0070 con 0x55b3ff20b6b0 2024-04-27T03:19:55.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fe9612640 1 -- 172.21.15.110:0/1583540794 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe00ab540 con 0x55b3ff20b6b0 2024-04-27T03:19:55.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fe9612640 1 -- 172.21.15.110:0/1583540794 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe00ab840 con 0x55b3ff20b6b0 2024-04-27T03:19:55.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1583540794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff20b6b0 msgr2=0x55b3ff20daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/1583540794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff20b6b0 0x55b3ff20daa0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000a480 tx=0x7f8fe001b280 comp rx=0 tx=0).stop 2024-04-27T03:19:55.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1583540794 shutdown_connections 2024-04-27T03:19:55.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/1583540794 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff20b6b0 0x55b3ff20daa0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1583540794 >> 172.21.15.110:0/1583540794 conn(0x55b3ff202a20 msgr2=0x55b3ff204e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1583540794 shutdown_connections 2024-04-27T03:19:55.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.754+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1583540794 wait complete. 2024-04-27T03:19:55.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fecaf9980 1 Processor -- start 2024-04-27T03:19:55.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fecaf9980 1 -- start start 2024-04-27T03:19:55.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fecaf9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2f3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fecaf9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b3ff10b2d0 con 0x55b3ff207150 2024-04-27T03:19:55.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fea614640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2f3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fea614640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2f3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50986/0 (socket says 172.21.15.110:50986) 2024-04-27T03:19:55.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fea614640 1 -- 172.21.15.110:0/2798805826 learned_addr learned my addr 172.21.15.110:0/2798805826 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.770+0000 7f8fea614640 1 -- 172.21.15.110:0/2798805826 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ff091ae0 con 0x55b3ff207150 2024-04-27T03:19:55.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fea614640 1 --2- 172.21.15.110:0/2798805826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2f3450 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc001e20 tx=0x7f8fdc00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476572368cc63716 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8ed8608640 1 -- 172.21.15.110:0/2798805826 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fdc012020 con 0x55b3ff207150 2024-04-27T03:19:55.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8ed8608640 1 -- 172.21.15.110:0/2798805826 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fdc00e7c0 con 0x55b3ff207150 2024-04-27T03:19:55.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8ed8608640 1 -- 172.21.15.110:0/2798805826 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fdc00eb00 con 0x55b3ff207150 2024-04-27T03:19:55.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- 172.21.15.110:0/2798805826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 msgr2=0x55b3ff2f3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/2798805826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2f3450 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc001e20 tx=0x7f8fdc00b790 comp rx=0 tx=0).stop 2024-04-27T03:19:55.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- 172.21.15.110:0/2798805826 shutdown_connections 2024-04-27T03:19:55.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/2798805826 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2f3450 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- 172.21.15.110:0/2798805826 >> 172.21.15.110:0/2798805826 conn(0x55b3ff202a20 msgr2=0x55b3ff17c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- 172.21.15.110:0/2798805826 shutdown_connections 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- 172.21.15.110:0/2798805826 wait complete. 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 Processor -- start 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- start start 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2fa530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b3ff210ad0 con 0x55b3ff207150 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fea614640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2fa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fea614640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2fa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:50998/0 (socket says 172.21.15.110:50998) 2024-04-27T03:19:55.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fea614640 1 -- 172.21.15.110:0/3397289662 learned_addr learned my addr 172.21.15.110:0/3397289662 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fea614640 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ff2faa70 con 0x55b3ff207150 2024-04-27T03:19:55.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fea614640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2fa530 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00e790 tx=0x7f8fdc000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8ed6e05640 1 -- 172.21.15.110:0/3397289662 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fdc012030 con 0x55b3ff207150 2024-04-27T03:19:55.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8ed6e05640 1 -- 172.21.15.110:0/3397289662 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fdc01a680 con 0x55b3ff207150 2024-04-27T03:19:55.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3ff2fad00 con 0x55b3ff207150 2024-04-27T03:19:55.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8ed6e05640 1 -- 172.21.15.110:0/3397289662 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fdc01a960 con 0x55b3ff207150 2024-04-27T03:19:55.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.774+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ff2fd980 con 0x55b3ff207150 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8ed6e05640 1 -- 172.21.15.110:0/3397289662 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8fdc015020 con 0x55b3ff207150 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8ed6e05640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ed00346a0 0x7f8ed0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8ed6e05640 1 -- 172.21.15.110:0/3397289662 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f8fdc020030 con 0x55b3ff207150 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fe9e13640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ed00346a0 0x7f8ed0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff20b6b0 0x55b3ff301460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff301a30 con 0x55b3ff20b6b0 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8feae15640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff20b6b0 0x55b3ff301460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fe9e13640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ed00346a0 0x7f8ed0036b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00ad000 tx=0x7f8fe00a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8feae15640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff20b6b0 0x55b3ff301460 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8feae15640 1 -- 172.21.15.110:0/3397289662 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8fe400a4d0 con 0x55b3ff20b6b0 2024-04-27T03:19:55.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff302820 0x55b3ff304c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff305240 con 0x55b3ff302820 2024-04-27T03:19:55.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fea614640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff302820 0x55b3ff304c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.778+0000 7f8fea614640 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff302820 0x55b3ff304c10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fea614640 1 -- 172.21.15.110:0/3397289662 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff302820 2024-04-27T03:19:55.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff306af0 con 0x55b3ff20b6b0 2024-04-27T03:19:55.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8feae15640 1 -- 172.21.15.110:0/3397289662 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f8fe400a4d0 con 0x55b3ff20b6b0 2024-04-27T03:19:55.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff305d20 con 0x55b3ff302820 2024-04-27T03:19:55.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fea614640 1 -- 172.21.15.110:0/3397289662 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff302820 2024-04-27T03:19:55.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff306110 con 0x55b3ff20b6b0 2024-04-27T03:19:55.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8feae15640 1 -- 172.21.15.110:0/3397289662 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f8fe400a4d0 con 0x55b3ff20b6b0 2024-04-27T03:19:55.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fecaf9980 1 Processor -- start 2024-04-27T03:19:55.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fecaf9980 1 -- start start 2024-04-27T03:19:55.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fecaf9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff2ef710 0x55b3ff2edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8fecaf9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b3ff10f310 con 0x55b3ff2ef710 2024-04-27T03:19:55.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8feae15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff2ef710 0x55b3ff2edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8feae15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff2ef710 0x55b3ff2edbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51006/0 (socket says 172.21.15.110:51006) 2024-04-27T03:19:55.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8feae15640 1 -- 172.21.15.110:0/974787507 learned_addr learned my addr 172.21.15.110:0/974787507 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8feae15640 1 -- 172.21.15.110:0/974787507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ff2ee0f0 con 0x55b3ff2ef710 2024-04-27T03:19:55.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8feae15640 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff2ef710 0x55b3ff2edbb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400dfc0 tx=0x7f8fe40178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.782+0000 7f8ecffff640 1 -- 172.21.15.110:0/974787507 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe40157d0 con 0x55b3ff2ef710 2024-04-27T03:19:55.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3ff2ee380 con 0x55b3ff2ef710 2024-04-27T03:19:55.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecffff640 1 -- 172.21.15.110:0/974787507 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe4015970 con 0x55b3ff2ef710 2024-04-27T03:19:55.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecffff640 1 -- 172.21.15.110:0/974787507 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe4015c70 con 0x55b3ff2ef710 2024-04-27T03:19:55.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ff2ee850 con 0x55b3ff2ef710 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecffff640 1 -- 172.21.15.110:0/974787507 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8fe4029440 con 0x55b3ff2ef710 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecffff640 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ec4034650 0x7f8ec4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecffff640 1 -- 172.21.15.110:0/974787507 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f8fe400e040 con 0x55b3ff2ef710 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fea614640 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ec4034650 0x7f8ec4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fecaf9980 1 Processor -- start 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fecaf9980 1 -- start start 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fea614640 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ec4034650 0x7f8ec4036b10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x55b3ff305840 tx=0x7f8fdc048400 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fecaf9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff324c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8fecaf9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b3ff2f3ba0 con 0x55b3ff324820 2024-04-27T03:19:55.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8feae15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff324c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8feae15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff324c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51012/0 (socket says 172.21.15.110:51012) 2024-04-27T03:19:55.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8feae15640 1 -- 172.21.15.110:0/1778499565 learned_addr learned my addr 172.21.15.110:0/1778499565 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8feae15640 1 -- 172.21.15.110:0/1778499565 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ff325210 con 0x55b3ff324820 2024-04-27T03:19:55.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8feae15640 1 --2- 172.21.15.110:0/1778499565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff324c40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4015fa0 tx=0x7f8fe4015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48adb8e1e80dd9c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecdffb640 1 -- 172.21.15.110:0/1778499565 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe40113a0 con 0x55b3ff324820 2024-04-27T03:19:55.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecdffb640 1 -- 172.21.15.110:0/1778499565 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe4011540 con 0x55b3ff324820 2024-04-27T03:19:55.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.786+0000 7f8ecdffb640 1 -- 172.21.15.110:0/1778499565 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe40283a0 con 0x55b3ff324820 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1778499565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 msgr2=0x55b3ff324c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/1778499565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff324c40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4015fa0 tx=0x7f8fe4015fd0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1778499565 shutdown_connections 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/1778499565 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff324c40 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1778499565 >> 172.21.15.110:0/1778499565 conn(0x55b3ff320b60 msgr2=0x55b3ff31ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1778499565 shutdown_connections 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- 172.21.15.110:0/1778499565 wait complete. 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 Processor -- start 2024-04-27T03:19:55.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- start start 2024-04-27T03:19:55.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff32bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55b3ff325910 con 0x55b3ff324820 2024-04-27T03:19:55.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8feae15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff32bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8feae15640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff32bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51020/0 (socket says 172.21.15.110:51020) 2024-04-27T03:19:55.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 learned_addr learned my addr 172.21.15.110:0/3837964008 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3ff32c1f0 con 0x55b3ff324820 2024-04-27T03:19:55.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8feae15640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff32bcb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8fe401eb40 tx=0x7f8fe404a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8ebbfff640 1 -- 172.21.15.110:0/3837964008 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe40113a0 con 0x55b3ff324820 2024-04-27T03:19:55.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3ff32c480 con 0x55b3ff324820 2024-04-27T03:19:55.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8ebbfff640 1 -- 172.21.15.110:0/3837964008 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe4013a50 con 0x55b3ff324820 2024-04-27T03:19:55.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8ebbfff640 1 -- 172.21.15.110:0/3837964008 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8fe4013c20 con 0x55b3ff324820 2024-04-27T03:19:55.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3ff32f160 con 0x55b3ff324820 2024-04-27T03:19:55.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.790+0000 7f8ebbfff640 1 -- 172.21.15.110:0/3837964008 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8fe40491b0 con 0x55b3ff324820 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8ebbfff640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8eb4034650 0x7f8eb4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8ebbfff640 1 -- 172.21.15.110:0/3837964008 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f8fe4033ad0 con 0x55b3ff324820 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fea614640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8eb4034650 0x7f8eb4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff337370 0x55b3ff339810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff339f60 con 0x55b3ff337370 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fe9e13640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff337370 0x55b3ff339810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fea614640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8eb4034650 0x7f8eb4036b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000fe0 tx=0x7f8fdc034000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fe9e13640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff337370 0x55b3ff339810 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 realm 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff33a390 con 0x55b3ff337370 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff33ac60 con 0x55b3ff337370 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff33b680 0x55b3ff33da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff33e1e0 con 0x55b3ff33b680 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8feae15640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff33b680 0x55b3ff33da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8feae15640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff33b680 0x55b3ff33da70 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff33ea80 con 0x55b3ff337370 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 20 searching for the correct realm 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b3ff3407f0 0x55b3ff342be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff3407f0 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.794+0000 7f8fea614640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b3ff3407f0 0x55b3ff342be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fea614640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b3ff3407f0 0x55b3ff342be0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8feae15640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8fe4014030 0x7f8fe40133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x7f8fe4014030 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8fe4014030 0x7f8fe40133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8fe4014030 0x7f8fe40133b0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff3407f0 2024-04-27T03:19:55.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.798+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x7f8fe4014030 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x7f8fe4014030 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff3407f0 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.802+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff3407f0 2024-04-27T03:19:55.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x7f8fe4014030 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff3407f0 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x7f8fe4014030 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff337370 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x7f8fe4014030 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff3407f0 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x7f8fe4014030 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.806+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff3407f0 2024-04-27T03:19:55.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3431c0 con 0x55b3ff33b680 2024-04-27T03:19:55.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:55.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:55.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff33f3d0 con 0x55b3ff337370 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff33f820 con 0x55b3ff33b680 2024-04-27T03:19:55.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:55.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff33fc40 con 0x55b3ff337370 2024-04-27T03:19:55.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 zone default found 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 4 Realm: () 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 zonegroup default 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff343ad0 con 0x7f8fe4014030 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.810+0000 7f8fecaf9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3454b0 con 0x55b3ff3407f0 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff345990 con 0x55b3ff3407f0 2024-04-27T03:19:55.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 20 started sync module instance, tier type = 2024-04-27T03:19:55.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:55.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344940 con 0x55b3ff33b680 2024-04-27T03:19:55.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94231569124656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3462a0 con 0x55b3ff33b680 2024-04-27T03:19:55.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.814+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff346af0 con 0x55b3ff337370 2024-04-27T03:19:55.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94231569130144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff346c30 con 0x55b3ff337370 2024-04-27T03:19:55.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff347750 con 0x55b3ff33b680 2024-04-27T03:19:55.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94231569124656] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94231569130144] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94231569131200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff347710 con 0x55b3ff33b680 2024-04-27T03:19:55.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff347ff0 con 0x7f8fe4014030 2024-04-27T03:19:55.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94231569136448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff347710 con 0x7f8fe4014030 2024-04-27T03:19:55.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.818+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff348aa0 con 0x55b3ff33b680 2024-04-27T03:19:55.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94231569131200] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94231569136448] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94231569137504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff348aa0 con 0x55b3ff33b680 2024-04-27T03:19:55.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff348e90 con 0x55b3ff3407f0 2024-04-27T03:19:55.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94231569134112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3496e0 con 0x55b3ff3407f0 2024-04-27T03:19:55.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff34a0e0 con 0x55b3ff3407f0 2024-04-27T03:19:55.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.822+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94231569137504] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94231569134112] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94231569142320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff349bf0 con 0x55b3ff3407f0 2024-04-27T03:19:55.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff34aaf0 con 0x7f8fe4014030 2024-04-27T03:19:55.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'231 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94231569147456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff34af10 con 0x7f8fe4014030 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94231569142320] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=0 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=1 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=2 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=3 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=4 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=5 2024-04-27T03:19:55.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=6 2024-04-27T03:19:55.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94231569147456] v31'232 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 20 add_watcher() i=7 2024-04-27T03:19:55.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:55.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3485b0 con 0x55b3ff337370 2024-04-27T03:19:55.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.826+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94231569148512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3485b0 con 0x55b3ff337370 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94231569148512] v31'236 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8ea0fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 5 note: GC not initialized 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8e1b7fe640 20 reqs_thread_entry: start 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 20 Filter name: none 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8e197fa640 20 reqs_thread_entry: start 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3711a0 con 0x55b3ff337370 2024-04-27T03:19:55.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:55.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 6.13 6:cea599e4:usage::usage.10:head [call rgw.user_usage_log_read in=63b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff3726f0 con 0x55b3ff3407f0 2024-04-27T03:19:55.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.830+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 usage.10 [call out=15b] v0'0 uv32 ondisk = 0) ==== 152+0+15 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.834+0000 7f8ea0fe1640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:55.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.834+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94231569148512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff34ff60 con 0x55b3ff337370 2024-04-27T03:19:55.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.834+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(64 data_loggenerations_metadata [watch unwatch cookie 94231569148512] v31'237 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.834+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94231569124656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff34aa60 con 0x55b3ff33b680 2024-04-27T03:19:55.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(65 notify.0 [watch unwatch cookie 94231569124656] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fecaf9980 20 remove_watcher() i=0 2024-04-27T03:19:55.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fecaf9980 2 removed watcher, disabling cache 2024-04-27T03:19:55.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94231569130144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344d30 con 0x55b3ff337370 2024-04-27T03:19:55.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 notify.1 [watch unwatch cookie 94231569130144] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x55b3ff337370 2024-04-27T03:19:55.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fecaf9980 20 remove_watcher() i=1 2024-04-27T03:19:55.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94231569131200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344d30 con 0x55b3ff33b680 2024-04-27T03:19:55.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(67 notify.2 [watch unwatch cookie 94231569131200] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fecaf9980 20 remove_watcher() i=2 2024-04-27T03:19:55.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.838+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:68 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94231569136448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344d30 con 0x7f8fe4014030 2024-04-27T03:19:55.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.842+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(68 notify.3 [watch unwatch cookie 94231569136448] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.842+0000 7f8fecaf9980 20 remove_watcher() i=3 2024-04-27T03:19:55.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.842+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94231569137504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344d30 con 0x55b3ff33b680 2024-04-27T03:19:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.842+0000 7f8feae15640 1 -- 172.21.15.110:0/3837964008 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(69 notify.4 [watch unwatch cookie 94231569137504] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe4026020 con 0x55b3ff33b680 2024-04-27T03:19:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.842+0000 7f8fecaf9980 20 remove_watcher() i=4 2024-04-27T03:19:55.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.842+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94231569134112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344d30 con 0x55b3ff3407f0 2024-04-27T03:19:55.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.5 [watch unwatch cookie 94231569134112] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 20 remove_watcher() i=5 2024-04-27T03:19:55.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:71 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94231569142320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344d30 con 0x55b3ff3407f0 2024-04-27T03:19:55.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fea614640 1 -- 172.21.15.110:0/3837964008 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(71 notify.6 [watch unwatch cookie 94231569142320] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fdc008030 con 0x55b3ff3407f0 2024-04-27T03:19:55.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 20 remove_watcher() i=6 2024-04-27T03:19:55.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:72 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94231569147456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55b3ff344d30 con 0x7f8fe4014030 2024-04-27T03:19:55.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(72 notify.7 [watch unwatch cookie 94231569147456] v31'233 uv231 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f8fe00b3040 con 0x7f8fe4014030 2024-04-27T03:19:55.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 20 remove_watcher() i=7 2024-04-27T03:19:55.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff33b680 msgr2=0x55b3ff33da70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff33b680 0x55b3ff33da70 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff337370 msgr2=0x55b3ff339810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff337370 0x55b3ff339810 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8fe4014030 msgr2=0x7f8fe40133b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f8fe4014030 0x7f8fe40133b0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b3ff3407f0 msgr2=0x55b3ff342be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55b3ff3407f0 0x55b3ff342be0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8eb4034650 msgr2=0x7f8eb4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8eb4034650 0x7f8eb4036b10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000fe0 tx=0x7f8fdc034000 comp rx=0 tx=0).stop 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 msgr2=0x55b3ff32bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fe9e13640 1 -- 172.21.15.110:0/3837964008 reap_dead start 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff32bcb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8fe401eb40 tx=0x7f8fe404a880 comp rx=0 tx=0).stop 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 shutdown_connections 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3837964008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff324820 0x55b3ff32bcb0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 >> 172.21.15.110:0/3837964008 conn(0x55b3ff320b60 msgr2=0x55b3ff31e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 shutdown_connections 2024-04-27T03:19:55.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3837964008 wait complete. 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ec4034650 msgr2=0x7f8ec4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ec4034650 0x7f8ec4036b10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x55b3ff305840 tx=0x7f8fdc048400 comp rx=0 tx=0).stop 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff2ef710 msgr2=0x55b3ff2edbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff2ef710 0x55b3ff2edbb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400dfc0 tx=0x7f8fe40178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 shutdown_connections 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ec4034650 0x7f8ec4036b10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/974787507 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff2ef710 0x55b3ff2edbb0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 >> 172.21.15.110:0/974787507 conn(0x55b3ff30a240 msgr2=0x55b3ff30ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 shutdown_connections 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.846+0000 7f8fecaf9980 1 -- 172.21.15.110:0/974787507 wait complete. 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff302820 msgr2=0x55b3ff304c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff302820 0x55b3ff304c10 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff20b6b0 msgr2=0x55b3ff301460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff20b6b0 0x55b3ff301460 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ed00346a0 msgr2=0x7f8ed0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ed00346a0 0x7f8ed0036b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8fe00ad000 tx=0x7f8fe00a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 msgr2=0x55b3ff2fa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2fa530 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00e790 tx=0x7f8fdc000ee0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 shutdown_connections 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55b3ff302820 0x55b3ff304c10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55b3ff20b6b0 0x55b3ff301460 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8ed00346a0 0x7f8ed0036b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 --2- 172.21.15.110:0/3397289662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55b3ff207150 0x55b3ff2fa530 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 >> 172.21.15.110:0/3397289662 conn(0x55b3ff202a20 msgr2=0x55b3ff2eca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 shutdown_connections 2024-04-27T03:19:55.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.850+0000 7f8fecaf9980 1 -- 172.21.15.110:0/3397289662 wait complete. 2024-04-27T03:19:55.869 INFO:teuthology.orchestra.run.smithi110.stdout:{"entries":[],"summary":[]} 2024-04-27T03:19:55.870 DEBUG:tasks.util.rgw: json result: {'entries': [], 'summary': []} 2024-04-27T03:19:55.870 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'foo', '--purge-data'] 2024-04-27T03:19:55.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'foo', '--purge-data'] 2024-04-27T03:19:55.870 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid foo --purge-data 2024-04-27T03:19:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:55.941 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:55.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.930+0000 7fd2f99e4980 1 Processor -- start 2024-04-27T03:19:55.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.930+0000 7fd2f99e4980 1 -- start start 2024-04-27T03:19:55.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.930+0000 7fd2f99e4980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x561190c5c670 0x561190c5ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.930+0000 7fd2f99e4980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x561190b170d0 con 0x561190bcc860 2024-04-27T03:19:55.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.930+0000 7fd2f99e4980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x561190b0c430 con 0x561190c5c670 2024-04-27T03:19:55.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f6cfe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x561190c5c670 0x561190c5ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f74ff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x561190bcc860 0x561190c5af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44474/0 (socket says 172.21.15.110:44474) 2024-04-27T03:19:55.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f74ff640 1 -- 172.21.15.110:0/2961237986 learned_addr learned my addr 172.21.15.110:0/2961237986 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:55.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/2961237986 >> v1:172.21.15.77:6789/0 conn(0x561190bcc860 legacy=0x561190c5af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/2961237986 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x561190b05510 con 0x561190c5c670 2024-04-27T03:19:55.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f6cfe640 1 --2- 172.21.15.110:0/2961237986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c5c670 0x561190c5ea60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000a480 tx=0x7fd2e001b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f7a147731cecbf server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f64fd640 1 -- 172.21.15.110:0/2961237986 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00b0070 con 0x561190c5c670 2024-04-27T03:19:55.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f64fd640 1 -- 172.21.15.110:0/2961237986 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e00ab540 con 0x561190c5c670 2024-04-27T03:19:55.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.934+0000 7fd2f64fd640 1 -- 172.21.15.110:0/2961237986 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e00ab840 con 0x561190c5c670 2024-04-27T03:19:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.938+0000 7fd2f99e4980 1 -- 172.21.15.110:0/2961237986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c5c670 msgr2=0x561190c5ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.938+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/2961237986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c5c670 0x561190c5ea60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000a480 tx=0x7fd2e001b280 comp rx=0 tx=0).stop 2024-04-27T03:19:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.938+0000 7fd2f99e4980 1 -- 172.21.15.110:0/2961237986 shutdown_connections 2024-04-27T03:19:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.938+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/2961237986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c5c670 0x561190c5ea60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.938+0000 7fd2f99e4980 1 -- 172.21.15.110:0/2961237986 >> 172.21.15.110:0/2961237986 conn(0x561190c52a30 msgr2=0x561190c54e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.938+0000 7fd2f99e4980 1 -- 172.21.15.110:0/2961237986 shutdown_connections 2024-04-27T03:19:55.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.938+0000 7fd2f99e4980 1 -- 172.21.15.110:0/2961237986 wait complete. 2024-04-27T03:19:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.950+0000 7fd2f99e4980 1 Processor -- start 2024-04-27T03:19:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.950+0000 7fd2f99e4980 1 -- start start 2024-04-27T03:19:55.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.950+0000 7fd2f99e4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.950+0000 7fd2f99e4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561190b5b2d0 con 0x561190bcc860 2024-04-27T03:19:55.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d434f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51046/0 (socket says 172.21.15.110:51046) 2024-04-27T03:19:55.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 -- 172.21.15.110:0/372849249 learned_addr learned my addr 172.21.15.110:0/372849249 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 -- 172.21.15.110:0/372849249 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561190ae1ae0 con 0x561190bcc860 2024-04-27T03:19:55.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/372849249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d434f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00017b0 tx=0x7fd2f0001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483d510a4232db96 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd1e560a640 1 -- 172.21.15.110:0/372849249 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2f0016070 con 0x561190bcc860 2024-04-27T03:19:55.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd1e560a640 1 -- 172.21.15.110:0/372849249 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f0004530 con 0x561190bcc860 2024-04-27T03:19:55.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd1e560a640 1 -- 172.21.15.110:0/372849249 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2f0004870 con 0x561190bcc860 2024-04-27T03:19:55.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 -- 172.21.15.110:0/372849249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 msgr2=0x561190d434f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/372849249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d434f0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00017b0 tx=0x7fd2f0001c70 comp rx=0 tx=0).stop 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 -- 172.21.15.110:0/372849249 shutdown_connections 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/372849249 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d434f0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 -- 172.21.15.110:0/372849249 >> 172.21.15.110:0/372849249 conn(0x561190c52a30 msgr2=0x561190d3f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 -- 172.21.15.110:0/372849249 shutdown_connections 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 -- 172.21.15.110:0/372849249 wait complete. 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 Processor -- start 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 -- start start 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d4a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f99e4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561190c5fb40 con 0x561190bcc860 2024-04-27T03:19:55.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d4a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d4a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51056/0 (socket says 172.21.15.110:51056) 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 -- 172.21.15.110:0/1404465487 learned_addr learned my addr 172.21.15.110:0/1404465487 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.954+0000 7fd2f74ff640 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561190d4aa40 con 0x561190bcc860 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d4a500 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0000f90 tx=0x7fd2f0001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd1e3e07640 1 -- 172.21.15.110:0/1404465487 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2f0020070 con 0x561190bcc860 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd1e3e07640 1 -- 172.21.15.110:0/1404465487 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f0011400 con 0x561190bcc860 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd1e3e07640 1 -- 172.21.15.110:0/1404465487 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2f0016040 con 0x561190bcc860 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561190d4acd0 con 0x561190bcc860 2024-04-27T03:19:55.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561190d4d950 con 0x561190bcc860 2024-04-27T03:19:55.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd1e3e07640 1 -- 172.21.15.110:0/1404465487 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd2f0023070 con 0x561190bcc860 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd1e3e07640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1dc0346a0 0x7fd1dc036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd1e3e07640 1 -- 172.21.15.110:0/1404465487 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fd2f0049350 con 0x561190bcc860 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f6cfe640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1dc0346a0 0x7fd1dc036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190c5c670 0x561190d515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d51ae0 con 0x561190c5c670 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190c5c670 0x561190d515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f6cfe640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1dc0346a0 0x7fd1dc036b60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00ad000 tx=0x7fd2e00a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190c5c670 0x561190d515a0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f7d00640 1 -- 172.21.15.110:0/1404465487 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e80083f0 con 0x561190c5c670 2024-04-27T03:19:55.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d52590 0x561190d54a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.958+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d551c0 con 0x561190d52590 2024-04-27T03:19:55.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d52590 0x561190d54a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d52590 0x561190d54a50 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f74ff640 1 -- 172.21.15.110:0/1404465487 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d52590 2024-04-27T03:19:55.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d56a70 con 0x561190c5c670 2024-04-27T03:19:55.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f7d00640 1 -- 172.21.15.110:0/1404465487 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fd2e80083f0 con 0x561190c5c670 2024-04-27T03:19:55.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d55ca0 con 0x561190d52590 2024-04-27T03:19:55.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f74ff640 1 -- 172.21.15.110:0/1404465487 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d52590 2024-04-27T03:19:55.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d56090 con 0x561190c5c670 2024-04-27T03:19:55.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f7d00640 1 -- 172.21.15.110:0/1404465487 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fd2e80083f0 con 0x561190c5c670 2024-04-27T03:19:55.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f99e4980 1 Processor -- start 2024-04-27T03:19:55.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f99e4980 1 -- start start 2024-04-27T03:19:55.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f99e4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c58ab0 0x561190c56f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f99e4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561190b5f310 con 0x561190c58ab0 2024-04-27T03:19:55.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f7d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c58ab0 0x561190c56f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f7d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c58ab0 0x561190c56f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51066/0 (socket says 172.21.15.110:51066) 2024-04-27T03:19:55.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.962+0000 7fd2f7d00640 1 -- 172.21.15.110:0/1132157674 learned_addr learned my addr 172.21.15.110:0/1132157674 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f7d00640 1 -- 172.21.15.110:0/1132157674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561190c57490 con 0x561190c58ab0 2024-04-27T03:19:55.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c58ab0 0x561190c56f50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800aab0 tx=0x7fd2e800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd1e1602640 1 -- 172.21.15.110:0/1132157674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e8007d30 con 0x561190c58ab0 2024-04-27T03:19:55.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd1e1602640 1 -- 172.21.15.110:0/1132157674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e801d070 con 0x561190c58ab0 2024-04-27T03:19:55.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561190c57720 con 0x561190c58ab0 2024-04-27T03:19:55.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd1e1602640 1 -- 172.21.15.110:0/1132157674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e8018440 con 0x561190c58ab0 2024-04-27T03:19:55.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561190c57bf0 con 0x561190c58ab0 2024-04-27T03:19:55.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd1e1602640 1 -- 172.21.15.110:0/1132157674 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd2e8013070 con 0x561190c58ab0 2024-04-27T03:19:55.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd1e1602640 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d0034650 0x7fd1d0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd1e1602640 1 -- 172.21.15.110:0/1132157674 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fd2e804b720 con 0x561190c58ab0 2024-04-27T03:19:55.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d0034650 0x7fd1d0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f99e4980 1 Processor -- start 2024-04-27T03:19:55.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f99e4980 1 -- start start 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f99e4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d74b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d0034650 0x7fd1d0036b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x561190d557c0 tx=0x7fd2f0024410 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f99e4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561190d43c40 con 0x561190d74730 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f7d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d74b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f7d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d74b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51072/0 (socket says 172.21.15.110:51072) 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3708040373 learned_addr learned my addr 172.21.15.110:0/3708040373 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.966+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3708040373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561190d75120 con 0x561190d74730 2024-04-27T03:19:55.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/3708040373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d74b50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8049930 tx=0x7fd2e8049b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702342c1755c0bb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd1daffd640 1 -- 172.21.15.110:0/3708040373 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e8047070 con 0x561190d74730 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd1daffd640 1 -- 172.21.15.110:0/3708040373 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e8022460 con 0x561190d74730 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd1daffd640 1 -- 172.21.15.110:0/3708040373 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e8018bc0 con 0x561190d74730 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3708040373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 msgr2=0x561190d74b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3708040373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d74b50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8049930 tx=0x7fd2e8049b30 comp rx=0 tx=0).stop 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3708040373 shutdown_connections 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3708040373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d74b50 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:55.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3708040373 >> 172.21.15.110:0/3708040373 conn(0x561190d70ac0 msgr2=0x561190d6eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3708040373 shutdown_connections 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3708040373 wait complete. 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 Processor -- start 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- start start 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d7bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x561190d75890 con 0x561190d74730 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f7d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d7bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f7d00640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d7bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51084/0 (socket says 172.21.15.110:51084) 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 learned_addr learned my addr 172.21.15.110:0/3619781954 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:55.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561190d7c1a0 con 0x561190d74730 2024-04-27T03:19:55.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d7bc60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x561190d711c0 tx=0x7fd2e80498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd1d97fa640 1 -- 172.21.15.110:0/3619781954 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e8031070 con 0x561190d74730 2024-04-27T03:19:55.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561190d7c430 con 0x561190d74730 2024-04-27T03:19:55.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd1d97fa640 1 -- 172.21.15.110:0/3619781954 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e8020900 con 0x561190d74730 2024-04-27T03:19:55.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561190d7f110 con 0x561190d74730 2024-04-27T03:19:55.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.970+0000 7fd1d97fa640 1 -- 172.21.15.110:0/3619781954 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2e8017b90 con 0x561190d74730 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd1d97fa640 1 -- 172.21.15.110:0/3619781954 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd2e8034070 con 0x561190d74730 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd1d97fa640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c0034650 0x7fd1c0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd1d97fa640 1 -- 172.21.15.110:0/3619781954 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fd2e8037dd0 con 0x561190d74730 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c0034650 0x7fd1c0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190d87270 0x561190d89730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d89e80 con 0x561190d87270 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f6cfe640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190d87270 0x561190d89730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c0034650 0x7fd1c0036b10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0024b60 tx=0x7fd2f0031000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f6cfe640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190d87270 0x561190d89730 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 realm 2024-04-27T03:19:55.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d8a250 con 0x561190d87270 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d8ab20 con 0x561190d87270 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d8b540 0x561190d8d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d8e0c0 con 0x561190d8b540 2024-04-27T03:19:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d8b540 0x561190d8d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d8b540 0x561190d8d950 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.974+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d8e910 con 0x561190d87270 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f99e4980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f99e4980 20 searching for the correct realm 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561190d90680 0x561190d92a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d90680 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561190d90680 0x561190d92a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f74ff640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561190d90680 0x561190d92a70 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:55.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f7d00640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8036070 0x7fd2e8030a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x7fd2e8036070 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8036070 0x7fd2e8030a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8036070 0x7fd2e8030a30 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d90680 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.978+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:55.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x7fd2e8036070 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x7fd2e8036070 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d90680 2024-04-27T03:19:55.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:55.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.982+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d90680 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x7fd2e8036070 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d90680 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x7fd2e8036070 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d87270 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x7fd2e8036070 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d90680 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:55.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x7fd2e8036070 2024-04-27T03:19:56.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.986+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d90680 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x561190d930f0 con 0x561190d8b540 2024-04-27T03:19:56.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:56.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:56.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d8f290 con 0x561190d87270 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d8eea0 con 0x561190d8b540 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d8fb80 con 0x561190d87270 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.990+0000 7fd2f99e4980 20 zone default found 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 4 Realm: () 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 zonegroup default 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d93a00 con 0x7fd2e8036070 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d95670 con 0x561190d90680 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d95b20 con 0x561190d90680 2024-04-27T03:19:56.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 started sync module instance, tier type = 2024-04-27T03:19:56.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:56.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d94540 con 0x561190d8b540 2024-04-27T03:19:56.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94633444591920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d96430 con 0x561190d8b540 2024-04-27T03:19:56.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.994+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d96c80 con 0x561190d87270 2024-04-27T03:19:56.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94633444598832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d96d40 con 0x561190d87270 2024-04-27T03:19:56.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d97840 con 0x561190d8b540 2024-04-27T03:19:56.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94633444591920] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94633444598832] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94633444602672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d97800 con 0x561190d8b540 2024-04-27T03:19:56.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d980e0 con 0x7fd2e8036070 2024-04-27T03:19:56.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:55.998+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94633444604976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d97800 con 0x7fd2e8036070 2024-04-27T03:19:56.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d98b70 con 0x561190d8b540 2024-04-27T03:19:56.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94633444602672] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94633444604976] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94633444606032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d98b70 con 0x561190d8b540 2024-04-27T03:19:56.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d98f60 con 0x561190d90680 2024-04-27T03:19:56.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94633444597632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d997b0 con 0x561190d90680 2024-04-27T03:19:56.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.002+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d9a1b0 con 0x561190d90680 2024-04-27T03:19:56.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94633444606032] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94633444597632] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94633444607648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d9a5a0 con 0x561190d90680 2024-04-27T03:19:56.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d9adf0 con 0x7fd2e8036070 2024-04-27T03:19:56.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94633444607648] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'234 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94633444615712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d9aef0 con 0x7fd2e8036070 2024-04-27T03:19:56.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=0 2024-04-27T03:19:56.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=1 2024-04-27T03:19:56.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=2 2024-04-27T03:19:56.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=3 2024-04-27T03:19:56.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=4 2024-04-27T03:19:56.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=5 2024-04-27T03:19:56.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=6 2024-04-27T03:19:56.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94633444615712] v31'235 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 20 add_watcher() i=7 2024-04-27T03:19:56.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:56.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.006+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190d9e220 con 0x561190d87270 2024-04-27T03:19:56.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.010+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.010+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94633444629584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d9b060 con 0x561190d87270 2024-04-27T03:19:56.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.010+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94633444629584] v31'238 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.010+0000 7fd1adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:56.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.010+0000 7fd2f99e4980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:56.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.010+0000 7fd2f99e4980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:56.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.010+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d9d650 con 0x561190d8b540 2024-04-27T03:19:56.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.014+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v31'58 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.014+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d9d630 con 0x7fd2e8036070 2024-04-27T03:19:56.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.014+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 gc.1 [create,call,call,call] v31'36 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.014+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da74c0 con 0x561190d90680 2024-04-27T03:19:56.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.018+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v31'35 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.018+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da7a30 con 0x561190d87270 2024-04-27T03:19:56.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.018+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(65 gc.3 [create,call,call,call] v31'239 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.018+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da7ee0 con 0x561190d8b540 2024-04-27T03:19:56.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.022+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v31'82 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.022+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da7ee0 con 0x561190d90680 2024-04-27T03:19:56.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.022+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v31'61 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.022+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.022+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v31'38 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.022+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d87270 2024-04-27T03:19:56.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.026+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 gc.7 [create,call,call,call] v31'56 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.026+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.026+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v31'28 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.026+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.030+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v31'34 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.030+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.034+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v31'83 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.034+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d90680 2024-04-27T03:19:56.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.034+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v31'34 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.034+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.034+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v31'34 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.034+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d87270 2024-04-27T03:19:56.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.038+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(75 gc.13 [create,call,call,call] v31'67 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.038+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d87270 2024-04-27T03:19:56.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.038+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(76 gc.14 [create,call,call,call] v31'240 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.038+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.042+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v31'74 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.042+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.042+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v31'84 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.042+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d90680 2024-04-27T03:19:56.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.046+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v31'41 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.046+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.046+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v31'36 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.046+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d87270 2024-04-27T03:19:56.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.050+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(81 gc.19 [create,call,call,call] v31'241 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.050+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6950 con 0x561190d8b540 2024-04-27T03:19:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.054+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v31'75 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.054+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d8b540 2024-04-27T03:19:56.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.054+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v31'44 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.054+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d90680 2024-04-27T03:19:56.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.054+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v31'36 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.054+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d87270 2024-04-27T03:19:56.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.058+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 gc.23 [create,call,call,call] v31'46 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.058+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d90680 2024-04-27T03:19:56.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.058+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v31'30 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.058+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d90680 2024-04-27T03:19:56.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.062+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v31'62 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.062+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d90680 2024-04-27T03:19:56.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.062+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v31'37 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.062+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d8b540 2024-04-27T03:19:56.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.066+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(89 gc.27 [create,call,call,call] v31'41 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.066+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d87270 2024-04-27T03:19:56.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.066+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v31'57 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.066+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d8b540 2024-04-27T03:19:56.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.070+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(91 gc.29 [create,call,call,call] v31'76 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.070+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x561190d87270 2024-04-27T03:19:56.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.070+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v31'68 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.070+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190da6780 con 0x7fd2e8036070 2024-04-27T03:19:56.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(93 gc.31 [create,call,call,call] v31'34 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd1acfe1640 20 reqs_thread_entry: start 2024-04-27T03:19:56.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:56.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 20 Filter name: none 2024-04-27T03:19:56.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd1267fc640 20 reqs_thread_entry: start 2024-04-27T03:19:56.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 10 cache get: name=default.rgw.meta+users.uid+foo : miss 2024-04-27T03:19:56.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:94 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190da6840 con 0x561190d87270 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(94 foo [call out=48b,read 0~391 out=391b,getxattrs out=75b] v0'0 uv15 ondisk = 0) ==== 231+0+514 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 20 rados_obj.operate() r=0 bl.length=391 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 10 cache put: name=default.rgw.meta+users.uid+foo info.flags=0x13 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 10 adding default.rgw.meta+users.uid+foo to cache LRU end 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:95 8.9 8:96d98480:users.uid::foo.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190da6340 con 0x7fd2e8036070 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(95 foo.buckets [call out=183b] v0'0 uv43 ondisk = 0) ==== 155+0+183 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 10 cache get: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 : miss 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.074+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 8.11 8:89dd8ff7:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.7:head [call version.read in=11b,stat,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dc6100 con 0x561190d8b540 2024-04-27T03:19:56.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(96 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 [call out=48b,stat out=16b,read 0~379 out=379b,getxattrs out=218b] v0'0 uv1 ondisk = 0) ==== 332+0+661 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 20 rados_obj.operate() r=0 bl.length=379 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 10 cache put: name=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 info.flags=0x17 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 10 adding default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 to cache LRU end 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 10 updating xattr: name=user.rgw.acl bl.length()=123 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:97 4.6 4:622a2849:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e437b0 con 0x561190d90680 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(97 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 4.3e 4:7c9125d1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e437f0 con 0x561190d8b540 2024-04-27T03:19:56.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(98 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:99 4.3b 4:de8cd189:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x561190d90680 2024-04-27T03:19:56.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(99 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:100 4.1e 4:7bc6852b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x561190d87270 2024-04-27T03:19:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(100 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:101 4.3a 4:5c1af71e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x561190d90680 2024-04-27T03:19:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(101 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4 [call out=426b] v0'0 uv5 ondisk = 0) ==== 194+0+426 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 10 cls_bucket_list_unordered: got 15f6f8dc036519d7fe15b39338f6e5db 2024-04-27T03:19:56.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.078+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:102 4.20 4:040babb4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x561190d8b540 2024-04-27T03:19:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(102 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 4.7 4:e0693231:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x561190d87270 2024-04-27T03:19:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(103 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:104 4.30 4:0d615415:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x561190d8b540 2024-04-27T03:19:56.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(104 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 4.1e 4:786bebf2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x561190d87270 2024-04-27T03:19:56.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(105 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00b3040 con 0x561190d87270 2024-04-27T03:19:56.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:106 4.13 4:c84080dd:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x7fd2e8036070 2024-04-27T03:19:56.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(106 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00b3040 con 0x7fd2e8036070 2024-04-27T03:19:56.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:107 4.25 4:a45fef87:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e43830 con 0x7fd2e8036070 2024-04-27T03:19:56.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(107 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10 [call out=85b] v0'0 uv5 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7fd2e00fd060 con 0x7fd2e8036070 2024-04-27T03:19:56.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f99e4980 20 get_obj_state: octx=0x7fd2e00ab740 obj=myfoo:15f6f8dc036519d7fe15b39338f6e5db state=0x561190e41818 s->prefetch_data=0 2024-04-27T03:19:56.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 3.1as0 3:5bc8eaa9:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.7_15f6f8dc036519d7fe15b39338f6e5db:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190e42700 con 0x561190d8b540 2024-04-27T03:19:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.082+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(108 27a51273-3ccc-49e2-8682-1122abc3d898.4220.7_15f6f8dc036519d7fe15b39338f6e5db [getxattrs out=879b,stat out=16b] v0'0 uv1 ondisk = 0) ==== 262+0+895 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f99e4980 10 manifest: total_size = 6 2024-04-27T03:19:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f99e4980 20 get_obj_state: setting s->obj_tag to 27a51273-3ccc-49e2-8682-1122abc3d898.4220.12360926882154543801 2024-04-27T03:19:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f99e4980 20 get_obj_state: octx=0x7fd2e00ab740 obj=myfoo:15f6f8dc036519d7fe15b39338f6e5db state=0x561190e41818 s->prefetch_data=0 2024-04-27T03:19:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f99e4980 20 prepare_atomic_modification: state is not atomic. state=0x561190e41818 2024-04-27T03:19:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f99e4980 20 bucket index object: :.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4 2024-04-27T03:19:56.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:109 4.3a 4:5c1af71e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=210b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190e430e0 con 0x561190d90680 2024-04-27T03:19:56.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(109 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4 [stat,call,call] v31'8 uv8 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.086+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:110 3.1as0 3:5bc8eaa9:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.7_15f6f8dc036519d7fe15b39338f6e5db:head [call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190e430e0 con 0x561190d8b540 2024-04-27T03:19:56.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(110 27a51273-3ccc-49e2-8682-1122abc3d898.4220.7_15f6f8dc036519d7fe15b39338f6e5db [call] v31'2 uv2 ondisk = 0) ==== 220+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:111 4.3a 4:5c1af71e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_complete_op in=287b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dcad50 con 0x561190d90680 2024-04-27T03:19:56.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 20 list_objects_ordered: starting attempt 1 2024-04-27T03:19:56.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 10 cls_bucket_list_ordered: request from each of 11 shard(s) for 112 entries to get 1001 total entries 2024-04-27T03:19:56.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:112 4.6 4:622a2849:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dcad50 con 0x561190d90680 2024-04-27T03:19:56.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:113 4.3e 4:7c9125d1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dcefd0 con 0x561190d8b540 2024-04-27T03:19:56.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:114 4.3b 4:de8cd189:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd1890 con 0x561190d90680 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:115 4.1e 4:7bc6852b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd41f0 con 0x561190d87270 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:116 4.3a 4:5c1af71e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd6cd0 con 0x561190d90680 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:117 4.20 4:040babb4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9770 con 0x561190d8b540 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:118 4.7 4:e0693231:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190ddc4d0 con 0x561190d87270 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:119 4.30 4:0d615415:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190ddf2d0 con 0x561190d8b540 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:120 4.1e 4:786bebf2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190de1bc0 con 0x561190d87270 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:121 4.13 4:c84080dd:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190de4550 con 0x7fd2e8036070 2024-04-27T03:19:56.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:122 4.25 4:a45fef87:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10:head [call rgw.bucket_list in=69b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190de6e40 con 0x7fd2e8036070 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(113 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(112 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(115 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00fd060 con 0x561190d87270 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(114 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(118 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00fd060 con 0x561190d87270 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(117 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(119 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(120 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00fd060 con 0x561190d87270 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(121 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e00fd060 con 0x7fd2e8036070 2024-04-27T03:19:56.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(122 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10 [call out=85b] v0'0 uv5 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7fd2e00fd060 con 0x7fd2e8036070 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(111 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4 [stat,call,call] v31'9 uv9 ondisk = 0) ==== 278+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.090+0000 7fd1da7fc640 20 handle_completion(): completion ok for obj=15f6f8dc036519d7fe15b39338f6e5db 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(116 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4 [call out=124b] v0'0 uv9 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":myfoo[27a51273-3ccc-49e2-8682-1122abc3d898.4220.7])"; uploads.size()=0; is_truncated=0 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:123 4.6 4:622a2849:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd6cd0 con 0x561190d90680 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:124 4.3e 4:7c9125d1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d8b540 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:125 4.3b 4:de8cd189:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd2d30 con 0x561190d90680 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:126 4.1e 4:7bc6852b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd6cd0 con 0x561190d87270 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:127 4.3a 4:5c1af71e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d90680 2024-04-27T03:19:56.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:128 4.20 4:040babb4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd2d30 con 0x561190d8b540 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:129 4.7 4:e0693231:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd0880 con 0x561190d87270 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:130 4.30 4:0d615415:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d8b540 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:131 4.1e 4:786bebf2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd16b0 con 0x561190d87270 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:132 4.13 4:c84080dd:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd2d30 con 0x7fd2e8036070 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:133 4.25 4:a45fef87:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10:head [call rgw.bucket_list in=47b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190de3070 con 0x7fd2e8036070 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(123 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(125 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(126 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(124 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1 [call out=85b] v0'0 uv7 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(132 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(127 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4 [call out=124b] v0'0 uv9 ondisk = 0) ==== 194+0+124 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(128 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5 [call out=85b] v0'0 uv5 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(130 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7 [call out=85b] v0'0 uv3 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(129 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6 [call out=85b] v0'0 uv1 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(131 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8 [call out=85b] v0'0 uv2 ondisk = 0) ==== 194+0+85 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(133 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10 [call out=85b] v0'0 uv5 ondisk = 0) ==== 195+0+85 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:134 8.9 8:96d98480:users.uid::foo.buckets:head [call user.set_buckets_info in=207b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x7fd2e8036070 2024-04-27T03:19:56.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 22 ==== osd_op_reply(134 foo.buckets [call] v31'46 uv46 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 10 cache get: name=default.rgw.meta+root+myfoo : miss 2024-04-27T03:19:56.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:135 8.9 8:91b71841:root::myfoo:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x7fd2e8036070 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 23 ==== osd_op_reply(135 myfoo [call out=48b,read 0~165 out=165b] v0'0 uv40 ondisk = 0) ==== 191+0+213 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 20 rados_obj.operate() r=0 bl.length=165 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 10 cache put: name=default.rgw.meta+root+myfoo info.flags=0x11 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 10 adding default.rgw.meta+root+myfoo to cache LRU end 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.094+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:136 8.9 8:91b71841:root::myfoo:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x7fd2e8036070 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 24 ==== osd_op_reply(136 myfoo [call,call,delete] v31'47 uv47 ondisk = 0) ==== 233+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f99e4980 10 removing default.rgw.meta+root+myfoo from cache 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f99e4980 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:myfoo, ofs0, ns] 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:137 7.14 7:2b04a3e9:::notify.6:head [notify cookie 94633444813952 in=161b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d90680 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== watch-notify(notify (1) cookie 94633444607648 notify 133143986210 ret 0) ==== 191+0+0 (crc 0 0 0) 0x561190d9ce70 con 0x561190d90680 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd1da7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986210 cookie 94633444607648 notifier 4772 bl.length()=149 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(137 notify.6 [notify cookie 94633444813952 out=8b] v0'0 uv234 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd1da7fc640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:138 7.14 7:2b04a3e9:::notify.6:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fd12c0032d0 con 0x561190d90680 2024-04-27T03:19:56.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(138 notify.6 [notify-ack] v0'0 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== watch-notify(notify_complete (2) cookie 94633444813952 notify 133143986210 ret 0) ==== 42+0+48 (crc 0 0 0) 0x561190d9d5f0 con 0x561190d90680 2024-04-27T03:19:56.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.098+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:139 8.11 8:89dd8ff7:root::.bucket.meta.myfoo%3a27a51273-3ccc-49e2-8682-1122abc3d898.4220.7:head [call version.check_conds in=122b,call version.inc in=38b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d8b540 2024-04-27T03:19:56.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== osd_op_reply(139 .bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 [call,call,delete] v31'2 uv2 ondisk = 0) ==== 290+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 10 removing default.rgw.meta+root+.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 from cache 2024-04-27T03:19:56.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.myfoo:27a51273-3ccc-49e2-8682-1122abc3d898.4220.7, ofs0, ns] 2024-04-27T03:19:56.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:140 7.15 7:a93a5511:::notify.2:head [notify cookie 94633444813872 in=218b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d8b540 2024-04-27T03:19:56.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 45 ==== watch-notify(notify (1) cookie 94633444602672 notify 133143986223 ret 0) ==== 248+0+0 (crc 0 0 0) 0x561190dc5690 con 0x561190d8b540 2024-04-27T03:19:56.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd1daffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986223 cookie 94633444602672 notifier 4772 bl.length()=206 2024-04-27T03:19:56.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 46 ==== osd_op_reply(140 notify.2 [notify cookie 94633444813872 out=8b] v0'0 uv234 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd1daffd640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:141 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fd1bc003d70 con 0x561190d8b540 2024-04-27T03:19:56.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 47 ==== osd_op_reply(141 notify.2 [notify-ack] v0'0 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 48 ==== watch-notify(notify_complete (2) cookie 94633444813872 notify 133143986223 ret 0) ==== 42+0+48 (crc 0 0 0) 0x561190d7c430 con 0x561190d8b540 2024-04-27T03:19:56.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:142 4.6 4:622a2849:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d90680 2024-04-27T03:19:56.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:143 4.3e 4:7c9125d1:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190ddbd70 con 0x561190d8b540 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:144 4.3b 4:de8cd189:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190ddce70 con 0x561190d90680 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:145 4.1e 4:7bc6852b:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dc4680 con 0x561190d87270 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:146 4.3a 4:5c1af71e:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dc4f90 con 0x561190d90680 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:147 4.20 4:040babb4:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dca850 con 0x561190d8b540 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:148 4.7 4:e0693231:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190de2970 con 0x561190d87270 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:149 4.30 4:0d615415:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190ddce70 con 0x561190d8b540 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:150 4.1e 4:786bebf2:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dddf90 con 0x561190d87270 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:151 4.13 4:c84080dd:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190ddbd70 con 0x7fd2e8036070 2024-04-27T03:19:56.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.102+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:152 4.25 4:a45fef87:::.dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dc4680 con 0x7fd2e8036070 2024-04-27T03:19:56.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(148 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.6 [delete] v31'4 uv4 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 49 ==== osd_op_reply(147 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.5 [delete] v31'8 uv8 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 25 ==== osd_op_reply(152 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.10 [delete] v31'6 uv6 ondisk = 0) ==== 195+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 50 ==== osd_op_reply(149 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.7 [delete] v31'4 uv4 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(144 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.2 [delete] v31'12 uv12 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 51 ==== osd_op_reply(143 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.1 [delete] v31'12 uv12 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== osd_op_reply(146 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.4 [delete] v31'10 uv10 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(145 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.3 [delete] v31'5 uv5 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== osd_op_reply(142 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.0 [delete] v31'2 uv2 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(150 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.8 [delete] v31'6 uv6 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 26 ==== osd_op_reply(151 .dir.27a51273-3ccc-49e2-8682-1122abc3d898.4220.7.9 [delete] v31'6 uv6 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f99e4980 10 cache get: name=default.rgw.log++pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 : miss 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f99e4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:153 6.1d 6:bd4f8e46:::pubsub..bucket.myfoo%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.7:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190de2970 con 0x561190d87270 2024-04-27T03:19:56.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(153 pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 208+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f99e4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f99e4980 10 cache put: name=default.rgw.log++pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 info.flags=0x0 2024-04-27T03:19:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f99e4980 10 adding default.rgw.log++pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 to cache LRU end 2024-04-27T03:19:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.106+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:154 6.1d 6:bd4f8e46:::pubsub..bucket.myfoo%2f27a51273-3ccc-49e2-8682-1122abc3d898.4220.7:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dc4680 con 0x561190d87270 2024-04-27T03:19:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 42 ==== osd_op_reply(154 pubsub..bucket.myfoo/27a51273-3ccc-49e2-8682-1122abc3d898.4220.7 [delete] v31'58 uv43 ondisk = -2 ((2) No such file or directory)) ==== 208+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:155 8.9 8:96d98480:users.uid::foo.buckets:head [call user.remove_bucket in=144b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dc4680 con 0x7fd2e8036070 2024-04-27T03:19:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 27 ==== osd_op_reply(155 foo.buckets [call] v31'48 uv48 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f99e4980 10 removing key index: 9te6NH5mcdcq0Tc5i8i1 2024-04-27T03:19:56.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:156 8.7 8:e4e0e270:users.keys::9te6NH5mcdcq0Tc5i8i1:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x561190d87270 2024-04-27T03:19:56.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 43 ==== osd_op_reply(156 9te6NH5mcdcq0Tc5i8i1 [delete] v31'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f99e4980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:9te6NH5mcdcq0Tc5i8i1, ofs0, ns] 2024-04-27T03:19:56.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:157 7.e 7:7759931f:::notify.3:head [notify cookie 94633444792640 in=182b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x7fd2e8036070 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 28 ==== watch-notify(notify (1) cookie 94633444604976 notify 133143986193 ret 0) ==== 212+0+0 (crc 0 0 0) 0x561190da8ac0 con 0x7fd2e8036070 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd1da7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986193 cookie 94633444604976 notifier 4772 bl.length()=170 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 29 ==== osd_op_reply(157 notify.3 [notify cookie 94633444792640 out=8b] v0'0 uv234 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd1da7fc640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:158 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fd12c0032d0 con 0x7fd2e8036070 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.110+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 30 ==== osd_op_reply(158 notify.3 [notify-ack] v0'0 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 31 ==== watch-notify(notify_complete (2) cookie 94633444792640 notify 133143986193 ret 0) ==== 42+0+48 (crc 0 0 0) 0x561190da6cf0 con 0x7fd2e8036070 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f99e4980 10 removing email index: foo@foo.com 2024-04-27T03:19:56.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:159 8.3 8:c162d5d1:users.email::foo@foo.com:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x561190d87270 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 44 ==== osd_op_reply(159 foo@foo.com [delete] v31'4 uv4 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f99e4980 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.email:foo@foo.com, ofs0, ns] 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:160 7.e 7:7759931f:::notify.3:head [notify cookie 94633444792640 in=174b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x7fd2e8036070 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 32 ==== watch-notify(notify (1) cookie 94633444604976 notify 133143986194 ret 0) ==== 204+0+0 (crc 0 0 0) 0x561190da6cf0 con 0x7fd2e8036070 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd1daffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986194 cookie 94633444604976 notifier 4772 bl.length()=162 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 33 ==== osd_op_reply(160 notify.3 [notify cookie 94633444792640 out=8b] v0'0 uv234 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd1daffd640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:161 7.e 7:7759931f:::notify.3:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fd1bc003d70 con 0x7fd2e8036070 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 34 ==== osd_op_reply(161 notify.3 [notify-ack] v0'0 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 35 ==== watch-notify(notify_complete (2) cookie 94633444792640 notify 133143986194 ret 0) ==== 42+0+48 (crc 0 0 0) 0x561190d9d100 con 0x7fd2e8036070 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f99e4980 10 removing user buckets index 2024-04-27T03:19:56.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.114+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:162 8.9 8:96d98480:users.uid::foo.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x7fd2e8036070 2024-04-27T03:19:56.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 36 ==== osd_op_reply(162 foo.buckets [delete] v31'49 uv49 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f99e4980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.buckets, ofs0, ns] 2024-04-27T03:19:56.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:163 7.3 7:c609908c:::notify.5:head [notify cookie 94633444792640 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x561190d90680 2024-04-27T03:19:56.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== watch-notify(notify (1) cookie 94633444597632 notify 133143986211 ret 0) ==== 202+0+0 (crc 0 0 0) 0x561190d9d5f0 con 0x561190d90680 2024-04-27T03:19:56.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd1da7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986211 cookie 94633444597632 notifier 4772 bl.length()=160 2024-04-27T03:19:56.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd1da7fc640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:164 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fd12c0032d0 con 0x561190d90680 2024-04-27T03:19:56.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== osd_op_reply(163 notify.5 [notify cookie 94633444792640 out=8b] v0'0 uv234 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== watch-notify(notify_complete (2) cookie 94633444792640 notify 133143986211 ret 0) ==== 42+0+48 (crc 0 0 0) 0x561190d4d950 con 0x561190d90680 2024-04-27T03:19:56.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f99e4980 10 removing user index: foo 2024-04-27T03:19:56.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== osd_op_reply(164 notify.5 [notify-ack] v0'0 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:165 8.18 8:1e6ca5ed:users.uid::foo:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x561190d87270 2024-04-27T03:19:56.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 45 ==== osd_op_reply(165 foo [call,call,delete] v31'21 uv21 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f99e4980 10 removing default.rgw.meta+users.uid+foo from cache 2024-04-27T03:19:56.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f99e4980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.uid:foo, ofs0, ns] 2024-04-27T03:19:56.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:166 7.d 7:b4812045:::notify.4:head [notify cookie 94633444883824 in=164b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x561190d8b540 2024-04-27T03:19:56.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 52 ==== watch-notify(notify (1) cookie 94633444606032 notify 133143986224 ret 0) ==== 194+0+0 (crc 0 0 0) 0x561190d7c430 con 0x561190d8b540 2024-04-27T03:19:56.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd1daffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986224 cookie 94633444606032 notifier 4772 bl.length()=152 2024-04-27T03:19:56.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd1daffd640 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:167 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fd1bc003d70 con 0x561190d8b540 2024-04-27T03:19:56.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.118+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 53 ==== osd_op_reply(166 notify.4 [notify cookie 94633444883824 out=8b] v0'0 uv234 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.122+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 54 ==== watch-notify(notify_complete (2) cookie 94633444883824 notify 133143986224 ret 0) ==== 42+0+48 (crc 0 0 0) 0x561190d7c1a0 con 0x561190d8b540 2024-04-27T03:19:56.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.122+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 55 ==== osd_op_reply(167 notify.4 [notify-ack] v0'0 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.122+0000 7fd1adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:56.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.122+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:168 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94633444629584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x561190d87270 2024-04-27T03:19:56.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.122+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 46 ==== osd_op_reply(168 data_loggenerations_metadata [watch unwatch cookie 94633444629584] v31'242 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:169 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94633444591920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190dd9800 con 0x561190d8b540 2024-04-27T03:19:56.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 56 ==== osd_op_reply(169 notify.0 [watch unwatch cookie 94633444591920] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f99e4980 20 remove_watcher() i=0 2024-04-27T03:19:56.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f99e4980 2 removed watcher, disabling cache 2024-04-27T03:19:56.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:170 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94633444598832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d94930 con 0x561190d87270 2024-04-27T03:19:56.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.1 v2:172.21.15.110:6800/1860462708 47 ==== osd_op_reply(170 notify.1 [watch unwatch cookie 94633444598832] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x561190d87270 2024-04-27T03:19:56.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f99e4980 20 remove_watcher() i=1 2024-04-27T03:19:56.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.126+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:171 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94633444602672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d94930 con 0x561190d8b540 2024-04-27T03:19:56.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.130+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 57 ==== osd_op_reply(171 notify.2 [watch unwatch cookie 94633444602672] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.130+0000 7fd2f99e4980 20 remove_watcher() i=2 2024-04-27T03:19:56.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.130+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:172 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94633444604976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d97330 con 0x7fd2e8036070 2024-04-27T03:19:56.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.130+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 37 ==== osd_op_reply(172 notify.3 [watch unwatch cookie 94633444604976] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.130+0000 7fd2f99e4980 20 remove_watcher() i=3 2024-04-27T03:19:56.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.130+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:173 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94633444606032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d94930 con 0x561190d8b540 2024-04-27T03:19:56.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f7d00640 1 -- 172.21.15.110:0/3619781954 <== osd.0 v2:172.21.15.77:6800/3496324319 58 ==== osd_op_reply(173 notify.4 [watch unwatch cookie 94633444606032] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e8033030 con 0x561190d8b540 2024-04-27T03:19:56.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f99e4980 20 remove_watcher() i=4 2024-04-27T03:19:56.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:174 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94633444597632] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d97330 con 0x561190d90680 2024-04-27T03:19:56.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(174 notify.5 [watch unwatch cookie 94633444597632] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f99e4980 20 remove_watcher() i=5 2024-04-27T03:19:56.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:175 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94633444607648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d95f80 con 0x561190d90680 2024-04-27T03:19:56.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f74ff640 1 -- 172.21.15.110:0/3619781954 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== osd_op_reply(175 notify.6 [watch unwatch cookie 94633444607648] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2f002c070 con 0x561190d90680 2024-04-27T03:19:56.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f99e4980 20 remove_watcher() i=6 2024-04-27T03:19:56.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.134+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:176 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94633444615712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x561190d97330 con 0x7fd2e8036070 2024-04-27T03:19:56.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 <== osd.2 v2:172.21.15.110:6808/3575017494 38 ==== osd_op_reply(176 notify.7 [watch unwatch cookie 94633444615712] v31'236 uv234 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fd2e0106040 con 0x7fd2e8036070 2024-04-27T03:19:56.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 20 remove_watcher() i=7 2024-04-27T03:19:56.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d8b540 msgr2=0x561190d8d950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d8b540 0x561190d8d950 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190d87270 msgr2=0x561190d89730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190d87270 0x561190d89730 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8036070 msgr2=0x7fd2e8030a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fd2e8036070 0x7fd2e8030a30 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561190d90680 msgr2=0x561190d92a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x561190d90680 0x561190d92a70 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c0034650 msgr2=0x7fd1c0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1c0034650 0x7fd1c0036b10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0024b60 tx=0x7fd2f0031000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 msgr2=0x561190d7bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f6cfe640 1 -- 172.21.15.110:0/3619781954 reap_dead start 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d7bc60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x561190d711c0 tx=0x7fd2e80498c0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 shutdown_connections 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/3619781954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190d74730 0x561190d7bc60 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 >> 172.21.15.110:0/3619781954 conn(0x561190d70ac0 msgr2=0x561190d75fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 shutdown_connections 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/3619781954 wait complete. 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d0034650 msgr2=0x7fd1d0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d0034650 0x7fd1d0036b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x561190d557c0 tx=0x7fd2f0024410 comp rx=0 tx=0).stop 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c58ab0 msgr2=0x561190c56f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c58ab0 0x561190c56f50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800aab0 tx=0x7fd2e800af70 comp rx=0 tx=0).stop 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 shutdown_connections 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1d0034650 0x7fd1d0036b10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1132157674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190c58ab0 0x561190c56f50 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 >> 172.21.15.110:0/1132157674 conn(0x561190d5a180 msgr2=0x561190d5ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 shutdown_connections 2024-04-27T03:19:56.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1132157674 wait complete. 2024-04-27T03:19:56.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d52590 msgr2=0x561190d54a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d52590 0x561190d54a50 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190c5c670 msgr2=0x561190d515a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190c5c670 0x561190d515a0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1dc0346a0 msgr2=0x7fd1dc036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1dc0346a0 0x7fd1dc036b60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd2e00ad000 tx=0x7fd2e00a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 msgr2=0x561190d4a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d4a500 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0000f90 tx=0x7fd2f0001260 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 shutdown_connections 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x561190d52590 0x561190d54a50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x561190c5c670 0x561190d515a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fd1dc0346a0 0x7fd1dc036b60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 --2- 172.21.15.110:0/1404465487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x561190bcc860 0x561190d4a500 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 >> 172.21.15.110:0/1404465487 conn(0x561190c52a30 msgr2=0x561190d3db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 shutdown_connections 2024-04-27T03:19:56.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.138+0000 7fd2f99e4980 1 -- 172.21.15.110:0/1404465487 wait complete. 2024-04-27T03:19:56.162 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo'] 2024-04-27T03:19:56.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'info', '--uid', 'foo'] 2024-04-27T03:19:56.162 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user info --uid foo 2024-04-27T03:19:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:56.244 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:56.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c7239a980 1 Processor -- start 2024-04-27T03:19:56.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c7239a980 1 -- start start 2024-04-27T03:19:56.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c7239a980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564ad5f6b9c0 0x564ad5f6ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c7239a980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x564ad5e240d0 con 0x564ad5f674b0 2024-04-27T03:19:56.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c7239a980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x564ad5e19430 con 0x564ad5f6b9c0 2024-04-27T03:19:56.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c6f6b4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564ad5f6b9c0 0x564ad5f6ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c6f6b4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x564ad5f6b9c0 0x564ad5f6ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51090/0 (socket says 172.21.15.110:51090) 2024-04-27T03:19:56.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c6feb5640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x564ad5f674b0 0x564ad5f67890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44482/0 (socket says 172.21.15.110:44482) 2024-04-27T03:19:56.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/3134186805 learned_addr learned my addr 172.21.15.110:0/3134186805 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.416 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/3134186805 >> v1:172.21.15.77:6789/0 conn(0x564ad5f674b0 legacy=0x564ad5f67890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c6eeb3640 1 -- 172.21.15.110:0/3134186805 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439369638 0 0) 0x564ad5e240d0 con 0x564ad5f674b0 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.234+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/3134186805 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad5e12510 con 0x564ad5f6b9c0 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c6f6b4640 1 --2- 172.21.15.110:0/3134186805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f6b9c0 0x564ad5f6ddb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400a5c0 tx=0x7f0c6401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f113bf2dca8d96b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c6eeb3640 1 -- 172.21.15.110:0/3134186805 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c640a6070 con 0x564ad5f6b9c0 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c6eeb3640 1 -- 172.21.15.110:0/3134186805 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c640af070 con 0x564ad5f6b9c0 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c6eeb3640 1 -- 172.21.15.110:0/3134186805 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c640aa5c0 con 0x564ad5f6b9c0 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c7239a980 1 -- 172.21.15.110:0/3134186805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f6b9c0 msgr2=0x564ad5f6ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3134186805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f6b9c0 0x564ad5f6ddb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400a5c0 tx=0x7f0c6401b340 comp rx=0 tx=0).stop 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c7239a980 1 -- 172.21.15.110:0/3134186805 shutdown_connections 2024-04-27T03:19:56.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3134186805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f6b9c0 0x564ad5f6ddb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c7239a980 1 -- 172.21.15.110:0/3134186805 >> 172.21.15.110:0/3134186805 conn(0x564ad5edab90 msgr2=0x564ad5edafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c7239a980 1 -- 172.21.15.110:0/3134186805 shutdown_connections 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.238+0000 7f0c7239a980 1 -- 172.21.15.110:0/3134186805 wait complete. 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.254+0000 7f0c7239a980 1 Processor -- start 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.254+0000 7f0c7239a980 1 -- start start 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.254+0000 7f0c7239a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6052380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.254+0000 7f0c7239a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ad5e682d0 con 0x564ad5f674b0 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6052380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6052380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51106/0 (socket says 172.21.15.110:51106) 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 -- 172.21.15.110:0/215255461 learned_addr learned my addr 172.21.15.110:0/215255461 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 -- 172.21.15.110:0/215255461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad5deeae0 con 0x564ad5f674b0 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/215255461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6052380 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000b250 tx=0x7f0c6000b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f42f4d42d0713 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0b5de0b640 1 -- 172.21.15.110:0/215255461 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c6000d640 con 0x564ad5f674b0 2024-04-27T03:19:56.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0b5de0b640 1 -- 172.21.15.110:0/215255461 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c60008070 con 0x564ad5f674b0 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0b5de0b640 1 -- 172.21.15.110:0/215255461 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c6000d990 con 0x564ad5f674b0 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 -- 172.21.15.110:0/215255461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 msgr2=0x564ad6052380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 --2- 172.21.15.110:0/215255461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6052380 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000b250 tx=0x7f0c6000b710 comp rx=0 tx=0).stop 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 -- 172.21.15.110:0/215255461 shutdown_connections 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 --2- 172.21.15.110:0/215255461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6052380 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 -- 172.21.15.110:0/215255461 >> 172.21.15.110:0/215255461 conn(0x564ad5edab90 msgr2=0x564ad604df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 -- 172.21.15.110:0/215255461 shutdown_connections 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 -- 172.21.15.110:0/215255461 wait complete. 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 Processor -- start 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 -- start start 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6059420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c7239a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ad5f70de0 con 0x564ad5f674b0 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6059420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6059420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51122/0 (socket says 172.21.15.110:51122) 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 -- 172.21.15.110:0/397539597 learned_addr learned my addr 172.21.15.110:0/397539597 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad6059960 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.258+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6059420 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0c60000c10 tx=0x7f0c60000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0b57fff640 1 -- 172.21.15.110:0/397539597 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c6001a7c0 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0b57fff640 1 -- 172.21.15.110:0/397539597 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c6001a960 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564ad6059bf0 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0b57fff640 1 -- 172.21.15.110:0/397539597 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c6001ac60 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad605c870 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0b57fff640 1 -- 172.21.15.110:0/397539597 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0c60017070 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0b57fff640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b580346f0 0x7f0b58036bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0b57fff640 1 -- 172.21.15.110:0/397539597 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f0c60015020 con 0x564ad5f674b0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c6f6b4640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b580346f0 0x7f0b58036bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad5f6b9c0 0x564ad6060350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6060970 con 0x564ad5f6b9c0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c706b6640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad5f6b9c0 0x564ad6060350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c6f6b4640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b580346f0 0x7f0b58036bb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0c640ae000 tx=0x7f0c640a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c706b6640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad5f6b9c0 0x564ad6060350 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c706b6640 1 -- 172.21.15.110:0/397539597 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c6800a4d0 con 0x564ad5f6b9c0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6061760 0x564ad6063b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6064190 con 0x564ad6061760 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.262+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6061760 0x564ad6063b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6061760 0x564ad6063b50 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c6feb5640 1 -- 172.21.15.110:0/397539597 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad6061760 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6065a40 con 0x564ad5f6b9c0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c706b6640 1 -- 172.21.15.110:0/397539597 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f0c6800a4d0 con 0x564ad5f6b9c0 2024-04-27T03:19:56.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6064c70 con 0x564ad6061760 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c6feb5640 1 -- 172.21.15.110:0/397539597 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad6061760 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6065060 con 0x564ad5f6b9c0 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c706b6640 1 -- 172.21.15.110:0/397539597 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f0c6800a4d0 con 0x564ad5f6b9c0 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c7239a980 1 Processor -- start 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c7239a980 1 -- start start 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c7239a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6074350 0x564ad6074770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.266+0000 7f0c7239a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ad5e6c310 con 0x564ad6074350 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6074350 0x564ad6074770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6074350 0x564ad6074770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51138/0 (socket says 172.21.15.110:51138) 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 -- 172.21.15.110:0/3217879817 learned_addr learned my addr 172.21.15.110:0/3217879817 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 -- 172.21.15.110:0/3217879817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad6074cb0 con 0x564ad6074350 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6074350 0x564ad6074770 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800dfc0 tx=0x7f0c680178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0b557fa640 1 -- 172.21.15.110:0/3217879817 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c680157d0 con 0x564ad6074350 2024-04-27T03:19:56.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0b557fa640 1 -- 172.21.15.110:0/3217879817 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c68015970 con 0x564ad6074350 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564ad6075750 con 0x564ad6074350 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0b557fa640 1 -- 172.21.15.110:0/3217879817 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c68015c70 con 0x564ad6074350 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad6075c20 con 0x564ad6074350 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0b557fa640 1 -- 172.21.15.110:0/3217879817 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0c68029440 con 0x564ad6074350 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0b557fa640 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b48034650 0x7f0b48036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0b557fa640 1 -- 172.21.15.110:0/3217879817 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f0c6800e040 con 0x564ad6074350 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b48034650 0x7f0b48036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c7239a980 1 Processor -- start 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c7239a980 1 -- start start 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b48034650 0x7f0b48036b10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x564ad6064790 tx=0x7f0c600014a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c7239a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6081920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c7239a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ad6052ad0 con 0x564ad6081500 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6081920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6081920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51140/0 (socket says 172.21.15.110:51140) 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.270+0000 7f0c706b6640 1 -- 172.21.15.110:0/3216103873 learned_addr learned my addr 172.21.15.110:0/3216103873 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c706b6640 1 -- 172.21.15.110:0/3216103873 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad6081ef0 con 0x564ad6081500 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c706b6640 1 --2- 172.21.15.110:0/3216103873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6081920 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0c68015fa0 tx=0x7f0c68015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dd3137a2235e76 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0b4effd640 1 -- 172.21.15.110:0/3216103873 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c680113a0 con 0x564ad6081500 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0b4effd640 1 -- 172.21.15.110:0/3216103873 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c68011540 con 0x564ad6081500 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0b4effd640 1 -- 172.21.15.110:0/3216103873 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c680284d0 con 0x564ad6081500 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 -- 172.21.15.110:0/3216103873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 msgr2=0x564ad6081920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3216103873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6081920 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0c68015fa0 tx=0x7f0c68015fd0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 -- 172.21.15.110:0/3216103873 shutdown_connections 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3216103873 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6081920 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 -- 172.21.15.110:0/3216103873 >> 172.21.15.110:0/3216103873 conn(0x564ad607d890 msgr2=0x564ad607b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 -- 172.21.15.110:0/3216103873 shutdown_connections 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 -- 172.21.15.110:0/3216103873 wait complete. 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 Processor -- start 2024-04-27T03:19:56.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 -- start start 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6088a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c7239a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x564ad6082660 con 0x564ad6081500 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c706b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6088a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c706b6640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6088a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51150/0 (socket says 172.21.15.110:51150) 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 learned_addr learned my addr 172.21.15.110:0/1416584551 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad6088f90 con 0x564ad6081500 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.274+0000 7f0c706b6640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6088a50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0c6803e040 tx=0x7f0c6804a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0b4d7fa640 1 -- 172.21.15.110:0/1416584551 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c68038070 con 0x564ad6081500 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564ad6089220 con 0x564ad6081500 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0b4d7fa640 1 -- 172.21.15.110:0/1416584551 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c6804c410 con 0x564ad6081500 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad608bf00 con 0x564ad6081500 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0b4d7fa640 1 -- 172.21.15.110:0/1416584551 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c68013a50 con 0x564ad6081500 2024-04-27T03:19:56.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0b4d7fa640 1 -- 172.21.15.110:0/1416584551 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0c68013bf0 con 0x564ad6081500 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0b4d7fa640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b38034650 0x7f0b38036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0b4d7fa640 1 -- 172.21.15.110:0/1416584551 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f0c68030070 con 0x564ad6081500 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b38034650 0x7f0b38036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad6093e70 0x564ad6096300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6096a50 con 0x564ad6093e70 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c6f6b4640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad6093e70 0x564ad6096300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b38034650 0x7f0b38036b10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e560 tx=0x7f0c6004a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c6f6b4640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad6093e70 0x564ad6096300 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 realm 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6096e80 con 0x564ad6093e70 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad6097720 con 0x564ad6093e70 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6098140 0x564ad609a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad609acc0 con 0x564ad6098140 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.278+0000 7f0c706b6640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6098140 0x564ad609a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c706b6640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6098140 0x564ad609a550 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c7239a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:56.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad609b560 con 0x564ad6093e70 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c7239a980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c7239a980 20 searching for the correct realm 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564ad609d2d0 0x564ad609f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad609d2d0 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564ad609d2d0 0x564ad609f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6feb5640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564ad609d2d0 0x564ad609f6c0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c706b6640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c68014030 0x7f0c68028820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x7f0c68014030 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6f6b4640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c68014030 0x7f0c68028820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6f6b4640 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c68014030 0x7f0c68028820 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.282+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad609d2d0 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x7f0c68014030 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x7f0c68014030 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.286+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad609d2d0 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad609d2d0 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x7f0c68014030 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad609d2d0 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x7f0c68014030 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6093e70 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x7f0c68014030 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.290+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad609d2d0 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x7f0c68014030 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad609d2d0 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x564ad609fcd0 con 0x564ad6098140 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad609bee0 con 0x564ad6093e70 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.294+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad609c360 con 0x564ad6098140 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad609c7c0 con 0x564ad6093e70 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 zone default found 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 4 Realm: () 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:56.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 zonegroup default 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a05e0 con 0x7f0c68014030 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a2390 con 0x564ad609d2d0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a2860 con 0x564ad609d2d0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 started sync module instance, tier type = 2024-04-27T03:19:56.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.298+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a1800 con 0x564ad6098140 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94879418555648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a1800 con 0x564ad6098140 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a33a0 con 0x564ad6093e70 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94879418560144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a3810 con 0x564ad6093e70 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a4280 con 0x564ad6098140 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94879418555648] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94879418560144] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94879418563920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a3360 con 0x564ad6098140 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.302+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a4b20 con 0x7f0c68014030 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94879418566256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a3360 con 0x7f0c68014030 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a55a0 con 0x564ad6098140 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94879418563920] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94879418566256] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94879418567312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a50b0 con 0x564ad6098140 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.306+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a5e40 con 0x564ad609d2d0 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94879418571152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a50b0 con 0x564ad609d2d0 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a6bc0 con 0x564ad609d2d0 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94879418567312] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94879418571152] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94879418573280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a6fb0 con 0x564ad609d2d0 2024-04-27T03:19:56.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a7840 con 0x7f0c68014030 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94879418573280] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'237 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94879418574336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a7460 con 0x7f0c68014030 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 20 add_watcher() i=0 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 20 add_watcher() i=1 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 20 add_watcher() i=2 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 20 add_watcher() i=3 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 20 add_watcher() i=4 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 20 add_watcher() i=5 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.310+0000 7f0c7239a980 20 add_watcher() i=6 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94879418574336] v31'238 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 20 add_watcher() i=7 2024-04-27T03:19:56.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad60aacf0 con 0x564ad6093e70 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94879418577280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60a8210 con 0x564ad6093e70 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94879418577280] v31'243 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0b25fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 5 note: GC not initialized 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0b24fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:56.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.314+0000 7f0c7239a980 20 Filter name: none 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0a9e7fc640 20 reqs_thread_entry: start 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0c7239a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.18 8:1e6ca5ed:users.uid::foo:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdff0 con 0x564ad6093e70 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 foo [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0c7239a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:could not fetch user info: no user info saved 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0b25fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94879418577280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x564ad6093e70 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 data_loggenerations_metadata [watch unwatch cookie 94879418577280] v31'244 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.318+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94879418555648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x564ad6098140 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.322+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(64 notify.0 [watch unwatch cookie 94879418555648] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.322+0000 7f0c7239a980 20 remove_watcher() i=0 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.322+0000 7f0c7239a980 2 removed watcher, disabling cache 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.322+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94879418560144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x564ad6093e70 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.322+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(65 notify.1 [watch unwatch cookie 94879418560144] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x564ad6093e70 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.322+0000 7f0c7239a980 20 remove_watcher() i=1 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.322+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94879418563920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x564ad6098140 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.326+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 notify.2 [watch unwatch cookie 94879418563920] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.326+0000 7f0c7239a980 20 remove_watcher() i=2 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.326+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:67 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94879418566256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x7f0c68014030 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.326+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(67 notify.3 [watch unwatch cookie 94879418566256] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.326+0000 7f0c7239a980 20 remove_watcher() i=3 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.326+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94879418567312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x564ad6098140 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c706b6640 1 -- 172.21.15.110:0/1416584551 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 notify.4 [watch unwatch cookie 94879418567312] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c68026020 con 0x564ad6098140 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c7239a980 20 remove_watcher() i=4 2024-04-27T03:19:56.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94879418571152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x564ad609d2d0 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(69 notify.5 [watch unwatch cookie 94879418571152] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c7239a980 20 remove_watcher() i=5 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:70 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94879418573280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x564ad609d2d0 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c6feb5640 1 -- 172.21.15.110:0/1416584551 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(70 notify.6 [watch unwatch cookie 94879418573280] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c6002e070 con 0x564ad609d2d0 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c7239a980 20 remove_watcher() i=6 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:71 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94879418574336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x564ad60cdfb0 con 0x7f0c68014030 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(71 notify.7 [watch unwatch cookie 94879418574336] v31'239 uv237 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c640aa960 con 0x7f0c68014030 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.330+0000 7f0c7239a980 20 remove_watcher() i=7 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6098140 msgr2=0x564ad609a550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6098140 0x564ad609a550 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad6093e70 msgr2=0x564ad6096300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad6093e70 0x564ad6096300 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c68014030 msgr2=0x7f0c68028820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c68014030 0x7f0c68028820 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564ad609d2d0 msgr2=0x564ad609f6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x564ad609d2d0 0x564ad609f6c0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b38034650 msgr2=0x7f0b38036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b38034650 0x7f0b38036b10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000e560 tx=0x7f0c6004a000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 msgr2=0x564ad6088a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c6f6b4640 1 -- 172.21.15.110:0/1416584551 reap_dead start 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6088a50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0c6803e040 tx=0x7f0c6804a880 comp rx=0 tx=0).stop 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 shutdown_connections 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/1416584551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6081500 0x564ad6088a50 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 >> 172.21.15.110:0/1416584551 conn(0x564ad607d890 msgr2=0x564ad6082d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 shutdown_connections 2024-04-27T03:19:56.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/1416584551 wait complete. 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b48034650 msgr2=0x7f0b48036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b48034650 0x7f0b48036b10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x564ad6064790 tx=0x7f0c600014a0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6074350 msgr2=0x564ad6074770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6074350 0x564ad6074770 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800dfc0 tx=0x7f0c680178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 shutdown_connections 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b48034650 0x7f0b48036b10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/3217879817 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad6074350 0x564ad6074770 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 >> 172.21.15.110:0/3217879817 conn(0x564ad6069160 msgr2=0x564ad6069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 shutdown_connections 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/3217879817 wait complete. 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6061760 msgr2=0x564ad6063b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6061760 0x564ad6063b50 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad5f6b9c0 msgr2=0x564ad6060350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad5f6b9c0 0x564ad6060350 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b580346f0 msgr2=0x7f0b58036bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b580346f0 0x7f0b58036bb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0c640ae000 tx=0x7f0c640a6000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 msgr2=0x564ad6059420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6059420 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0c60000c10 tx=0x7f0c60000ee0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 shutdown_connections 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x564ad6061760 0x564ad6063b50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x564ad5f6b9c0 0x564ad6060350 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b580346f0 0x7f0b58036bb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 --2- 172.21.15.110:0/397539597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x564ad5f674b0 0x564ad6059420 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 >> 172.21.15.110:0/397539597 conn(0x564ad5edab90 msgr2=0x564ad604ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 shutdown_connections 2024-04-27T03:19:56.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.334+0000 7f0c7239a980 1 -- 172.21.15.110:0/397539597 wait complete. 2024-04-27T03:19:56.446 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'get', '--rgw-zone', 'default'] 2024-04-27T03:19:56.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'get', '--rgw-zone', 'default'] 2024-04-27T03:19:56.446 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone get --rgw-zone default 2024-04-27T03:19:56.496 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:56.496 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:56.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c413c980 1 Processor -- start 2024-04-27T03:19:56.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c413c980 1 -- start start 2024-04-27T03:19:56.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c413c980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ceb5fa2a10 0x55ceb5fa4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c413c980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55ceb5e5b0d0 con 0x55ceb5f9e550 2024-04-27T03:19:56.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c413c980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55ceb5e50430 con 0x55ceb5fa2a10 2024-04-27T03:19:56.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c1456640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ceb5fa2a10 0x55ceb5fa4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c1456640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ceb5fa2a10 0x55ceb5fa4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51164/0 (socket says 172.21.15.110:51164) 2024-04-27T03:19:56.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c1c57640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55ceb5f9e550 0x55ceb5f9e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44496/0 (socket says 172.21.15.110:44496) 2024-04-27T03:19:56.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c1456640 1 -- 172.21.15.110:0/51646153 learned_addr learned my addr 172.21.15.110:0/51646153 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c1456640 1 -- 172.21.15.110:0/51646153 >> v1:172.21.15.77:6789/0 conn(0x55ceb5f9e550 legacy=0x55ceb5f9e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.486+0000 7f12c1456640 1 -- 172.21.15.110:0/51646153 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb5e49510 con 0x55ceb5fa2a10 2024-04-27T03:19:56.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c1456640 1 --2- 172.21.15.110:0/51646153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb5fa4e00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f12b800a5c0 tx=0x7f12b801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c34b3049ed9ba38 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c0c55640 1 -- 172.21.15.110:0/51646153 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b80a6070 con 0x55ceb5fa2a10 2024-04-27T03:19:56.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c0c55640 1 -- 172.21.15.110:0/51646153 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b80af070 con 0x55ceb5fa2a10 2024-04-27T03:19:56.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c0c55640 1 -- 172.21.15.110:0/51646153 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b80aa5c0 con 0x55ceb5fa2a10 2024-04-27T03:19:56.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c413c980 1 -- 172.21.15.110:0/51646153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 msgr2=0x55ceb5fa4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c413c980 1 --2- 172.21.15.110:0/51646153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb5fa4e00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f12b800a5c0 tx=0x7f12b801b340 comp rx=0 tx=0).stop 2024-04-27T03:19:56.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c413c980 1 -- 172.21.15.110:0/51646153 shutdown_connections 2024-04-27T03:19:56.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c413c980 1 --2- 172.21.15.110:0/51646153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb5fa4e00 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c413c980 1 -- 172.21.15.110:0/51646153 >> 172.21.15.110:0/51646153 conn(0x55ceb5f11d20 msgr2=0x55ceb5f9ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c413c980 1 -- 172.21.15.110:0/51646153 shutdown_connections 2024-04-27T03:19:56.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.490+0000 7f12c413c980 1 -- 172.21.15.110:0/51646153 wait complete. 2024-04-27T03:19:56.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.506+0000 7f12c413c980 1 Processor -- start 2024-04-27T03:19:56.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.506+0000 7f12c413c980 1 -- start start 2024-04-27T03:19:56.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.506+0000 7f12c413c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5f9e550 0x55ceb6089650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.506+0000 7f12c413c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ceb5e9f2d0 con 0x55ceb5f9e550 2024-04-27T03:19:56.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.506+0000 7f12c1c57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5f9e550 0x55ceb6089650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.506+0000 7f12c1c57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5f9e550 0x55ceb6089650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51174/0 (socket says 172.21.15.110:51174) 2024-04-27T03:19:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.506+0000 7f12c1c57640 1 -- 172.21.15.110:0/1995235321 learned_addr learned my addr 172.21.15.110:0/1995235321 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c1c57640 1 -- 172.21.15.110:0/1995235321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb5e25ae0 con 0x55ceb5f9e550 2024-04-27T03:19:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c1c57640 1 --2- 172.21.15.110:0/1995235321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5f9e550 0x55ceb6089650 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f12b400b240 tx=0x7f12b400b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a032d332d0c6959c server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f11afe07640 1 -- 172.21.15.110:0/1995235321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b400e700 con 0x55ceb5f9e550 2024-04-27T03:19:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f11afe07640 1 -- 172.21.15.110:0/1995235321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b400e8a0 con 0x55ceb5f9e550 2024-04-27T03:19:56.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f11afe07640 1 -- 172.21.15.110:0/1995235321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b400eb80 con 0x55ceb5f9e550 2024-04-27T03:19:56.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 -- 172.21.15.110:0/1995235321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5f9e550 msgr2=0x55ceb6089650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 --2- 172.21.15.110:0/1995235321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5f9e550 0x55ceb6089650 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f12b400b240 tx=0x7f12b400b700 comp rx=0 tx=0).stop 2024-04-27T03:19:56.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 -- 172.21.15.110:0/1995235321 shutdown_connections 2024-04-27T03:19:56.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 --2- 172.21.15.110:0/1995235321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5f9e550 0x55ceb6089650 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 -- 172.21.15.110:0/1995235321 >> 172.21.15.110:0/1995235321 conn(0x55ceb5f11d20 msgr2=0x55ceb60851e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 -- 172.21.15.110:0/1995235321 shutdown_connections 2024-04-27T03:19:56.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 -- 172.21.15.110:0/1995235321 wait complete. 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 Processor -- start 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 -- start start 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb60908b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c413c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ceb5fa7e30 con 0x55ceb5fa2a10 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c1c57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb60908b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c1c57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb60908b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51176/0 (socket says 172.21.15.110:51176) 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c1c57640 1 -- 172.21.15.110:0/2673693843 learned_addr learned my addr 172.21.15.110:0/2673693843 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c1c57640 1 -- 172.21.15.110:0/2673693843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb6090df0 con 0x55ceb5fa2a10 2024-04-27T03:19:56.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.510+0000 7f12c1c57640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb60908b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f12b4000c10 tx=0x7f12b4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11ae604640 1 -- 172.21.15.110:0/2673693843 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b4001530 con 0x55ceb5fa2a10 2024-04-27T03:19:56.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11ae604640 1 -- 172.21.15.110:0/2673693843 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b400f9f0 con 0x55ceb5fa2a10 2024-04-27T03:19:56.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ceb6091080 con 0x55ceb5fa2a10 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11ae604640 1 -- 172.21.15.110:0/2673693843 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b400fcf0 con 0x55ceb5fa2a10 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb6093d00 con 0x55ceb5fa2a10 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 Processor -- start 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- start start 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb609e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ceb5ea3310 con 0x55ceb609e500 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11ae604640 1 -- 172.21.15.110:0/2673693843 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f12b4017070 con 0x55ceb5fa2a10 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c2458640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb609e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c2458640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb609e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51178/0 (socket says 172.21.15.110:51178) 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c2458640 1 -- 172.21.15.110:0/289563189 learned_addr learned my addr 172.21.15.110:0/289563189 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11ae604640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f11a8034650 0x7f11a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11ae604640 1 -- 172.21.15.110:0/2673693843 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f12b4015020 con 0x55ceb5fa2a10 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c1c57640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f11a8034650 0x7f11a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c2458640 1 -- 172.21.15.110:0/289563189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb609ee60 con 0x55ceb609e500 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c1c57640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f11a8034650 0x7f11a8036b10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f12b4001500 tx=0x7f12b400ff70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c2458640 1 --2- 172.21.15.110:0/289563189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb609e920 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00a430 tx=0x7f12bc00a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d60e3137236fab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11a7fff640 1 -- 172.21.15.110:0/289563189 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc00f6e0 con 0x55ceb609e500 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11a7fff640 1 -- 172.21.15.110:0/289563189 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12bc015070 con 0x55ceb609e500 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f11a7fff640 1 -- 172.21.15.110:0/289563189 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc00fa10 con 0x55ceb609e500 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- 172.21.15.110:0/289563189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 msgr2=0x55ceb609e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 --2- 172.21.15.110:0/289563189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb609e920 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f12bc00a430 tx=0x7f12bc00a8f0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- 172.21.15.110:0/289563189 shutdown_connections 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 --2- 172.21.15.110:0/289563189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb609e920 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- 172.21.15.110:0/289563189 >> 172.21.15.110:0/289563189 conn(0x55ceb6099120 msgr2=0x55ceb609a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- 172.21.15.110:0/289563189 shutdown_connections 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- 172.21.15.110:0/289563189 wait complete. 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 Processor -- start 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- start start 2024-04-27T03:19:56.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb60a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.514+0000 7f12c413c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ceb609f3a0 con 0x55ceb609e500 2024-04-27T03:19:56.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c2458640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb60a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c2458640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb60a53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51190/0 (socket says 172.21.15.110:51190) 2024-04-27T03:19:56.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c2458640 1 -- 172.21.15.110:0/352430168 learned_addr learned my addr 172.21.15.110:0/352430168 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c2458640 1 -- 172.21.15.110:0/352430168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb60a58f0 con 0x55ceb609e500 2024-04-27T03:19:56.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c2458640 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb60a53b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x55ceb609ac50 tx=0x7f12bc003380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f11a67fc640 1 -- 172.21.15.110:0/352430168 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc00fe00 con 0x55ceb609e500 2024-04-27T03:19:56.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ceb60a5b80 con 0x55ceb609e500 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f11a67fc640 1 -- 172.21.15.110:0/352430168 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12bc010a20 con 0x55ceb609e500 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f11a67fc640 1 -- 172.21.15.110:0/352430168 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc010dc0 con 0x55ceb609e500 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb60a8860 con 0x55ceb609e500 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f11a67fc640 1 -- 172.21.15.110:0/352430168 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f12bc01a480 con 0x55ceb609e500 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f11a67fc640 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f119c034650 0x7f119c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 1 Processor -- start 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f11a67fc640 1 -- 172.21.15.110:0/352430168 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f12bc04a5e0 con 0x55ceb609e500 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c2458640 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f119c034650 0x7f119c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 1 -- start start 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb60bf280 0x55ceb60bf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c413c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ceb6089da0 con 0x55ceb60bf280 2024-04-27T03:19:56.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c2458640 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f119c034650 0x7f119c036b10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f12bc02efd0 tx=0x7f12bc01be90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c1c57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb60bf280 0x55ceb60bf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c1c57640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb60bf280 0x55ceb60bf6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51202/0 (socket says 172.21.15.110:51202) 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.518+0000 7f12c1c57640 1 -- 172.21.15.110:0/138330553 learned_addr learned my addr 172.21.15.110:0/138330553 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c1c57640 1 -- 172.21.15.110:0/138330553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ceb60bfbe0 con 0x55ceb60bf280 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c1c57640 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb60bf280 0x55ceb60bf6a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f12b4048cf0 tx=0x7f12b4013500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f1186fe5640 1 -- 172.21.15.110:0/138330553 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b401de60 con 0x55ceb60bf280 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f1186fe5640 1 -- 172.21.15.110:0/138330553 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12b4013630 con 0x55ceb60bf280 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ceb60c0680 con 0x55ceb60bf280 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f1186fe5640 1 -- 172.21.15.110:0/138330553 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12b4013c50 con 0x55ceb60bf280 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ceb60c0680 con 0x55ceb60bf280 2024-04-27T03:19:56.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f1186fe5640 1 -- 172.21.15.110:0/138330553 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f12b404ab40 con 0x55ceb60bf280 2024-04-27T03:19:56.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f1186fe5640 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f10f8034650 0x7f10f8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f1186fe5640 1 -- 172.21.15.110:0/138330553 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f12b4038070 con 0x55ceb60bf280 2024-04-27T03:19:56.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c1456640 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f10f8034650 0x7f10f8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f1184fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:56.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ceb60c4d60 0x55ceb60c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:1 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ceb60c77d0 con 0x55ceb60c4d60 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c1456640 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f10f8034650 0x7f10f8036b10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f12b80ae000 tx=0x7f12b80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c2458640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ceb60c4d60 0x55ceb60c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.522+0000 7f12c2458640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ceb60c4d60 0x55ceb60c7150 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c2458640 1 -- 172.21.15.110:0/2673693843 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(1 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f12bc00e070 con 0x55ceb60c4d60 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ceb60c95f0 0x55ceb60cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ceb60cc0b0 con 0x55ceb60c95f0 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c1c57640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ceb60c95f0 0x55ceb60cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c1c57640 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ceb60c95f0 0x55ceb60cb9e0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c1c57640 1 -- 172.21.15.110:0/2673693843 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(2 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f12b4046020 con 0x55ceb60c95f0 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f119c034650 msgr2=0x7f119c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f119c034650 0x7f119c036b10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f12bc02efd0 tx=0x7f12bc01be90 comp rx=0 tx=0).stop 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 msgr2=0x55ceb60a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb60a53b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x55ceb609ac50 tx=0x7f12bc003380 comp rx=0 tx=0).stop 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 shutdown_connections 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f119c034650 0x7f119c036b10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/352430168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb609e500 0x55ceb60a53b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 >> 172.21.15.110:0/352430168 conn(0x55ceb6099120 msgr2=0x55ceb609a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 shutdown_connections 2024-04-27T03:19:56.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/352430168 wait complete. 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f10f8034650 msgr2=0x7f10f8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f10f8034650 0x7f10f8036b10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f12b80ae000 tx=0x7f12b80a6000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb60bf280 msgr2=0x55ceb60bf6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb60bf280 0x55ceb60bf6a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f12b4048cf0 tx=0x7f12b4013500 comp rx=0 tx=0).stop 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 shutdown_connections 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f10f8034650 0x7f10f8036b10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/138330553 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb60bf280 0x55ceb60bf6a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 >> 172.21.15.110:0/138330553 conn(0x55ceb60b3e60 msgr2=0x55ceb60b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 shutdown_connections 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/138330553 wait complete. 2024-04-27T03:19:56.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ceb60c4d60 msgr2=0x55ceb60c7150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ceb60c4d60 0x55ceb60c7150 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ceb60c95f0 msgr2=0x55ceb60cb9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ceb60c95f0 0x55ceb60cb9e0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f11a8034650 msgr2=0x7f11a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f11a8034650 0x7f11a8036b10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f12b4001500 tx=0x7f12b400ff70 comp rx=0 tx=0).stop 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 msgr2=0x55ceb60908b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb60908b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f12b4000c10 tx=0x7f12b4000ee0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 shutdown_connections 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ceb60c95f0 0x55ceb60cb9e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ceb60c4d60 0x55ceb60c7150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f11a8034650 0x7f11a8036b10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 --2- 172.21.15.110:0/2673693843 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ceb5fa2a10 0x55ceb60908b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 >> 172.21.15.110:0/2673693843 conn(0x55ceb5f11d20 msgr2=0x55ceb6083cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 shutdown_connections 2024-04-27T03:19:56.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.526+0000 7f12c413c980 1 -- 172.21.15.110:0/2673693843 wait complete. 2024-04-27T03:19:56.549 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"27a51273-3ccc-49e2-8682-1122abc3d898","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":""} 2024-04-27T03:19:56.549 DEBUG:tasks.util.rgw: json result: {'id': '27a51273-3ccc-49e2-8682-1122abc3d898', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': ''} 2024-04-27T03:19:56.550 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'set'] 2024-04-27T03:19:56.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'set'] 2024-04-27T03:19:56.550 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone set 2024-04-27T03:19:56.619 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:56.619 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:56.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabe1fbf980 1 Processor -- start 2024-04-27T03:19:56.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabe1fbf980 1 -- start start 2024-04-27T03:19:56.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabe1fbf980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5592579372c0 0x5592579376a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabe1fbf980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5592577f60d0 con 0x559257937be0 2024-04-27T03:19:56.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabe1fbf980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5592577eb430 con 0x5592579372c0 2024-04-27T03:19:56.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabdfada640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5592579372c0 0x5592579376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabdf2d9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x559257937be0 0x559257938550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44500/0 (socket says 172.21.15.110:44500) 2024-04-27T03:19:56.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2390342954 learned_addr learned my addr 172.21.15.110:0/2390342954 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:56.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabdfada640 1 --2- 172.21.15.110:0/2390342954 >> v2:172.21.15.77:3300/0 conn(0x5592579372c0 0x5592579376a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51212/0 (socket says 172.21.15.110:51212) 2024-04-27T03:19:56.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabdfada640 1 -- 172.21.15.110:0/2390342954 >> v1:172.21.15.77:6789/0 conn(0x559257937be0 legacy=0x559257938550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.610+0000 7fabdfada640 1 -- 172.21.15.110:0/2390342954 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592577e4510 con 0x5592579372c0 2024-04-27T03:19:56.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabdfada640 1 --2- 172.21.15.110:0/2390342954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592579376a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fabd800a420 tx=0x7fabd801afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7e0be206857bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabdead8640 1 -- 172.21.15.110:0/2390342954 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80af070 con 0x5592579372c0 2024-04-27T03:19:56.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabdead8640 1 -- 172.21.15.110:0/2390342954 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd80aa530 con 0x5592579372c0 2024-04-27T03:19:56.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabdead8640 1 -- 172.21.15.110:0/2390342954 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80aa850 con 0x5592579372c0 2024-04-27T03:19:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2390342954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 msgr2=0x5592579376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2390342954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592579376a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fabd800a420 tx=0x7fabd801afd0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2390342954 shutdown_connections 2024-04-27T03:19:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2390342954 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592579376a0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2390342954 >> 172.21.15.110:0/2390342954 conn(0x559257932b90 msgr2=0x559257934fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2390342954 shutdown_connections 2024-04-27T03:19:56.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.614+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2390342954 wait complete. 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 Processor -- start 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- start start 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592578ace30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55925783a2d0 con 0x5592579372c0 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabdfada640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592578ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabdfada640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592578ace30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51224/0 (socket says 172.21.15.110:51224) 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabdfada640 1 -- 172.21.15.110:0/768249972 learned_addr learned my addr 172.21.15.110:0/768249972 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabdfada640 1 -- 172.21.15.110:0/768249972 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592577c0ae0 con 0x5592579372c0 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabdfada640 1 --2- 172.21.15.110:0/768249972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592578ace30 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fabd800a420 tx=0x7fabd80033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6a066a17f0d6af server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7faacde0b640 1 -- 172.21.15.110:0/768249972 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80af050 con 0x5592579372c0 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7faacde0b640 1 -- 172.21.15.110:0/768249972 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd80b0040 con 0x5592579372c0 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7faacde0b640 1 -- 172.21.15.110:0/768249972 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80aac60 con 0x5592579372c0 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- 172.21.15.110:0/768249972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 msgr2=0x5592578ace30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/768249972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592578ace30 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fabd800a420 tx=0x7fabd80033b0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- 172.21.15.110:0/768249972 shutdown_connections 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/768249972 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x5592578ace30 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- 172.21.15.110:0/768249972 >> 172.21.15.110:0/768249972 conn(0x559257932b90 msgr2=0x559257a1e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- 172.21.15.110:0/768249972 shutdown_connections 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- 172.21.15.110:0/768249972 wait complete. 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 Processor -- start 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.630+0000 7fabe1fbf980 1 -- start start 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x559257a2bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5592579412c0 con 0x5592579372c0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabdfada640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x559257a2bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabdfada640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x559257a2bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51236/0 (socket says 172.21.15.110:51236) 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 learned_addr learned my addr 172.21.15.110:0/2283488160 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559257a2c160 con 0x5592579372c0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabdfada640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x559257a2bc20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fabd801b010 tx=0x7fabd8019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7faac7fff640 1 -- 172.21.15.110:0/2283488160 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80c5030 con 0x5592579372c0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7faac7fff640 1 -- 172.21.15.110:0/2283488160 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd80c15b0 con 0x5592579372c0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559257a2c3f0 con 0x5592579372c0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7faac7fff640 1 -- 172.21.15.110:0/2283488160 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80c1890 con 0x5592579372c0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559257a2efd0 con 0x5592579372c0 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 Processor -- start 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 -- start start 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a393f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe1fbf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55925783e310 con 0x559257a38fd0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7faac7fff640 1 -- 172.21.15.110:0/2283488160 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fabd80c8070 con 0x5592579372c0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe02db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7faac7fff640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faac8034650 0x7faac8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe02db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a393f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51252/0 (socket says 172.21.15.110:51252) 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe02db640 1 -- 172.21.15.110:0/3384662584 learned_addr learned my addr 172.21.15.110:0/3384662584 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7faac7fff640 1 -- 172.21.15.110:0/2283488160 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fabd80c6070 con 0x5592579372c0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabdfada640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faac8034650 0x7faac8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe02db640 1 -- 172.21.15.110:0/3384662584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559257a39930 con 0x559257a38fd0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabdfada640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faac8034650 0x7faac8036b10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fabd8005790 tx=0x7fabd80056a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.634+0000 7fabe02db640 1 --2- 172.21.15.110:0/3384662584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a393f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fabd000a440 tx=0x7fabd000a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8ac1d2014b3e1a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7faac5ffb640 1 -- 172.21.15.110:0/3384662584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd0008850 con 0x559257a38fd0 2024-04-27T03:19:56.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7faac5ffb640 1 -- 172.21.15.110:0/3384662584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd0008a10 con 0x559257a38fd0 2024-04-27T03:19:56.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7faac5ffb640 1 -- 172.21.15.110:0/3384662584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd0008d30 con 0x559257a38fd0 2024-04-27T03:19:56.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- 172.21.15.110:0/3384662584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 msgr2=0x559257a393f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/3384662584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a393f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fabd000a440 tx=0x7fabd000a900 comp rx=0 tx=0).stop 2024-04-27T03:19:56.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- 172.21.15.110:0/3384662584 shutdown_connections 2024-04-27T03:19:56.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/3384662584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a393f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- 172.21.15.110:0/3384662584 >> 172.21.15.110:0/3384662584 conn(0x559257937be0 msgr2=0x559257a34440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- 172.21.15.110:0/3384662584 shutdown_connections 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- 172.21.15.110:0/3384662584 wait complete. 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 Processor -- start 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- start start 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a40260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559257a3a2f0 con 0x559257a38fd0 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe02db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a40260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe02db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a40260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51262/0 (socket says 172.21.15.110:51262) 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe02db640 1 -- 172.21.15.110:0/4232075837 learned_addr learned my addr 172.21.15.110:0/4232075837 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe02db640 1 -- 172.21.15.110:0/4232075837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559257a407a0 con 0x559257a38fd0 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe02db640 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a40260 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x559257a39db0 tx=0x7fabd0003000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7faabffff640 1 -- 172.21.15.110:0/4232075837 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd0013a70 con 0x559257a38fd0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559257a40a30 con 0x559257a38fd0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7faabffff640 1 -- 172.21.15.110:0/4232075837 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd0023070 con 0x559257a38fd0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7faabffff640 1 -- 172.21.15.110:0/4232075837 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd0013d40 con 0x559257a38fd0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559257a43710 con 0x559257a38fd0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.638+0000 7faabffff640 1 -- 172.21.15.110:0/4232075837 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fabd0028460 con 0x559257a38fd0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faabffff640 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faab8034650 0x7faab8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faabffff640 1 -- 172.21.15.110:0/4232075837 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fabd004a130 con 0x559257a38fd0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdf2d9640 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faab8034650 0x7faab8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 1 Processor -- start 2024-04-27T03:19:56.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 1 -- start start 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdf2d9640 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faab8034650 0x7faab8036b10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x559257a35080 tx=0x7fabd4001280 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a5a180 0x559257a5a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5592578ad4c0 con 0x559257a5a180 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdfada640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a5a180 0x559257a5a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdfada640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a5a180 0x559257a5a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51272/0 (socket says 172.21.15.110:51272) 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdfada640 1 -- 172.21.15.110:0/222015356 learned_addr learned my addr 172.21.15.110:0/222015356 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdfada640 1 -- 172.21.15.110:0/222015356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559257a5aae0 con 0x559257a5a180 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdfada640 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a5a180 0x559257a5a5a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fabd80e3910 tx=0x7fabd80bfe90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faa47fff640 1 -- 172.21.15.110:0/222015356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80c93b0 con 0x559257a5a180 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559257a5b580 con 0x559257a5a180 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faa47fff640 1 -- 172.21.15.110:0/222015356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd80ea070 con 0x559257a5a180 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faa47fff640 1 -- 172.21.15.110:0/222015356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fabd80c1a70 con 0x559257a5a180 2024-04-27T03:19:56.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559257a5b580 con 0x559257a5a180 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faa47fff640 1 -- 172.21.15.110:0/222015356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fabd80c1c10 con 0x559257a5a180 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faa47fff640 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faa20034650 0x7faa20036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7faa47fff640 1 -- 172.21.15.110:0/222015356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fabd80bb070 con 0x559257a5a180 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.642+0000 7fabdf2d9640 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faa20034650 0x7faa20036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7faa45ffb640 20 reqs_thread_entry: start 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559257a5fba0 0x559257a61fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559257a62630 con 0x559257a5fba0 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabdf2d9640 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faa20034650 0x7faa20036b10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fabd4004180 tx=0x7fabd400f000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe02db640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559257a5fba0 0x559257a61fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe02db640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559257a5fba0 0x559257a61fb0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559257a633f0 0x559257a657e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559257a65e60 con 0x559257a633f0 2024-04-27T03:19:56.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabdfada640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559257a633f0 0x559257a657e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabdfada640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559257a633f0 0x559257a657e0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559257a67710 con 0x559257a5fba0 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:4 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559257a6df20 con 0x559257a5fba0 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(4 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:NOTICE: set zone's realm_id= 2024-04-27T03:19:56.665 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559257a71f80 0x559257a6d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.646+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:5 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a71f80 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdf2d9640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559257a71f80 0x559257a6d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdf2d9640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559257a71f80 0x559257a6d7b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(5 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd401d7e0 con 0x559257a71f80 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:6 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a633f0 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdfada640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fabd80ca030 0x7fabd80c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:7 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x7fabd80ca030 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fabd80ca030 0x7fabd80c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fabd80ca030 0x7fabd80c85f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:8 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:10 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a71f80 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd401d7e0 con 0x559257a71f80 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:11 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.650+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:12 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a633f0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a633f0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:14 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(14 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:15 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x7fabd80ca030 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(15 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:16 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:17 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x7fabd80ca030 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(17 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:18 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a633f0 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(18 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:19 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a71f80 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd401d7e0 con 0x559257a71f80 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:20 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(20 [pgnls start_epoch 31 out=107b] v27'2 uv2 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.654+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:21 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a633f0 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(21 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:22 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a71f80 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd401d7e0 con 0x559257a71f80 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:23 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:24 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x7fabd80ca030 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(24 [pgnls start_epoch 31 out=74b] v20'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:25 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(25 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:26 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a71f80 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd401d7e0 con 0x559257a71f80 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:27 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x7fabd80ca030 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:28 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x559257a5fba0 2024-04-27T03:19:56.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:29 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559257a6f550 con 0x7fabd80ca030 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:30 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7fabd005c040 con 0x559257a71f80 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd401d7e0 con 0x559257a71f80 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:31 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7fabd005c040 con 0x7fabd80ca030 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:32 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7fabd005c040 con 0x559257a633f0 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.658+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7fabd005c040 con 0x559257a633f0 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:34 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7fabd005c040 con 0x559257a71f80 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd401d7e0 con 0x559257a71f80 2024-04-27T03:19:56.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdf2d9640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:35 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7fabd005c040 con 0x559257a633f0 2024-04-27T03:19:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(35 [pgnls start_epoch 31 out=115b] v20'2 uv2 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7fabd005c040 con 0x559257a633f0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(36 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559257a67ab0 con 0x559257a633f0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(37 zone_names.default [read 0~46 out=46b] v0'0 uv1 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559257a6fdd0 con 0x559257a5fba0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(38 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv2 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.662+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:39 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [create,call version.set in=58b,writefull 0~1155 in=1155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559257a70170 con 0x559257a5fba0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.666+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(39 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [create,call,writefull 0~1155] v31'3 uv3 ondisk = 0) ==== 274+0+0 (crc 0 0 0) 0x7fabd001e7a0 con 0x559257a5fba0 2024-04-27T03:19:56.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.666+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:40 5.f 5:f43b6ece:::zone_names.default:head [create,call version.set in=58b,writefull 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559257a6ca70 con 0x559257a633f0 2024-04-27T03:19:56.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.666+0000 7fabdfada640 1 -- 172.21.15.110:0/2283488160 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(40 zone_names.default [create,call,writefull 0~46] v31'3 uv3 ondisk = 0) ==== 246+0+0 (crc 0 0 0) 0x7fabd80b2070 con 0x559257a633f0 2024-04-27T03:19:56.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.666+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.19 5:9f02474b:::default.zone.:head [create,writefull 0~46 in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559257a729d0 con 0x7fabd80ca030 2024-04-27T03:19:56.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 default.zone. [create,writefull 0~46] v31'2 uv1 ondisk = -17 ((17) File exists)) ==== 199+0+0 (crc 0 0 0) 0x7fabd001e7a0 con 0x7fabd80ca030 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faab8034650 msgr2=0x7faab8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faab8034650 0x7faab8036b10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x559257a35080 tx=0x7fabd4001280 comp rx=0 tx=0).stop 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 msgr2=0x559257a40260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a40260 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x559257a39db0 tx=0x7fabd0003000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 shutdown_connections 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faab8034650 0x7faab8036b10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/4232075837 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a38fd0 0x559257a40260 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 >> 172.21.15.110:0/4232075837 conn(0x559257937be0 msgr2=0x559257a33240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 shutdown_connections 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/4232075837 wait complete. 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faa20034650 msgr2=0x7faa20036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faa20034650 0x7faa20036b10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fabd4004180 tx=0x7fabd400f000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a5a180 msgr2=0x559257a5a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a5a180 0x559257a5a5a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fabd80e3910 tx=0x7fabd80bfe90 comp rx=0 tx=0).stop 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 shutdown_connections 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faa20034650 0x7faa20036b10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/222015356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559257a5a180 0x559257a5a5a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 >> 172.21.15.110:0/222015356 conn(0x559257a4ed10 msgr2=0x559257a4f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 shutdown_connections 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/222015356 wait complete. 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559257a633f0 msgr2=0x559257a657e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559257a633f0 0x559257a657e0 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559257a5fba0 msgr2=0x559257a61fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559257a5fba0 0x559257a61fb0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fabd80ca030 msgr2=0x7fabd80c85f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fabd80ca030 0x7fabd80c85f0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559257a71f80 msgr2=0x559257a6d7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559257a71f80 0x559257a6d7b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faac8034650 msgr2=0x7faac8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7faac8034650 0x7faac8036b10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fabd8005790 tx=0x7fabd80056a0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 msgr2=0x559257a2bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe02db640 1 -- 172.21.15.110:0/2283488160 reap_dead start 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x559257a2bc20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fabd801b010 tx=0x7fabd8019810 comp rx=0 tx=0).stop 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 shutdown_connections 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 --2- 172.21.15.110:0/2283488160 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5592579372c0 0x559257a2bc20 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 >> 172.21.15.110:0/2283488160 conn(0x559257932b90 msgr2=0x5592578ab8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 shutdown_connections 2024-04-27T03:19:56.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.670+0000 7fabe1fbf980 1 -- 172.21.15.110:0/2283488160 wait complete. 2024-04-27T03:19:56.692 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"27a51273-3ccc-49e2-8682-1122abc3d898","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}},{"key":"new-placement","val":{"index_pool":".rgw.buckets.index.2","storage_classes":{"STANDARD":{"data_pool":".rgw.buckets.2"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":""} 2024-04-27T03:19:56.692 DEBUG:tasks.util.rgw: json result: {'id': '27a51273-3ccc-49e2-8682-1122abc3d898', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}, {'key': 'new-placement', 'val': {'index_pool': '.rgw.buckets.index.2', 'storage_classes': {'STANDARD': {'data_pool': '.rgw.buckets.2'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': ''} 2024-04-27T03:19:56.692 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'get'] 2024-04-27T03:19:56.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'get'] 2024-04-27T03:19:56.692 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone get 2024-04-27T03:19:56.769 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.758+0000 7f5fba6e9980 1 Processor -- start 2024-04-27T03:19:56.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.758+0000 7f5fba6e9980 1 -- start start 2024-04-27T03:19:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.758+0000 7f5fba6e9980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c6d3b72140 0x55c6d3b72520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.758+0000 7f5fba6e9980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55c6d3a320d0 con 0x55c6d3b72a60 2024-04-27T03:19:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.758+0000 7f5fba6e9980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55c6d3a27430 con 0x55c6d3b72140 2024-04-27T03:19:56.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb8204640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c6d3b72140 0x55c6d3b72520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb7a03640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55c6d3b72a60 0x55c6d3b73470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44508/0 (socket says 172.21.15.110:44508) 2024-04-27T03:19:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb8204640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55c6d3b72140 0x55c6d3b72520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51276/0 (socket says 172.21.15.110:51276) 2024-04-27T03:19:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb7a03640 1 -- 172.21.15.110:0/3148262105 learned_addr learned my addr 172.21.15.110:0/3148262105 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb8204640 1 -- 172.21.15.110:0/3148262105 >> v1:172.21.15.77:6789/0 conn(0x55c6d3b72a60 legacy=0x55c6d3b73470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb8204640 1 -- 172.21.15.110:0/3148262105 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d3a20510 con 0x55c6d3b72140 2024-04-27T03:19:56.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb8204640 1 --2- 172.21.15.110:0/3148262105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3b72520 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800a420 tx=0x7f5fa801afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66acdf0c241d4f08 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb7202640 1 -- 172.21.15.110:0/3148262105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80af070 con 0x55c6d3b72140 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb7202640 1 -- 172.21.15.110:0/3148262105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa80aa530 con 0x55c6d3b72140 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.762+0000 7f5fb7202640 1 -- 172.21.15.110:0/3148262105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80aa850 con 0x55c6d3b72140 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.766+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3148262105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 msgr2=0x55c6d3b72520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.766+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/3148262105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3b72520 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800a420 tx=0x7f5fa801afd0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.766+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3148262105 shutdown_connections 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.766+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/3148262105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3b72520 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.766+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3148262105 >> 172.21.15.110:0/3148262105 conn(0x55c6d3b6da10 msgr2=0x55c6d3b6fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.766+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3148262105 shutdown_connections 2024-04-27T03:19:56.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.766+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3148262105 wait complete. 2024-04-27T03:19:56.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 Processor -- start 2024-04-27T03:19:56.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- start start 2024-04-27T03:19:56.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3ae6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c6d3a762d0 con 0x55c6d3b72140 2024-04-27T03:19:56.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3ae6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3ae6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51292/0 (socket says 172.21.15.110:51292) 2024-04-27T03:19:56.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 -- 172.21.15.110:0/2438842772 learned_addr learned my addr 172.21.15.110:0/2438842772 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 -- 172.21.15.110:0/2438842772 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d39fcae0 con 0x55c6d3b72140 2024-04-27T03:19:56.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 --2- 172.21.15.110:0/2438842772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3ae6f50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800a420 tx=0x7f5fa80033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df190b7bce213b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5ea660c640 1 -- 172.21.15.110:0/2438842772 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80af050 con 0x55c6d3b72140 2024-04-27T03:19:56.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5ea660c640 1 -- 172.21.15.110:0/2438842772 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa80b0040 con 0x55c6d3b72140 2024-04-27T03:19:56.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5ea660c640 1 -- 172.21.15.110:0/2438842772 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80aac60 con 0x55c6d3b72140 2024-04-27T03:19:56.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- 172.21.15.110:0/2438842772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 msgr2=0x55c6d3ae6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/2438842772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3ae6f50 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800a420 tx=0x7f5fa80033b0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- 172.21.15.110:0/2438842772 shutdown_connections 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/2438842772 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3ae6f50 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- 172.21.15.110:0/2438842772 >> 172.21.15.110:0/2438842772 conn(0x55c6d3b6da10 msgr2=0x55c6d3c59590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- 172.21.15.110:0/2438842772 shutdown_connections 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- 172.21.15.110:0/2438842772 wait complete. 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 Processor -- start 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- start start 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3c65a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fba6e9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c6d3b7c1e0 con 0x55c6d3b72140 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3c65a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3c65a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51304/0 (socket says 172.21.15.110:51304) 2024-04-27T03:19:56.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.782+0000 7f5fb8204640 1 -- 172.21.15.110:0/495748120 learned_addr learned my addr 172.21.15.110:0/495748120 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8204640 1 -- 172.21.15.110:0/495748120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d3c65f50 con 0x55c6d3b72140 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8204640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3c65a10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5fa801b010 tx=0x7f5fa8019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5ea4e09640 1 -- 172.21.15.110:0/495748120 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80c5030 con 0x55c6d3b72140 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5ea4e09640 1 -- 172.21.15.110:0/495748120 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa80c15b0 con 0x55c6d3b72140 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c6d3c661e0 con 0x55c6d3b72140 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5ea4e09640 1 -- 172.21.15.110:0/495748120 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80c1890 con 0x55c6d3b72140 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6d3c68e10 con 0x55c6d3b72140 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 Processor -- start 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- start start 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c73250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c6d3a7a310 con 0x55c6d3c72e30 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5ea4e09640 1 -- 172.21.15.110:0/495748120 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5fa80c8070 con 0x55c6d3b72140 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8a05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c73250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8a05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c73250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51310/0 (socket says 172.21.15.110:51310) 2024-04-27T03:19:56.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8a05640 1 -- 172.21.15.110:0/873059613 learned_addr learned my addr 172.21.15.110:0/873059613 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5ea4e09640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea0034650 0x7f5ea0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5ea4e09640 1 -- 172.21.15.110:0/495748120 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f5fa80c6070 con 0x55c6d3b72140 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8204640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea0034650 0x7f5ea0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8a05640 1 -- 172.21.15.110:0/873059613 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d3c73790 con 0x55c6d3c72e30 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8a05640 1 --2- 172.21.15.110:0/873059613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c73250 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000a440 tx=0x7f5fb000a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e73de59f847ab16 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fb8204640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea0034650 0x7f5ea0036b10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8000f50 tx=0x7f5fa80056a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5e9e7fc640 1 -- 172.21.15.110:0/873059613 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0008850 con 0x55c6d3c72e30 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5e9e7fc640 1 -- 172.21.15.110:0/873059613 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb0008a10 con 0x55c6d3c72e30 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5e9e7fc640 1 -- 172.21.15.110:0/873059613 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0008d30 con 0x55c6d3c72e30 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- 172.21.15.110:0/873059613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 msgr2=0x55c6d3c73250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/873059613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c73250 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000a440 tx=0x7f5fb000a900 comp rx=0 tx=0).stop 2024-04-27T03:19:56.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- 172.21.15.110:0/873059613 shutdown_connections 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/873059613 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c73250 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- 172.21.15.110:0/873059613 >> 172.21.15.110:0/873059613 conn(0x55c6d3b72a60 msgr2=0x55c6d3c6e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- 172.21.15.110:0/873059613 shutdown_connections 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.786+0000 7f5fba6e9980 1 -- 172.21.15.110:0/873059613 wait complete. 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 Processor -- start 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 -- start start 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c7a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c6d3c74150 con 0x55c6d3c72e30 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fb8a05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c7a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fb8a05640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c7a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51324/0 (socket says 172.21.15.110:51324) 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fb8a05640 1 -- 172.21.15.110:0/3575829517 learned_addr learned my addr 172.21.15.110:0/3575829517 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fb8a05640 1 -- 172.21.15.110:0/3575829517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d3c7a600 con 0x55c6d3c72e30 2024-04-27T03:19:56.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fb8a05640 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c7a0c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x55c6d3c73c10 tx=0x7f5fb0003000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5e9cff9640 1 -- 172.21.15.110:0/3575829517 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0013a70 con 0x55c6d3c72e30 2024-04-27T03:19:56.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c6d3c7a890 con 0x55c6d3c72e30 2024-04-27T03:19:56.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5e9cff9640 1 -- 172.21.15.110:0/3575829517 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb0023070 con 0x55c6d3c72e30 2024-04-27T03:19:56.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5e9cff9640 1 -- 172.21.15.110:0/3575829517 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0013d40 con 0x55c6d3c72e30 2024-04-27T03:19:56.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6d3c7d570 con 0x55c6d3c72e30 2024-04-27T03:19:56.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5e9cff9640 1 -- 172.21.15.110:0/3575829517 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5fb0028460 con 0x55c6d3c72e30 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5e9cff9640 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5e90034650 0x7f5e90036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 Processor -- start 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5e9cff9640 1 -- 172.21.15.110:0/3575829517 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f5fb004a0b0 con 0x55c6d3c72e30 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fb8a05640 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5e90034650 0x7f5e90036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 -- start start 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c94030 0x55c6d3c94450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.790+0000 7f5fba6e9980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55c6d3ae75e0 con 0x55c6d3c94030 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8a05640 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5e90034650 0x7f5e90036b10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0013fd0 tx=0x7f5fb0013f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8204640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c94030 0x55c6d3c94450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8204640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c94030 0x55c6d3c94450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51332/0 (socket says 172.21.15.110:51332) 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8204640 1 -- 172.21.15.110:0/86928159 learned_addr learned my addr 172.21.15.110:0/86928159 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8204640 1 -- 172.21.15.110:0/86928159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6d3c94990 con 0x55c6d3c94030 2024-04-27T03:19:56.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8204640 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c94030 0x55c6d3c94450 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80c0dd0 tx=0x7f5fa80b4f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5e7cfe1640 1 -- 172.21.15.110:0/86928159 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80e3030 con 0x55c6d3c94030 2024-04-27T03:19:56.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c6d3c95430 con 0x55c6d3c94030 2024-04-27T03:19:56.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5e7cfe1640 1 -- 172.21.15.110:0/86928159 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa80c8360 con 0x55c6d3c94030 2024-04-27T03:19:56.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5e7cfe1640 1 -- 172.21.15.110:0/86928159 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fa80c6270 con 0x55c6d3c94030 2024-04-27T03:19:56.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6d3c95430 con 0x55c6d3c94030 2024-04-27T03:19:56.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5e7cfe1640 1 -- 172.21.15.110:0/86928159 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5fa80dd040 con 0x55c6d3c94030 2024-04-27T03:19:56.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5e7cfe1640 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5df40345b0 0x7f5df4036a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5e7cfe1640 1 -- 172.21.15.110:0/86928159 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f5fa80bb070 con 0x55c6d3c94030 2024-04-27T03:19:56.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb7a03640 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5df40345b0 0x7f5df4036a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5dfe7fc640 20 reqs_thread_entry: start 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c6d3c99a50 0x55c6d3c9be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55c6d3c9c450 con 0x55c6d3c99a50 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb7a03640 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5df40345b0 0x7f5df4036a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0000f70 tx=0x7f5fa0001240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8a05640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c6d3c99a50 0x55c6d3c9be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8a05640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c6d3c99a50 0x55c6d3c9be40 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.794+0000 7f5fb8a05640 1 -- 172.21.15.110:0/495748120 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f5fb001e7a0 con 0x55c6d3c99a50 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c6d3c9d1f0 0x55c6d3c9f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55c6d3c9fc60 con 0x55c6d3c9d1f0 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fb8204640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c6d3c9d1f0 0x55c6d3c9f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fb8204640 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c6d3c9d1f0 0x55c6d3c9f5e0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fb8204640 1 -- 172.21.15.110:0/495748120 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f5fa80b2070 con 0x55c6d3c9d1f0 2024-04-27T03:19:56.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55c6d3ca1510 con 0x55c6d3c99a50 2024-04-27T03:19:56.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fb8a05640 1 -- 172.21.15.110:0/495748120 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~1155 out=1155b] v0'0 uv3 ondisk = 0) ==== 232+0+1203 (crc 0 0 0) 0x7f5fb001e7a0 con 0x55c6d3c99a50 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5e90034650 msgr2=0x7f5e90036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5e90034650 0x7f5e90036b10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0013fd0 tx=0x7f5fb0013f60 comp rx=0 tx=0).stop 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 msgr2=0x55c6d3c7a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c7a0c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x55c6d3c73c10 tx=0x7f5fb0003000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 shutdown_connections 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5e90034650 0x7f5e90036b10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/3575829517 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c72e30 0x55c6d3c7a0c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 >> 172.21.15.110:0/3575829517 conn(0x55c6d3b72a60 msgr2=0x55c6d3c6d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 shutdown_connections 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/3575829517 wait complete. 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5df40345b0 msgr2=0x7f5df4036a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5df40345b0 0x7f5df4036a70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0000f70 tx=0x7f5fa0001240 comp rx=0 tx=0).stop 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c94030 msgr2=0x55c6d3c94450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c94030 0x55c6d3c94450 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80c0dd0 tx=0x7f5fa80b4f60 comp rx=0 tx=0).stop 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 shutdown_connections 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5df40345b0 0x7f5df4036a70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/86928159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3c94030 0x55c6d3c94450 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 >> 172.21.15.110:0/86928159 conn(0x55c6d3c88c10 msgr2=0x55c6d3c89630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 shutdown_connections 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/86928159 wait complete. 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c6d3c9d1f0 msgr2=0x55c6d3c9f5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c6d3c9d1f0 0x55c6d3c9f5e0 crc :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c6d3c99a50 msgr2=0x55c6d3c9be40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c6d3c99a50 0x55c6d3c9be40 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea0034650 msgr2=0x7f5ea0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea0034650 0x7f5ea0036b10 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8000f50 tx=0x7f5fa80056a0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 msgr2=0x55c6d3c65a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.798+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3c65a10 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5fa801b010 tx=0x7f5fa8019810 comp rx=0 tx=0).stop 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 shutdown_connections 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55c6d3c9d1f0 0x55c6d3c9f5e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55c6d3c99a50 0x55c6d3c9be40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5ea0034650 0x7f5ea0036b10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 --2- 172.21.15.110:0/495748120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55c6d3b72140 0x55c6d3c65a10 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 >> 172.21.15.110:0/495748120 conn(0x55c6d3b6da10 msgr2=0x55c6d3c580a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 shutdown_connections 2024-04-27T03:19:56.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.802+0000 7f5fba6e9980 1 -- 172.21.15.110:0/495748120 wait complete. 2024-04-27T03:19:56.821 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"27a51273-3ccc-49e2-8682-1122abc3d898","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}},{"key":"new-placement","val":{"index_pool":".rgw.buckets.index.2","storage_classes":{"STANDARD":{"data_pool":".rgw.buckets.2"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":""} 2024-04-27T03:19:56.821 DEBUG:tasks.util.rgw: json result: {'id': '27a51273-3ccc-49e2-8682-1122abc3d898', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}, {'key': 'new-placement', 'val': {'index_pool': '.rgw.buckets.index.2', 'storage_classes': {'STANDARD': {'data_pool': '.rgw.buckets.2'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': ''} 2024-04-27T03:19:56.821 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2024-04-27T03:19:56.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'rm', '--rgw-zone', 'default', '--placement-id', 'new-placement'] 2024-04-27T03:19:56.821 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement rm --rgw-zone default --placement-id new-placement 2024-04-27T03:19:56.905 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:56.905 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:56.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ee93c980 1 Processor -- start 2024-04-27T03:19:56.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ee93c980 1 -- start start 2024-04-27T03:19:56.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ee93c980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5639a2ca34c0 0x5639a2ca38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ee93c980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5639a2b600d0 con 0x5639a2ca3de0 2024-04-27T03:19:56.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ee93c980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5639a2b55430 con 0x5639a2ca34c0 2024-04-27T03:19:56.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ec457640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5639a2ca34c0 0x5639a2ca38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ec457640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5639a2ca34c0 0x5639a2ca38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51342/0 (socket says 172.21.15.110:51342) 2024-04-27T03:19:56.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.894+0000 7fb9ec457640 1 -- 172.21.15.110:0/1904855423 learned_addr learned my addr 172.21.15.110:0/1904855423 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ec457640 1 -- 172.21.15.110:0/1904855423 >> v1:172.21.15.77:6789/0 conn(0x5639a2ca3de0 legacy=0x5639a2ca4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ec457640 1 -- 172.21.15.110:0/1904855423 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2b4e510 con 0x5639a2ca34c0 2024-04-27T03:19:56.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ec457640 1 --2- 172.21.15.110:0/1904855423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2ca38a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00a5c0 tx=0x7fb9dc01b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac8d0f840fe5e1e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9eb455640 1 -- 172.21.15.110:0/1904855423 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0a6070 con 0x5639a2ca34c0 2024-04-27T03:19:56.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9eb455640 1 -- 172.21.15.110:0/1904855423 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9dc0af070 con 0x5639a2ca34c0 2024-04-27T03:19:56.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9eb455640 1 -- 172.21.15.110:0/1904855423 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0aa5c0 con 0x5639a2ca34c0 2024-04-27T03:19:56.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1904855423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 msgr2=0x5639a2ca38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1904855423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2ca38a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00a5c0 tx=0x7fb9dc01b340 comp rx=0 tx=0).stop 2024-04-27T03:19:56.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1904855423 shutdown_connections 2024-04-27T03:19:56.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1904855423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2ca38a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1904855423 >> 172.21.15.110:0/1904855423 conn(0x5639a2c16cc0 msgr2=0x5639a2ca19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1904855423 shutdown_connections 2024-04-27T03:19:56.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.898+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1904855423 wait complete. 2024-04-27T03:19:56.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ee93c980 1 Processor -- start 2024-04-27T03:19:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ee93c980 1 -- start start 2024-04-27T03:19:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ee93c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d8ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ee93c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639a2ba42d0 con 0x5639a2ca34c0 2024-04-27T03:19:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ec457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d8ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ec457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d8ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51350/0 (socket says 172.21.15.110:51350) 2024-04-27T03:19:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ec457640 1 -- 172.21.15.110:0/2929286540 learned_addr learned my addr 172.21.15.110:0/2929286540 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.914+0000 7fb9ec457640 1 -- 172.21.15.110:0/2929286540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2b2aae0 con 0x5639a2ca34c0 2024-04-27T03:19:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ec457640 1 --2- 172.21.15.110:0/2929286540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d8ed30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0199f0 tx=0x7fb9dc003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bab38a21d7bf1b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb8da60c640 1 -- 172.21.15.110:0/2929286540 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0aaa00 con 0x5639a2ca34c0 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb8da60c640 1 -- 172.21.15.110:0/2929286540 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9dc0aaba0 con 0x5639a2ca34c0 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb8da60c640 1 -- 172.21.15.110:0/2929286540 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0bd3f0 con 0x5639a2ca34c0 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2929286540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 msgr2=0x5639a2d8ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2929286540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d8ed30 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0199f0 tx=0x7fb9dc003030 comp rx=0 tx=0).stop 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2929286540 shutdown_connections 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2929286540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d8ed30 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2929286540 >> 172.21.15.110:0/2929286540 conn(0x5639a2c16cc0 msgr2=0x5639a2d8a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2929286540 shutdown_connections 2024-04-27T03:19:56.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2929286540 wait complete. 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 Processor -- start 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- start start 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d95d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639a2cad500 con 0x5639a2ca34c0 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ec457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d95d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ec457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d95d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51354/0 (socket says 172.21.15.110:51354) 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ec457640 1 -- 172.21.15.110:0/2949317070 learned_addr learned my addr 172.21.15.110:0/2949317070 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ec457640 1 -- 172.21.15.110:0/2949317070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2d96280 con 0x5639a2ca34c0 2024-04-27T03:19:56.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ec457640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d95d40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc01b170 tx=0x7fb9dc0180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb8d8e09640 1 -- 172.21.15.110:0/2949317070 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0bd7c0 con 0x5639a2ca34c0 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb8d8e09640 1 -- 172.21.15.110:0/2949317070 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9dc0bd960 con 0x5639a2ca34c0 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb8d8e09640 1 -- 172.21.15.110:0/2949317070 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0bdc40 con 0x5639a2ca34c0 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5639a2d96510 con 0x5639a2ca34c0 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.918+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a2d99140 con 0x5639a2ca34c0 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 Processor -- start 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- start start 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2da3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639a2ba8310 con 0x5639a2da3010 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2da3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2da3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51356/0 (socket says 172.21.15.110:51356) 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 -- 172.21.15.110:0/1906576344 learned_addr learned my addr 172.21.15.110:0/1906576344 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb8d8e09640 1 -- 172.21.15.110:0/2949317070 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb9dc0ce020 con 0x5639a2ca34c0 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb8d8e09640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8d4034650 0x7fb8d4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb8d8e09640 1 -- 172.21.15.110:0/2949317070 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fb9dc0cc020 con 0x5639a2ca34c0 2024-04-27T03:19:56.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ec457640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8d4034650 0x7fb8d4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 -- 172.21.15.110:0/1906576344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2da3970 con 0x5639a2da3010 2024-04-27T03:19:56.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 --2- 172.21.15.110:0/1906576344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2da3430 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400a440 tx=0x7fb9e400a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d993593b4dcbad server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ec457640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8d4034650 0x7fb8d4036b10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc01b380 tx=0x7fb9dc0d0450 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb8d27fc640 1 -- 172.21.15.110:0/1906576344 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4008850 con 0x5639a2da3010 2024-04-27T03:19:56.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb8d27fc640 1 -- 172.21.15.110:0/1906576344 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9e4008a10 con 0x5639a2da3010 2024-04-27T03:19:56.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb8d27fc640 1 -- 172.21.15.110:0/1906576344 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4008d30 con 0x5639a2da3010 2024-04-27T03:19:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1906576344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 msgr2=0x5639a2da3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1906576344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2da3430 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400a440 tx=0x7fb9e400a900 comp rx=0 tx=0).stop 2024-04-27T03:19:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1906576344 shutdown_connections 2024-04-27T03:19:56.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1906576344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2da3430 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1906576344 >> 172.21.15.110:0/1906576344 conn(0x5639a2ca3de0 msgr2=0x5639a2d9e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1906576344 shutdown_connections 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1906576344 wait complete. 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 Processor -- start 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- start start 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2daa320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ee93c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639a2da4330 con 0x5639a2da3010 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2daa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2daa320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51360/0 (socket says 172.21.15.110:51360) 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 -- 172.21.15.110:0/2403193119 learned_addr learned my addr 172.21.15.110:0/2403193119 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.922+0000 7fb9ecc58640 1 -- 172.21.15.110:0/2403193119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2daa860 con 0x5639a2da3010 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ecc58640 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2daa320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x5639a2da3df0 tx=0x7fb9e4003000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb8d0ff9640 1 -- 172.21.15.110:0/2403193119 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4013a70 con 0x5639a2da3010 2024-04-27T03:19:56.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5639a2daaaf0 con 0x5639a2da3010 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb8d0ff9640 1 -- 172.21.15.110:0/2403193119 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9e4023070 con 0x5639a2da3010 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb8d0ff9640 1 -- 172.21.15.110:0/2403193119 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9e4013d40 con 0x5639a2da3010 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a2dad7d0 con 0x5639a2da3010 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb8d0ff9640 1 -- 172.21.15.110:0/2403193119 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb9e4028460 con 0x5639a2da3010 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb8d0ff9640 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8c4034650 0x7fb8c4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb8d0ff9640 1 -- 172.21.15.110:0/2403193119 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fb9e404a0b0 con 0x5639a2da3010 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 1 Processor -- start 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ecc58640 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8c4034650 0x7fb8c4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 1 -- start start 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2dc4290 0x5639a2dc46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5639a2d8f3c0 con 0x5639a2dc4290 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ecc58640 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8c4034650 0x7fb8c4036b10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4027fd0 tx=0x7fb9e4013ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ec457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2dc4290 0x5639a2dc46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ec457640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2dc4290 0x5639a2dc46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51376/0 (socket says 172.21.15.110:51376) 2024-04-27T03:19:56.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ec457640 1 -- 172.21.15.110:0/1909535318 learned_addr learned my addr 172.21.15.110:0/1909535318 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ec457640 1 -- 172.21.15.110:0/1909535318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2dc4bf0 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ec457640 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2dc4290 0x5639a2dc46b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0bc970 tx=0x7fb9dc0e3820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb8b0fe1640 1 -- 172.21.15.110:0/1909535318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0f5070 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.926+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5639a2dc5690 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb8b0fe1640 1 -- 172.21.15.110:0/1909535318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9dc0e0040 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb8b0fe1640 1 -- 172.21.15.110:0/1909535318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9dc0cf370 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a2dc5b60 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb8b0fe1640 1 -- 172.21.15.110:0/1909535318 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb9dc0f8070 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb8b0fe1640 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb824034650 0x7fb824036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb8b0fe1640 1 -- 172.21.15.110:0/1909535318 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fb9dc0ec030 con 0x5639a2dc4290 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ebc56640 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb824034650 0x7fb824036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb83a7fc640 20 reqs_thread_entry: start 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5639a2dc9e70 0x5639a2dcc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ebc56640 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb824034650 0x7fb824036b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00012f0 tx=0x7fb9e0006630 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:1 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5639a2dcc940 con 0x5639a2dc9e70 2024-04-27T03:19:56.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ecc58640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5639a2dc9e70 0x5639a2dcc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ecc58640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5639a2dc9e70 0x5639a2dcc2a0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ecc58640 1 -- 172.21.15.110:0/2949317070 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(1 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fb9e401e7a0 con 0x5639a2dc9e70 2024-04-27T03:19:56.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639a2dce710 0x5639a2dd0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:56.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5639a2dd11f0 con 0x5639a2dce710 2024-04-27T03:19:56.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.930+0000 7fb9ec457640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639a2dce710 0x5639a2dd0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:56.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ec457640 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639a2dce710 0x5639a2dd0b00 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:56.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ec457640 1 -- 172.21.15.110:0/2949317070 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(2 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~1155 out=1155b] v0'0 uv3 ondisk = 0) ==== 232+0+1203 (crc 0 0 0) 0x7fb9dc102040 con 0x5639a2dce710 2024-04-27T03:19:56.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [stat,call version.check_conds in=122b,call version.inc in=38b,writefull 0~993 in=993b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5639a2dd6020 con 0x5639a2dce710 2024-04-27T03:19:56.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ec457640 1 -- 172.21.15.110:0/2949317070 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [stat,call,call,writefull 0~993] v31'4 uv4 ondisk = 0) ==== 316+0+0 (crc 0 0 0) 0x7fb9dc102040 con 0x5639a2dce710 2024-04-27T03:19:56.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8c4034650 msgr2=0x7fb8c4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8c4034650 0x7fb8c4036b10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4027fd0 tx=0x7fb9e4013ee0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 msgr2=0x5639a2daa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2daa320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x5639a2da3df0 tx=0x7fb9e4003000 comp rx=0 tx=0).stop 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 shutdown_connections 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8c4034650 0x7fb8c4036b10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2403193119 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2da3010 0x5639a2daa320 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 >> 172.21.15.110:0/2403193119 conn(0x5639a2ca3de0 msgr2=0x5639a2d9d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 shutdown_connections 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2403193119 wait complete. 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb824034650 msgr2=0x7fb824036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb824034650 0x7fb824036b10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00012f0 tx=0x7fb9e0006630 comp rx=0 tx=0).stop 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2dc4290 msgr2=0x5639a2dc46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2dc4290 0x5639a2dc46b0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0bc970 tx=0x7fb9dc0e3820 comp rx=0 tx=0).stop 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 shutdown_connections 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb824034650 0x7fb824036b10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/1909535318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2dc4290 0x5639a2dc46b0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 >> 172.21.15.110:0/1909535318 conn(0x5639a2db8e70 msgr2=0x5639a2db9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 shutdown_connections 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.934+0000 7fb9ee93c980 1 -- 172.21.15.110:0/1909535318 wait complete. 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5639a2dc9e70 msgr2=0x5639a2dcc2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5639a2dc9e70 0x5639a2dcc2a0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639a2dce710 msgr2=0x5639a2dd0b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639a2dce710 0x5639a2dd0b00 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8d4034650 msgr2=0x7fb8d4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8d4034650 0x7fb8d4036b10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc01b380 tx=0x7fb9dc0d0450 comp rx=0 tx=0).stop 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 msgr2=0x5639a2d95d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d95d40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc01b170 tx=0x7fb9dc0180e0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 shutdown_connections 2024-04-27T03:19:56.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5639a2dce710 0x5639a2dd0b00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5639a2dc9e70 0x5639a2dcc2a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fb8d4034650 0x7fb8d4036b10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 --2- 172.21.15.110:0/2949317070 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5639a2ca34c0 0x5639a2d95d40 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:56.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 >> 172.21.15.110:0/2949317070 conn(0x5639a2c16cc0 msgr2=0x5639a2d893d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:56.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 shutdown_connections 2024-04-27T03:19:56.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:56.938+0000 7fb9ee93c980 1 -- 172.21.15.110:0/2949317070 wait complete. 2024-04-27T03:19:56.957 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"27a51273-3ccc-49e2-8682-1122abc3d898","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","notif_pool":"default.rgw.log:notif","topics_pool":"default.rgw.meta:topics","account_pool":"default.rgw.meta:accounts","group_pool":"default.rgw.meta:groups","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":""} 2024-04-27T03:19:56.957 DEBUG:tasks.util.rgw: json result: {'id': '27a51273-3ccc-49e2-8682-1122abc3d898', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'notif_pool': 'default.rgw.log:notif', 'topics_pool': 'default.rgw.meta:topics', 'account_pool': 'default.rgw.meta:accounts', 'group_pool': 'default.rgw.meta:groups', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': ''} 2024-04-27T03:19:56.957 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'get'] 2024-04-27T03:19:56.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'get'] 2024-04-27T03:19:56.958 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup get 2024-04-27T03:19:57.029 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:57.029 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:57.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.018+0000 7f6547849980 1 Processor -- start 2024-04-27T03:19:57.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.018+0000 7f6547849980 1 -- start start 2024-04-27T03:19:57.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.018+0000 7f6547849980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5600a225d160 0x5600a225d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.018+0000 7f6547849980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5600a211d0d0 con 0x5600a225da80 2024-04-27T03:19:57.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.018+0000 7f6547849980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5600a2112430 con 0x5600a225d160 2024-04-27T03:19:57.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.022+0000 7f6545364640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5600a225d160 0x5600a225d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.022+0000 7f6544b63640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5600a225da80 0x5600a225e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44522/0 (socket says 172.21.15.110:44522) 2024-04-27T03:19:57.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.022+0000 7f6544b63640 1 -- 172.21.15.110:0/988749497 learned_addr learned my addr 172.21.15.110:0/988749497 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:57.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.022+0000 7f6545364640 1 -- 172.21.15.110:0/988749497 >> v1:172.21.15.77:6789/0 conn(0x5600a225da80 legacy=0x5600a225e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.022+0000 7f6545364640 1 -- 172.21.15.110:0/988749497 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a210b510 con 0x5600a225d160 2024-04-27T03:19:57.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.022+0000 7f6545364640 1 --2- 172.21.15.110:0/988749497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a225d540 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f653800a480 tx=0x7f653801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566feff74428634e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.022+0000 7f6537fff640 1 -- 172.21.15.110:0/988749497 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380b0070 con 0x5600a225d160 2024-04-27T03:19:57.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6537fff640 1 -- 172.21.15.110:0/988749497 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65380ab540 con 0x5600a225d160 2024-04-27T03:19:57.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6537fff640 1 -- 172.21.15.110:0/988749497 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380ab840 con 0x5600a225d160 2024-04-27T03:19:57.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6547849980 1 -- 172.21.15.110:0/988749497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 msgr2=0x5600a225d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6547849980 1 --2- 172.21.15.110:0/988749497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a225d540 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f653800a480 tx=0x7f653801b280 comp rx=0 tx=0).stop 2024-04-27T03:19:57.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6547849980 1 -- 172.21.15.110:0/988749497 shutdown_connections 2024-04-27T03:19:57.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6547849980 1 --2- 172.21.15.110:0/988749497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a225d540 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6547849980 1 -- 172.21.15.110:0/988749497 >> 172.21.15.110:0/988749497 conn(0x5600a2258a30 msgr2=0x5600a225ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6547849980 1 -- 172.21.15.110:0/988749497 shutdown_connections 2024-04-27T03:19:57.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.026+0000 7f6547849980 1 -- 172.21.15.110:0/988749497 wait complete. 2024-04-27T03:19:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.038+0000 7f6547849980 1 Processor -- start 2024-04-27T03:19:57.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.038+0000 7f6547849980 1 -- start start 2024-04-27T03:19:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.038+0000 7f6547849980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a2348a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.038+0000 7f6547849980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5600a21612d0 con 0x5600a225d160 2024-04-27T03:19:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a2348a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a2348a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51390/0 (socket says 172.21.15.110:51390) 2024-04-27T03:19:57.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 -- 172.21.15.110:0/2113222852 learned_addr learned my addr 172.21.15.110:0/2113222852 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 -- 172.21.15.110:0/2113222852 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a20e7ae0 con 0x5600a225d160 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 --2- 172.21.15.110:0/2113222852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a2348a80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f65380ae040 tx=0x7f6538003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685d3f48625817c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6433606640 1 -- 172.21.15.110:0/2113222852 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380b0040 con 0x5600a225d160 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6433606640 1 -- 172.21.15.110:0/2113222852 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65380ab9d0 con 0x5600a225d160 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6433606640 1 -- 172.21.15.110:0/2113222852 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380abcf0 con 0x5600a225d160 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 -- 172.21.15.110:0/2113222852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 msgr2=0x5600a2348a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 --2- 172.21.15.110:0/2113222852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a2348a80 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f65380ae040 tx=0x7f6538003030 comp rx=0 tx=0).stop 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 -- 172.21.15.110:0/2113222852 shutdown_connections 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 --2- 172.21.15.110:0/2113222852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a2348a80 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 -- 172.21.15.110:0/2113222852 >> 172.21.15.110:0/2113222852 conn(0x5600a2258a30 msgr2=0x5600a2344610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 -- 172.21.15.110:0/2113222852 shutdown_connections 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 -- 172.21.15.110:0/2113222852 wait complete. 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 Processor -- start 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 -- start start 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a234fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6547849980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5600a2267200 con 0x5600a225d160 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a234fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a234fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51400/0 (socket says 172.21.15.110:51400) 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 -- 172.21.15.110:0/4095010740 learned_addr learned my addr 172.21.15.110:0/4095010740 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 -- 172.21.15.110:0/4095010740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a234ff40 con 0x5600a225d160 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.042+0000 7f6545364640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a234fa00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6538018eb0 tx=0x7f6538018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6431e03640 1 -- 172.21.15.110:0/4095010740 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380b0070 con 0x5600a225d160 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6431e03640 1 -- 172.21.15.110:0/4095010740 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65380b68f0 con 0x5600a225d160 2024-04-27T03:19:57.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5600a23501d0 con 0x5600a225d160 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6431e03640 1 -- 172.21.15.110:0/4095010740 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380b6bd0 con 0x5600a225d160 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600a2352db0 con 0x5600a225d160 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 Processor -- start 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- start start 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a235d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5600a2165310 con 0x5600a235ccc0 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a235d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6431e03640 1 -- 172.21.15.110:0/4095010740 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f65380bf3f0 con 0x5600a225d160 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a235d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51408/0 (socket says 172.21.15.110:51408) 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 -- 172.21.15.110:0/3221518150 learned_addr learned my addr 172.21.15.110:0/3221518150 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6431e03640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f642c034650 0x7f642c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6431e03640 1 -- 172.21.15.110:0/4095010740 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f65380bc070 con 0x5600a225d160 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545364640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f642c034650 0x7f642c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 -- 172.21.15.110:0/3221518150 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a235d620 con 0x5600a235ccc0 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 --2- 172.21.15.110:0/3221518150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a235d0e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f654000c6a0 tx=0x7f654000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deeaf47fab0ba226 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545364640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f642c034650 0x7f642c036b10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f65380ab000 tx=0x7f65380aaed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f642b7fe640 1 -- 172.21.15.110:0/3221518150 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6540013070 con 0x5600a235ccc0 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f642b7fe640 1 -- 172.21.15.110:0/3221518150 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f654000aa80 con 0x5600a235ccc0 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f642b7fe640 1 -- 172.21.15.110:0/3221518150 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f654000ada0 con 0x5600a235ccc0 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- 172.21.15.110:0/3221518150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 msgr2=0x5600a235d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 --2- 172.21.15.110:0/3221518150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a235d0e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f654000c6a0 tx=0x7f654000cb60 comp rx=0 tx=0).stop 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- 172.21.15.110:0/3221518150 shutdown_connections 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 --2- 172.21.15.110:0/3221518150 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a235d0e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- 172.21.15.110:0/3221518150 >> 172.21.15.110:0/3221518150 conn(0x5600a225da80 msgr2=0x5600a2358130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- 172.21.15.110:0/3221518150 shutdown_connections 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- 172.21.15.110:0/3221518150 wait complete. 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 Processor -- start 2024-04-27T03:19:57.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- start start 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a2362ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6547849980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5600a235dfe0 con 0x5600a235ccc0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a2362ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a2362ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51416/0 (socket says 172.21.15.110:51416) 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.046+0000 7f6545b65640 1 -- 172.21.15.110:0/283133178 learned_addr learned my addr 172.21.15.110:0/283133178 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545b65640 1 -- 172.21.15.110:0/283133178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a2363400 con 0x5600a235ccc0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545b65640 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a2362ec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f654000cba0 tx=0x7f6540003380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6429ffb640 1 -- 172.21.15.110:0/283133178 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f654002a070 con 0x5600a235ccc0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6429ffb640 1 -- 172.21.15.110:0/283133178 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f654000f470 con 0x5600a235ccc0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5600a2363690 con 0x5600a235ccc0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6429ffb640 1 -- 172.21.15.110:0/283133178 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f654000f810 con 0x5600a235ccc0 2024-04-27T03:19:57.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600a2366370 con 0x5600a235ccc0 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6429ffb640 1 -- 172.21.15.110:0/283133178 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f654002d070 con 0x5600a235ccc0 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 1 Processor -- start 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6429ffb640 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c034650 0x7f641c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 1 -- start start 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6429ffb640 1 -- 172.21.15.110:0/283133178 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f654004c410 con 0x5600a235ccc0 2024-04-27T03:19:57.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545b65640 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c034650 0x7f641c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a237cdf0 0x5600a237d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6547849980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5600a2349110 con 0x5600a237cdf0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545364640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a237cdf0 0x5600a237d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545b65640 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c034650 0x7f641c036b10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f654000efd0 tx=0x7f6540005910 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545364640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a237cdf0 0x5600a237d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51422/0 (socket says 172.21.15.110:51422) 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545364640 1 -- 172.21.15.110:0/3598287209 learned_addr learned my addr 172.21.15.110:0/3598287209 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.050+0000 7f6545364640 1 -- 172.21.15.110:0/3598287209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a237d750 con 0x5600a237cdf0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6545364640 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a237cdf0 0x5600a237d210 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6538006b40 tx=0x7f65380e2be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6409fe3640 1 -- 172.21.15.110:0/3598287209 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380aac70 con 0x5600a237cdf0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5600a237e1f0 con 0x5600a237cdf0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6409fe3640 1 -- 172.21.15.110:0/3598287209 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65380b53d0 con 0x5600a237cdf0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6409fe3640 1 -- 172.21.15.110:0/3598287209 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65380e4a90 con 0x5600a237cdf0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600a237e6c0 con 0x5600a237cdf0 2024-04-27T03:19:57.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6409fe3640 1 -- 172.21.15.110:0/3598287209 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f65380eb060 con 0x5600a237cdf0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6409fe3640 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6388034650 0x7f6388036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6409fe3640 1 -- 172.21.15.110:0/3598287209 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f65380a8070 con 0x5600a237cdf0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6544b63640 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6388034650 0x7f6388036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f63837fe640 20 reqs_thread_entry: start 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5600a2382a90 0x5600a2384e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5600a2385400 con 0x5600a2382a90 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6544b63640 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6388034650 0x7f6388036b10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f653c000f70 tx=0x7f653c001240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6545b65640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5600a2382a90 0x5600a2384e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6545b65640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5600a2382a90 0x5600a2384e80 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6545b65640 1 -- 172.21.15.110:0/4095010740 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f654002c030 con 0x5600a2382a90 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5600a23861d0 0x5600a23885c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5600a2388c80 con 0x5600a23861d0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.054+0000 7f6545364640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5600a23861d0 0x5600a23885c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6545364640 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5600a23861d0 0x5600a23885c0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6545364640 1 -- 172.21.15.110:0/4095010740 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f65380b3040 con 0x5600a23861d0 2024-04-27T03:19:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5600a238a4e0 con 0x5600a2382a90 2024-04-27T03:19:57.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6545b65640 1 -- 172.21.15.110:0/4095010740 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f654002c030 con 0x5600a2382a90 2024-04-27T03:19:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c034650 msgr2=0x7f641c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c034650 0x7f641c036b10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f654000efd0 tx=0x7f6540005910 comp rx=0 tx=0).stop 2024-04-27T03:19:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 msgr2=0x5600a2362ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a2362ec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f654000cba0 tx=0x7f6540003380 comp rx=0 tx=0).stop 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 shutdown_connections 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c034650 0x7f641c036b10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/283133178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a235ccc0 0x5600a2362ec0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 >> 172.21.15.110:0/283133178 conn(0x5600a225da80 msgr2=0x5600a2356f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 shutdown_connections 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/283133178 wait complete. 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6388034650 msgr2=0x7f6388036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6388034650 0x7f6388036b10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f653c000f70 tx=0x7f653c001240 comp rx=0 tx=0).stop 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a237cdf0 msgr2=0x5600a237d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a237cdf0 0x5600a237d210 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6538006b40 tx=0x7f65380e2be0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 shutdown_connections 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6388034650 0x7f6388036b10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/3598287209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a237cdf0 0x5600a237d210 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 >> 172.21.15.110:0/3598287209 conn(0x5600a23719d0 msgr2=0x5600a23723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 shutdown_connections 2024-04-27T03:19:57.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/3598287209 wait complete. 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5600a23861d0 msgr2=0x5600a23885c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5600a23861d0 0x5600a23885c0 crc :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5600a2382a90 msgr2=0x5600a2384e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5600a2382a90 0x5600a2384e80 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f642c034650 msgr2=0x7f642c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f642c034650 0x7f642c036b10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f65380ab000 tx=0x7f65380aaed0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 msgr2=0x5600a234fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a234fa00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6538018eb0 tx=0x7f6538018020 comp rx=0 tx=0).stop 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 shutdown_connections 2024-04-27T03:19:57.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5600a23861d0 0x5600a23885c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5600a2382a90 0x5600a2384e80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f642c034650 0x7f642c036b10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 --2- 172.21.15.110:0/4095010740 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5600a225d160 0x5600a234fa00 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.058+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 >> 172.21.15.110:0/4095010740 conn(0x5600a2258a30 msgr2=0x5600a2343120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.062+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 shutdown_connections 2024-04-27T03:19:57.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.062+0000 7f6547849980 1 -- 172.21.15.110:0/4095010740 wait complete. 2024-04-27T03:19:57.082 INFO:teuthology.orchestra.run.smithi110.stdout:{"id":"3fb181a9-2f1b-4a6e-85a1-ca8305b0483c","name":"default","api_name":"default","is_master":true,"endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"27a51273-3ccc-49e2-8682-1122abc3d898","zones":[{"id":"27a51273-3ccc-49e2-8682-1122abc3d898","name":"default","endpoints":[],"log_meta":false,"log_data":false,"bucket_index_max_shards":11,"read_only":false,"tier_type":"","sync_from_all":true,"sync_from":[],"redirect_zone":"","supported_features":["notification_v2","resharding"]}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"","sync_policy":{"groups":[]},"enabled_features":["notification_v2","resharding"]} 2024-04-27T03:19:57.083 DEBUG:tasks.util.rgw: json result: {'id': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'name': 'default', 'api_name': 'default', 'is_master': True, 'endpoints': [], 'hostnames': [], 'hostnames_s3website': [], 'master_zone': '27a51273-3ccc-49e2-8682-1122abc3d898', 'zones': [{'id': '27a51273-3ccc-49e2-8682-1122abc3d898', 'name': 'default', 'endpoints': [], 'log_meta': False, 'log_data': False, 'bucket_index_max_shards': 11, 'read_only': False, 'tier_type': '', 'sync_from_all': True, 'sync_from': [], 'redirect_zone': '', 'supported_features': ['notification_v2', 'resharding']}], 'placement_targets': [{'name': 'default-placement', 'tags': [], 'storage_classes': ['STANDARD']}], 'default_placement': 'default-placement', 'realm_id': '', 'sync_policy': {'groups': []}, 'enabled_features': ['notification_v2', 'resharding']} 2024-04-27T03:19:57.085 INFO:teuthology.run_tasks:Running task radosgw-admin-rest... 2024-04-27T03:19:57.104 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'ada', '--display-name', 'Ms. Admin User', '--access-key', 'MH1WC2XQ1S8UISFDZC8W', '--secret', 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', '--max-buckets', '0', '--caps', 'users=read, write; usage=read, write; buckets=read, write; zone=read, write; info=read;ratelimit=read, write'] 2024-04-27T03:19:57.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'ada', '--display-name', 'Ms. Admin User', '--access-key', 'MH1WC2XQ1S8UISFDZC8W', '--secret', 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', '--max-buckets', '0', '--caps', 'users=read, write; usage=read, write; buckets=read, write; zone=read, write; info=read;ratelimit=read, write'] 2024-04-27T03:19:57.104 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid ada --display-name 'Ms. Admin User' --access-key MH1WC2XQ1S8UISFDZC8W --secret dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG --max-buckets 0 --caps 'users=read, write; usage=read, write; buckets=read, write; zone=read, write; info=read;ratelimit=read, write' 2024-04-27T03:19:57.168 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:57.168 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:57.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5526cfb980 1 Processor -- start 2024-04-27T03:19:57.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5526cfb980 1 -- start start 2024-04-27T03:19:57.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5526cfb980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5624d4ff3740 0x5624d4ff5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5526cfb980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5624d4eae0d0 con 0x5624d4fef230 2024-04-27T03:19:57.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5526cfb980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5624d4ea3430 con 0x5624d4ff3740 2024-04-27T03:19:57.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5524015640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5624d4ff3740 0x5624d4ff5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5524816640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x5624d4fef230 0x5624d4fef610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44538/0 (socket says 172.21.15.110:44538) 2024-04-27T03:19:57.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5524816640 1 -- 172.21.15.110:0/3312082888 learned_addr learned my addr 172.21.15.110:0/3312082888 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:57.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5524015640 1 -- 172.21.15.110:0/3312082888 >> v1:172.21.15.77:6789/0 conn(0x5624d4fef230 legacy=0x5624d4fef610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.158+0000 7f5524015640 1 -- 172.21.15.110:0/3312082888 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d4e9c510 con 0x5624d4ff3740 2024-04-27T03:19:57.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5524015640 1 --2- 172.21.15.110:0/3312082888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4ff3740 0x5624d4ff5b30 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f551800a480 tx=0x7f551801b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c00c565fd3da674 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5523814640 1 -- 172.21.15.110:0/3312082888 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55180b0070 con 0x5624d4ff3740 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5523814640 1 -- 172.21.15.110:0/3312082888 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55180ab540 con 0x5624d4ff3740 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5523814640 1 -- 172.21.15.110:0/3312082888 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55180ab840 con 0x5624d4ff3740 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5526cfb980 1 -- 172.21.15.110:0/3312082888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4ff3740 msgr2=0x5624d4ff5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3312082888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4ff3740 0x5624d4ff5b30 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f551800a480 tx=0x7f551801b280 comp rx=0 tx=0).stop 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5526cfb980 1 -- 172.21.15.110:0/3312082888 shutdown_connections 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3312082888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4ff3740 0x5624d4ff5b30 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5526cfb980 1 -- 172.21.15.110:0/3312082888 >> 172.21.15.110:0/3312082888 conn(0x5624d4feab00 msgr2=0x5624d4fecf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5526cfb980 1 -- 172.21.15.110:0/3312082888 shutdown_connections 2024-04-27T03:19:57.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.162+0000 7f5526cfb980 1 -- 172.21.15.110:0/3312082888 wait complete. 2024-04-27T03:19:57.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5526cfb980 1 Processor -- start 2024-04-27T03:19:57.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5526cfb980 1 -- start start 2024-04-27T03:19:57.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5526cfb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50da490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5526cfb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624d4ef22d0 con 0x5624d4fef230 2024-04-27T03:19:57.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5524816640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50da490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5524816640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50da490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51450/0 (socket says 172.21.15.110:51450) 2024-04-27T03:19:57.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5524816640 1 -- 172.21.15.110:0/456738647 learned_addr learned my addr 172.21.15.110:0/456738647 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5524816640 1 -- 172.21.15.110:0/456738647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d4e78ae0 con 0x5624d4fef230 2024-04-27T03:19:57.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5524816640 1 --2- 172.21.15.110:0/456738647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50da490 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f551c0017b0 tx=0x7f551c001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95372822db7de7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5412e0d640 1 -- 172.21.15.110:0/456738647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c016070 con 0x5624d4fef230 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5412e0d640 1 -- 172.21.15.110:0/456738647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f551c004530 con 0x5624d4fef230 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.178+0000 7f5412e0d640 1 -- 172.21.15.110:0/456738647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c004870 con 0x5624d4fef230 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/456738647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 msgr2=0x5624d50da490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 --2- 172.21.15.110:0/456738647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50da490 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f551c0017b0 tx=0x7f551c001c70 comp rx=0 tx=0).stop 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/456738647 shutdown_connections 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 --2- 172.21.15.110:0/456738647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50da490 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/456738647 >> 172.21.15.110:0/456738647 conn(0x5624d4feab00 msgr2=0x5624d50d6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/456738647 shutdown_connections 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/456738647 wait complete. 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 Processor -- start 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- start start 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50e1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624d4ff8b60 con 0x5624d4fef230 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5524816640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50e1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5524816640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50e1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51466/0 (socket says 172.21.15.110:51466) 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5524816640 1 -- 172.21.15.110:0/3800516111 learned_addr learned my addr 172.21.15.110:0/3800516111 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5524816640 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d50e1ad0 con 0x5624d4fef230 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5524816640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50e1590 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f551c000f90 tx=0x7f551c001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f541160a640 1 -- 172.21.15.110:0/3800516111 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c020070 con 0x5624d4fef230 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5624d50e1d60 con 0x5624d4fef230 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f541160a640 1 -- 172.21.15.110:0/3800516111 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f551c011400 con 0x5624d4fef230 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f541160a640 1 -- 172.21.15.110:0/3800516111 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551c016040 con 0x5624d4fef230 2024-04-27T03:19:57.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d50e4990 con 0x5624d4fef230 2024-04-27T03:19:57.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f541160a640 1 -- 172.21.15.110:0/3800516111 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f551c023070 con 0x5624d4fef230 2024-04-27T03:19:57.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f541160a640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f540c034650 0x7f540c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f541160a640 1 -- 172.21.15.110:0/3800516111 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f551c049350 con 0x5624d4fef230 2024-04-27T03:19:57.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5524015640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f540c034650 0x7f540c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d4ff3740 0x5624d50e85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.182+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d50e8b20 con 0x5624d4ff3740 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5525017640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d4ff3740 0x5624d50e85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5524015640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f540c034650 0x7f540c036b10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f55180ad000 tx=0x7f55180a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5525017640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d4ff3740 0x5624d50e85e0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5525017640 1 -- 172.21.15.110:0/3800516111 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d4ff3740 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d50e98d0 0x5624d50ebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d50ec2b0 con 0x5624d50e98d0 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5524816640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d50e98d0 0x5624d50ebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5524816640 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d50e98d0 0x5624d50ebcc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5524816640 1 -- 172.21.15.110:0/3800516111 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f551c02c070 con 0x5624d50e98d0 2024-04-27T03:19:57.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d50edb60 con 0x5624d4ff3740 2024-04-27T03:19:57.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5525017640 1 -- 172.21.15.110:0/3800516111 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f55140083f0 con 0x5624d4ff3740 2024-04-27T03:19:57.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d50ecdc0 con 0x5624d50e98d0 2024-04-27T03:19:57.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5524816640 1 -- 172.21.15.110:0/3800516111 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f551c02c070 con 0x5624d50e98d0 2024-04-27T03:19:57.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.186+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d50ed1b0 con 0x5624d4ff3740 2024-04-27T03:19:57.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5525017640 1 -- 172.21.15.110:0/3800516111 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f55140083f0 con 0x5624d4ff3740 2024-04-27T03:19:57.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5526cfb980 1 Processor -- start 2024-04-27T03:19:57.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5526cfb980 1 -- start start 2024-04-27T03:19:57.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5526cfb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4f65a50 0x5624d4f63ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5526cfb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624d4ef6310 con 0x5624d4f65a50 2024-04-27T03:19:57.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5525017640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4f65a50 0x5624d4f63ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5525017640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4f65a50 0x5624d4f63ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51468/0 (socket says 172.21.15.110:51468) 2024-04-27T03:19:57.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5525017640 1 -- 172.21.15.110:0/2776167439 learned_addr learned my addr 172.21.15.110:0/2776167439 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5525017640 1 -- 172.21.15.110:0/2776167439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d4f64430 con 0x5624d4f65a50 2024-04-27T03:19:57.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5525017640 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4f65a50 0x5624d4f63ef0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f551400aab0 tx=0x7f551400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f540a7fc640 1 -- 172.21.15.110:0/2776167439 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5514007d30 con 0x5624d4f65a50 2024-04-27T03:19:57.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f540a7fc640 1 -- 172.21.15.110:0/2776167439 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5514018440 con 0x5624d4f65a50 2024-04-27T03:19:57.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5624d4f64720 con 0x5624d4f65a50 2024-04-27T03:19:57.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f540a7fc640 1 -- 172.21.15.110:0/2776167439 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f55140185f0 con 0x5624d4f65a50 2024-04-27T03:19:57.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d4f64bf0 con 0x5624d4f65a50 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f540a7fc640 1 -- 172.21.15.110:0/2776167439 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5514013070 con 0x5624d4f65a50 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f540a7fc640 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53fc034650 0x7f53fc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f540a7fc640 1 -- 172.21.15.110:0/2776167439 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f551404b8d0 con 0x5624d4f65a50 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5524816640 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53fc034650 0x7f53fc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.190+0000 7f5526cfb980 1 Processor -- start 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- start start 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5524816640 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53fc034650 0x7f53fc036b10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5624d50ec8e0 tx=0x7f551c031040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d510bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624d50dabe0 con 0x5624d510b840 2024-04-27T03:19:57.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d510bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d510bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51484/0 (socket says 172.21.15.110:51484) 2024-04-27T03:19:57.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 -- 172.21.15.110:0/3510020671 learned_addr learned my addr 172.21.15.110:0/3510020671 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 -- 172.21.15.110:0/3510020671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d510c230 con 0x5624d510b840 2024-04-27T03:19:57.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 --2- 172.21.15.110:0/3510020671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d510bc60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5514049ae0 tx=0x7f5514016bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c0dbe6fa471dbd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f53fbfff640 1 -- 172.21.15.110:0/3510020671 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5514032070 con 0x5624d510b840 2024-04-27T03:19:57.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f53fbfff640 1 -- 172.21.15.110:0/3510020671 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55140237f0 con 0x5624d510b840 2024-04-27T03:19:57.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f53fbfff640 1 -- 172.21.15.110:0/3510020671 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5514016330 con 0x5624d510b840 2024-04-27T03:19:57.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- 172.21.15.110:0/3510020671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 msgr2=0x5624d510bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3510020671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d510bc60 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5514049ae0 tx=0x7f5514016bc0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- 172.21.15.110:0/3510020671 shutdown_connections 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3510020671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d510bc60 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- 172.21.15.110:0/3510020671 >> 172.21.15.110:0/3510020671 conn(0x5624d5107b80 msgr2=0x5624d5105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- 172.21.15.110:0/3510020671 shutdown_connections 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- 172.21.15.110:0/3510020671 wait complete. 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 Processor -- start 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- start start 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d5112cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5526cfb980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5624d510c990 con 0x5624d510b840 2024-04-27T03:19:57.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d5112cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d5112cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51494/0 (socket says 172.21.15.110:51494) 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 learned_addr learned my addr 172.21.15.110:0/1049152690 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d5113210 con 0x5624d510b840 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.194+0000 7f5525017640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d5112cd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f5514030040 tx=0x7f5514049a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f53fa7fc640 1 -- 172.21.15.110:0/1049152690 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f551403c020 con 0x5624d510b840 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5624d51134a0 con 0x5624d510b840 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f53fa7fc640 1 -- 172.21.15.110:0/1049152690 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5514033040 con 0x5624d510b840 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f53fa7fc640 1 -- 172.21.15.110:0/1049152690 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5514032070 con 0x5624d510b840 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d5116180 con 0x5624d510b840 2024-04-27T03:19:57.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f53fa7fc640 1 -- 172.21.15.110:0/1049152690 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f551403f020 con 0x5624d510b840 2024-04-27T03:19:57.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f53fa7fc640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53f0034650 0x7f53f0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f53fa7fc640 1 -- 172.21.15.110:0/1049152690 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f5514042da0 con 0x5624d510b840 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5524816640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53f0034650 0x7f53f0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d511e370 0x5624d5120810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5120f60 con 0x5624d511e370 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5524015640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d511e370 0x5624d5120810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5524816640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53f0034650 0x7f53f0036b10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f551c047620 tx=0x7f551c03b000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5524015640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d511e370 0x5624d5120810 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 realm 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5121390 con 0x5624d511e370 2024-04-27T03:19:57.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5121c60 con 0x5624d511e370 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d5122680 0x5624d5124ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5125220 con 0x5624d5122680 2024-04-27T03:19:57.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.198+0000 7f5525017640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d5122680 0x5624d5124ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5525017640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d5122680 0x5624d5124ab0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5526cfb980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5125a70 con 0x5624d511e370 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5526cfb980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5526cfb980 20 searching for the correct realm 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624d51277e0 0x5624d5129bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d51277e0 2024-04-27T03:19:57.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524816640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624d51277e0 0x5624d5129bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524816640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624d51277e0 0x5624d5129bd0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5525017640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5514041020 0x7f551402c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x7f5514041020 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524015640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5514041020 0x7f551402c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524015640 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5514041020 0x7f551402c820 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.202+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d51277e0 2024-04-27T03:19:57.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x7f5514041020 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x7f5514041020 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.206+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d51277e0 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d51277e0 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x7f5514041020 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d51277e0 2024-04-27T03:19:57.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x7f5514041020 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d511e370 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.210+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x7f5514041020 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d51277e0 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x7f5514041020 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d51277e0 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5624d512a200 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:57.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.214+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d51263c0 con 0x5624d511e370 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5126000 con 0x5624d5122680 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5126d20 con 0x5624d511e370 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 zone default found 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 4 Realm: () 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 zonegroup default 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d512ab30 con 0x7f5514041020 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d512c770 con 0x5624d51277e0 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d512cc50 con 0x5624d51277e0 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 started sync module instance, tier type = 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.218+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512b620 con 0x5624d5122680 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94716193583744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512bd80 con 0x5624d5122680 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512d7d0 con 0x5624d511e370 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94716193591968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512d790 con 0x5624d511e370 2024-04-27T03:19:57.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512e6e0 con 0x5624d5122680 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94716193583744] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94716193591968] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94716193594304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512e6e0 con 0x5624d5122680 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.222+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512ef80 con 0x7f5514041020 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94716193598160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512e6e0 con 0x7f5514041020 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512fa40 con 0x5624d5122680 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94716193594304] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94716193598160] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94716193599216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512fa00 con 0x5624d5122680 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.226+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d512fe30 con 0x5624d51277e0 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94716193595920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5130680 con 0x5624d51277e0 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5131080 con 0x5624d51277e0 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94716193599216] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94716193595920] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94716193600784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5131470 con 0x5624d51277e0 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5131d00 con 0x7f5514041020 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'240 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94716193608816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d51318d0 con 0x7f5514041020 2024-04-27T03:19:57.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 20 add_watcher() i=0 2024-04-27T03:19:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 20 add_watcher() i=1 2024-04-27T03:19:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 20 add_watcher() i=2 2024-04-27T03:19:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 20 add_watcher() i=3 2024-04-27T03:19:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 20 add_watcher() i=4 2024-04-27T03:19:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 20 add_watcher() i=5 2024-04-27T03:19:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94716193600784] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.230+0000 7f5526cfb980 20 add_watcher() i=6 2024-04-27T03:19:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94716193608816] v31'241 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 20 add_watcher() i=7 2024-04-27T03:19:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5130b90 con 0x5624d511e370 2024-04-27T03:19:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94716193606544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5132710 con 0x5624d511e370 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94716193606544] v31'245 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f53dafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 5 note: GC not initialized 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f53d9fe3640 20 reqs_thread_entry: start 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f5526cfb980 20 Filter name: none 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.234+0000 7f534b7fe640 20 reqs_thread_entry: start 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d5158a70 con 0x5624d511e370 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 231+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x0 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 cache get: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W : miss 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.18 8:1f190392:users.keys::MH1WC2XQ1S8UISFDZC8W:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d51570b0 con 0x5624d511e370 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 MH1WC2XQ1S8UISFDZC8W [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 206+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 cache put: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W info.flags=0x0 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 adding default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W to cache LRU end 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 cache get: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W : hit (negative entry) 2024-04-27T03:19:57.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 cache get: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W : hit (negative entry) 2024-04-27T03:19:57.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:64 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.set in=58b,writefull 0~473 in=473b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d51579f0 con 0x5624d511e370 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(64 ada [delete,create,call,writefull 0~473] v31'3 uv3 ondisk = 0) ==== 273+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify cookie 94716193764512 in=661b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d515c550 con 0x5624d5122680 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94716193599216 notify 133143986225 ret 0) ==== 691+0+0 (crc 0 0 0) 0x5624d5116180 con 0x5624d5122680 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(65 notify.4 [notify cookie 94716193764512 out=8b] v0'0 uv240 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f53fbfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986225 cookie 94716193599216 notifier 4821 bl.length()=649 2024-04-27T03:19:57.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f53fbfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f53fbfff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f53fbfff640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f53e8003eb0 con 0x5624d5122680 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== watch-notify(notify_complete (2) cookie 94716193764512 notify 133143986225 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5624d51134a0 con 0x5624d5122680 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:67 8.18 8:1f190392:users.keys::MH1WC2XQ1S8UISFDZC8W:head [delete,create,writefull 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d51574a0 con 0x5624d511e370 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.238+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(66 notify.4 [notify-ack] v0'0 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(67 MH1WC2XQ1S8UISFDZC8W [delete,create,writefull 0~7] v31'22 uv22 ondisk = 0) ==== 248+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5526cfb980 10 cache put: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W info.flags=0x7 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5526cfb980 10 moving default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W to cache LRU end 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5526cfb980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:MH1WC2XQ1S8UISFDZC8W, ofs0, ns] 2024-04-27T03:19:57.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 7.15 7:a93a5511:::notify.2:head [notify cookie 94716193764432 in=189b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5624d515c960 con 0x5624d5122680 2024-04-27T03:19:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify (1) cookie 94716193594304 notify 133143986226 ret 0) ==== 219+0+0 (crc 0 0 0) 0x5624d51134a0 con 0x5624d5122680 2024-04-27T03:19:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f53fbfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986226 cookie 94716193594304 notifier 4821 bl.length()=177 2024-04-27T03:19:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f53fbfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W info.flags=0x7 2024-04-27T03:19:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f53fbfff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+MH1WC2XQ1S8UISFDZC8W to cache LRU end 2024-04-27T03:19:57.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(68 notify.2 [notify cookie 94716193764432 out=8b] v0'0 uv240 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f53fbfff640 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f53e8003eb0 con 0x5624d5122680 2024-04-27T03:19:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== watch-notify(notify_complete (2) cookie 94716193764432 notify 133143986226 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5624d5113210 con 0x5624d5122680 2024-04-27T03:19:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(69 notify.2 [notify-ack] v0'0 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.255 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"users","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f53dafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.242+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94716193606544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x5624d511e370 2024-04-27T03:19:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.246+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94716193606544] v31'246 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.246+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94716193583744] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x5624d5122680 2024-04-27T03:19:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94716193583744] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5526cfb980 20 remove_watcher() i=0 2024-04-27T03:19:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5526cfb980 2 removed watcher, disabling cache 2024-04-27T03:19:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94716193591968] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x5624d511e370 2024-04-27T03:19:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94716193591968] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x5624d511e370 2024-04-27T03:19:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5526cfb980 20 remove_watcher() i=1 2024-04-27T03:19:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94716193594304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x5624d5122680 2024-04-27T03:19:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94716193594304] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5526cfb980 20 remove_watcher() i=2 2024-04-27T03:19:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.250+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94716193598160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x7f5514041020 2024-04-27T03:19:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.254+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94716193598160] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.254+0000 7f5526cfb980 20 remove_watcher() i=3 2024-04-27T03:19:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.254+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94716193599216] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x5624d5122680 2024-04-27T03:19:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.254+0000 7f5525017640 1 -- 172.21.15.110:0/1049152690 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94716193599216] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55140083f0 con 0x5624d5122680 2024-04-27T03:19:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.254+0000 7f5526cfb980 20 remove_watcher() i=4 2024-04-27T03:19:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.254+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94716193595920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x5624d51277e0 2024-04-27T03:19:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94716193595920] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 20 remove_watcher() i=5 2024-04-27T03:19:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94716193600784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x5624d51277e0 2024-04-27T03:19:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5524816640 1 -- 172.21.15.110:0/1049152690 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94716193600784] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f551c02c070 con 0x5624d51277e0 2024-04-27T03:19:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 20 remove_watcher() i=6 2024-04-27T03:19:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94716193608816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5624d5156710 con 0x7f5514041020 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94716193608816] v31'242 uv240 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f55180b3040 con 0x7f5514041020 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 20 remove_watcher() i=7 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d5122680 msgr2=0x5624d5124ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d5122680 0x5624d5124ab0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d511e370 msgr2=0x5624d5120810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d511e370 0x5624d5120810 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5514041020 msgr2=0x7f551402c820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f5514041020 0x7f551402c820 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624d51277e0 msgr2=0x5624d5129bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5624d51277e0 0x5624d5129bd0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53f0034650 msgr2=0x7f53f0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53f0034650 0x7f53f0036b10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f551c047620 tx=0x7f551c03b000 comp rx=0 tx=0).stop 2024-04-27T03:19:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 msgr2=0x5624d5112cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d5112cd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f5514030040 tx=0x7f5514049a70 comp rx=0 tx=0).stop 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5524015640 1 -- 172.21.15.110:0/1049152690 reap_dead start 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 shutdown_connections 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/1049152690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d510b840 0x5624d5112cd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 >> 172.21.15.110:0/1049152690 conn(0x5624d5107b80 msgr2=0x5624d51051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 shutdown_connections 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/1049152690 wait complete. 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53fc034650 msgr2=0x7f53fc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53fc034650 0x7f53fc036b10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x5624d50ec8e0 tx=0x7f551c031040 comp rx=0 tx=0).stop 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4f65a50 msgr2=0x5624d4f63ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4f65a50 0x5624d4f63ef0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f551400aab0 tx=0x7f551400af70 comp rx=0 tx=0).stop 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 shutdown_connections 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f53fc034650 0x7f53fc036b10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 --2- 172.21.15.110:0/2776167439 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4f65a50 0x5624d4f63ef0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 >> 172.21.15.110:0/2776167439 conn(0x5624d50f1240 msgr2=0x5624d50f1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 shutdown_connections 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.258+0000 7f5526cfb980 1 -- 172.21.15.110:0/2776167439 wait complete. 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d50e98d0 msgr2=0x5624d50ebcc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d50e98d0 0x5624d50ebcc0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d4ff3740 msgr2=0x5624d50e85e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d4ff3740 0x5624d50e85e0 crc :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f540c034650 msgr2=0x7f540c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f540c034650 0x7f540c036b10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f55180ad000 tx=0x7f55180a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 msgr2=0x5624d50e1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50e1590 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f551c000f90 tx=0x7f551c001260 comp rx=0 tx=0).stop 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 shutdown_connections 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5624d50e98d0 0x5624d50ebcc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5624d4ff3740 0x5624d50e85e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f540c034650 0x7f540c036b10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 --2- 172.21.15.110:0/3800516111 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5624d4fef230 0x5624d50e1590 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 >> 172.21.15.110:0/3800516111 conn(0x5624d4feab00 msgr2=0x5624d50d4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 shutdown_connections 2024-04-27T03:19:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.262+0000 7f5526cfb980 1 -- 172.21.15.110:0/3800516111 wait complete. 2024-04-27T03:19:57.282 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:57.282 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:57.282 ERROR:root:0 2024-04-27T03:19:57.283 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:57.289 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchUser', 'Message': '', 'RequestId': 'tx0000051d994df8014c1a8-00662c6edd-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:19:57.290 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'create'] {'uid': 'foo', 'display-name': 'Foo', 'email': 'foo@foo.com', 'access-key': '9te6NH5mcdcq0Tc5i8i1', 'secret-key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'max-buckets': '4'} 2024-04-27T03:19:57.303 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:57.303 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'list'] {'list': '', 'max-entries': 0} 2024-04-27T03:19:57.320 INFO:tasks.radosgw_admin_rest: json result: {'keys': [], 'truncated': True, 'count': 0, 'marker': '8:1e6ca5ed:users.uid::foo:head'} 2024-04-27T03:19:57.321 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'list'] {'list': '', 'max-entries': 1} 2024-04-27T03:19:57.342 INFO:tasks.radosgw_admin_rest: json result: {'keys': ['foo'], 'truncated': True, 'count': 1, 'marker': '8:a00000ee:users.uid::bar.buckets:head'} 2024-04-27T03:19:57.343 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'list'] {'list': '', 'max-entries': 1, 'marker': '8:a00000ee:users.uid::bar.buckets:head'} 2024-04-27T03:19:57.353 INFO:tasks.radosgw_admin_rest: json result: {'keys': ['ada'], 'truncated': False, 'count': 1} 2024-04-27T03:19:57.354 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:57.358 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:57.358 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'access-key': '9te6NH5mcdcq0Tc5i8i1', 'uid': 'uid_not_exist'} 2024-04-27T03:19:57.362 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:57.363 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'rm', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:57.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'rm', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:57.363 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps rm --uid ada --caps 'users=read, write' 2024-04-27T03:19:57.398 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:57.398 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6e0ba4980 1 Processor -- start 2024-04-27T03:19:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6e0ba4980 1 -- start start 2024-04-27T03:19:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6e0ba4980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55aa782026c0 0x55aa78204ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6e0ba4980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55aa780be0d0 con 0x55aa781fe160 2024-04-27T03:19:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6e0ba4980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55aa780b3430 con 0x55aa782026c0 2024-04-27T03:19:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6ddebe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55aa782026c0 0x55aa78204ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6de6bf640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55aa781fe160 0x55aa781fe540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44550/0 (socket says 172.21.15.110:44550) 2024-04-27T03:19:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6de6bf640 1 -- 172.21.15.110:0/610069051 learned_addr learned my addr 172.21.15.110:0/610069051 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6ddebe640 1 -- 172.21.15.110:0/610069051 >> v1:172.21.15.77:6789/0 conn(0x55aa781fe160 legacy=0x55aa781fe540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.390+0000 7fe6ddebe640 1 -- 172.21.15.110:0/610069051 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa780ac510 con 0x55aa782026c0 2024-04-27T03:19:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6ddebe640 1 --2- 172.21.15.110:0/610069051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa782026c0 0x55aa78204ab0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400a480 tx=0x7fe6d401b110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a2a3c2bad3ed77 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6dd6bd640 1 -- 172.21.15.110:0/610069051 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d40af070 con 0x55aa782026c0 2024-04-27T03:19:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6dd6bd640 1 -- 172.21.15.110:0/610069051 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d40a3e50 con 0x55aa782026c0 2024-04-27T03:19:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6dd6bd640 1 -- 172.21.15.110:0/610069051 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d40aa680 con 0x55aa782026c0 2024-04-27T03:19:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/610069051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa782026c0 msgr2=0x55aa78204ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/610069051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa782026c0 0x55aa78204ab0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe6d400a480 tx=0x7fe6d401b110 comp rx=0 tx=0).stop 2024-04-27T03:19:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/610069051 shutdown_connections 2024-04-27T03:19:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/610069051 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa782026c0 0x55aa78204ab0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/610069051 >> 172.21.15.110:0/610069051 conn(0x55aa781f9a30 msgr2=0x55aa781fbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/610069051 shutdown_connections 2024-04-27T03:19:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.394+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/610069051 wait complete. 2024-04-27T03:19:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6e0ba4980 1 Processor -- start 2024-04-27T03:19:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6e0ba4980 1 -- start start 2024-04-27T03:19:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6e0ba4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782e8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6e0ba4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aa781022d0 con 0x55aa781fe160 2024-04-27T03:19:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6de6bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6de6bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782e8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51516/0 (socket says 172.21.15.110:51516) 2024-04-27T03:19:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6de6bf640 1 -- 172.21.15.110:0/614223142 learned_addr learned my addr 172.21.15.110:0/614223142 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6de6bf640 1 -- 172.21.15.110:0/614223142 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa78088ae0 con 0x55aa781fe160 2024-04-27T03:19:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/614223142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782e8280 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8001860 tx=0x7fe6d8001d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67825e55a227f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe5cc608640 1 -- 172.21.15.110:0/614223142 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d8003460 con 0x55aa781fe160 2024-04-27T03:19:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe5cc608640 1 -- 172.21.15.110:0/614223142 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d8003600 con 0x55aa781fe160 2024-04-27T03:19:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe5cc608640 1 -- 172.21.15.110:0/614223142 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d8003920 con 0x55aa781fe160 2024-04-27T03:19:57.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/614223142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 msgr2=0x55aa782e8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.410+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/614223142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782e8280 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8001860 tx=0x7fe6d8001d20 comp rx=0 tx=0).stop 2024-04-27T03:19:57.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/614223142 shutdown_connections 2024-04-27T03:19:57.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/614223142 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782e8280 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/614223142 >> 172.21.15.110:0/614223142 conn(0x55aa781f9a30 msgr2=0x55aa782e3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/614223142 shutdown_connections 2024-04-27T03:19:57.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/614223142 wait complete. 2024-04-27T03:19:57.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 Processor -- start 2024-04-27T03:19:57.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- start start 2024-04-27T03:19:57.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aa78172f20 con 0x55aa781fe160 2024-04-27T03:19:57.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6de6bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6de6bf640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782ef320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51520/0 (socket says 172.21.15.110:51520) 2024-04-27T03:19:57.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6de6bf640 1 -- 172.21.15.110:0/3911251471 learned_addr learned my addr 172.21.15.110:0/3911251471 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6de6bf640 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa782ef860 con 0x55aa781fe160 2024-04-27T03:19:57.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782ef320 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8001d60 tx=0x7fe6d8000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe5cae05640 1 -- 172.21.15.110:0/3911251471 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d8013760 con 0x55aa781fe160 2024-04-27T03:19:57.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55aa782efaf0 con 0x55aa781fe160 2024-04-27T03:19:57.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe5cae05640 1 -- 172.21.15.110:0/3911251471 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d8023070 con 0x55aa781fe160 2024-04-27T03:19:57.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe5cae05640 1 -- 172.21.15.110:0/3911251471 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d8013af0 con 0x55aa781fe160 2024-04-27T03:19:57.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa782f2770 con 0x55aa781fe160 2024-04-27T03:19:57.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe5cae05640 1 -- 172.21.15.110:0/3911251471 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe6d8010070 con 0x55aa781fe160 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe5cae05640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5c4034650 0x7fe5c4036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe5cae05640 1 -- 172.21.15.110:0/3911251471 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fe6d8049e50 con 0x55aa781fe160 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.414+0000 7fe6ddebe640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5c4034650 0x7fe5c4036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa782026c0 0x55aa782f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa782f6830 con 0x55aa782026c0 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6deec0640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa782026c0 0x55aa782f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6ddebe640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5c4034650 0x7fe5c4036b10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40ac000 tx=0x7fe6d40a5000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6deec0640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa782026c0 0x55aa782f6250 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6deec0640 1 -- 172.21.15.110:0/3911251471 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe6d00083f0 con 0x55aa782026c0 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa782f7620 0x55aa782f9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa782fa050 con 0x55aa782f7620 2024-04-27T03:19:57.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa782f7620 0x55aa782f9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa782f7620 0x55aa782f9a10 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6de6bf640 1 -- 172.21.15.110:0/3911251471 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa782f7620 2024-04-27T03:19:57.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa782fb900 con 0x55aa782026c0 2024-04-27T03:19:57.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6deec0640 1 -- 172.21.15.110:0/3911251471 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fe6d00083f0 con 0x55aa782026c0 2024-04-27T03:19:57.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa782fab30 con 0x55aa782f7620 2024-04-27T03:19:57.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6de6bf640 1 -- 172.21.15.110:0/3911251471 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa782f7620 2024-04-27T03:19:57.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa782faf20 con 0x55aa782026c0 2024-04-27T03:19:57.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.418+0000 7fe6deec0640 1 -- 172.21.15.110:0/3911251471 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fe6d00083f0 con 0x55aa782026c0 2024-04-27T03:19:57.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6e0ba4980 1 Processor -- start 2024-04-27T03:19:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6e0ba4980 1 -- start start 2024-04-27T03:19:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6e0ba4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7830e610 0x55aa7830ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6e0ba4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aa78106310 con 0x55aa7830e610 2024-04-27T03:19:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6deec0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7830e610 0x55aa7830ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6deec0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7830e610 0x55aa7830ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51526/0 (socket says 172.21.15.110:51526) 2024-04-27T03:19:57.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6deec0640 1 -- 172.21.15.110:0/1854328075 learned_addr learned my addr 172.21.15.110:0/1854328075 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6deec0640 1 -- 172.21.15.110:0/1854328075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7830ef70 con 0x55aa7830e610 2024-04-27T03:19:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6deec0640 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7830e610 0x55aa7830ea30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000aab0 tx=0x7fe6d000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe5c3fff640 1 -- 172.21.15.110:0/1854328075 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0007d30 con 0x55aa7830e610 2024-04-27T03:19:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe5c3fff640 1 -- 172.21.15.110:0/1854328075 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d001d070 con 0x55aa7830e610 2024-04-27T03:19:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55aa7830fa10 con 0x55aa7830e610 2024-04-27T03:19:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe5c3fff640 1 -- 172.21.15.110:0/1854328075 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0018440 con 0x55aa7830e610 2024-04-27T03:19:57.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa7830fee0 con 0x55aa7830e610 2024-04-27T03:19:57.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe5c3fff640 1 -- 172.21.15.110:0/1854328075 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe6d0013070 con 0x55aa7830e610 2024-04-27T03:19:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe5c3fff640 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5b8034650 0x7fe5b8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe5c3fff640 1 -- 172.21.15.110:0/1854328075 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fe6d004b720 con 0x55aa7830e610 2024-04-27T03:19:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.422+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5b8034650 0x7fe5b8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 Processor -- start 2024-04-27T03:19:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- start start 2024-04-27T03:19:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa7831bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5b8034650 0x7fe5b8036b10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55aa782fa650 tx=0x7fe6d80480b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aa782e89d0 con 0x55aa7831b810 2024-04-27T03:19:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa7831bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa7831bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51530/0 (socket says 172.21.15.110:51530) 2024-04-27T03:19:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 -- 172.21.15.110:0/4267471406 learned_addr learned my addr 172.21.15.110:0/4267471406 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 -- 172.21.15.110:0/4267471406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa7831c1e0 con 0x55aa7831b810 2024-04-27T03:19:57.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 --2- 172.21.15.110:0/4267471406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa7831bc10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0049930 tx=0x7fe6d0049b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756e8afe831629c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe5c1ffb640 1 -- 172.21.15.110:0/4267471406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0047070 con 0x55aa7831b810 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe5c1ffb640 1 -- 172.21.15.110:0/4267471406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d0022460 con 0x55aa7831b810 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe5c1ffb640 1 -- 172.21.15.110:0/4267471406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0018bc0 con 0x55aa7831b810 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/4267471406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 msgr2=0x55aa7831bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/4267471406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa7831bc10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0049930 tx=0x7fe6d0049b30 comp rx=0 tx=0).stop 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/4267471406 shutdown_connections 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/4267471406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa7831bc10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/4267471406 >> 172.21.15.110:0/4267471406 conn(0x55aa78317b50 msgr2=0x55aa78315bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/4267471406 shutdown_connections 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/4267471406 wait complete. 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 Processor -- start 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- start start 2024-04-27T03:19:57.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa78322cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6e0ba4980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55aa7831c980 con 0x55aa7831b810 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa78322cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa78322cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51546/0 (socket says 172.21.15.110:51546) 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 learned_addr learned my addr 172.21.15.110:0/239968153 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.426+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa78323230 con 0x55aa7831b810 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6deec0640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa78322cf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0049930 tx=0x7fe6d00498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe5affff640 1 -- 172.21.15.110:0/239968153 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0031070 con 0x55aa7831b810 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe5affff640 1 -- 172.21.15.110:0/239968153 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d0020900 con 0x55aa7831b810 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe5affff640 1 -- 172.21.15.110:0/239968153 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6d0017b70 con 0x55aa7831b810 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55aa783234c0 con 0x55aa7831b810 2024-04-27T03:19:57.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa783261a0 con 0x55aa7831b810 2024-04-27T03:19:57.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe5affff640 1 -- 172.21.15.110:0/239968153 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe6d0034070 con 0x55aa7831b810 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe5affff640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5a80346a0 0x7fe5a8036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe5affff640 1 -- 172.21.15.110:0/239968153 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fe6d0037dd0 con 0x55aa7831b810 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5a80346a0 0x7fe5a8036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa7832e330 0x55aa783307c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78330f10 con 0x55aa7832e330 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6ddebe640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa7832e330 0x55aa783307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5a80346a0 0x7fe5a8036b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80127b0 tx=0x7fe6d8038000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6ddebe640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa7832e330 0x55aa783307c0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 20 realm 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78331340 con 0x55aa7832e330 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.430+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78331c10 con 0x55aa7832e330 2024-04-27T03:19:57.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa78332630 0x55aa78334a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa783351d0 con 0x55aa78332630 2024-04-27T03:19:57.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6deec0640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa78332630 0x55aa78334a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6deec0640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa78332630 0x55aa78334a60 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78335a20 con 0x55aa7832e330 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 20 searching for the correct realm 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55aa78337790 0x55aa78339b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78337790 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55aa78337790 0x55aa78339b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6de6bf640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55aa78337790 0x55aa78339b80 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6deec0640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe6d0056040 0x7fe6d003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x7fe6d0056040 2024-04-27T03:19:57.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.434+0000 7fe6ddebe640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe6d0056040 0x7fe6d003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe6d0056040 0x7fe6d003b620 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78337790 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x7fe6d0056040 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.438+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x7fe6d0056040 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78337790 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78337790 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x7fe6d0056040 2024-04-27T03:19:57.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.442+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78337790 2024-04-27T03:19:57.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x7fe6d0056040 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa7832e330 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x7fe6d0056040 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78337790 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x7fe6d0056040 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78337790 2024-04-27T03:19:57.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.446+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a190 con 0x55aa78332630 2024-04-27T03:19:57.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:57.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:57.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa783363a0 con 0x55aa7832e330 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78335fb0 con 0x55aa78332630 2024-04-27T03:19:57.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78336c90 con 0x55aa7832e330 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 zone default found 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 4 Realm: () 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 zonegroup default 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833a610 con 0x7fe6d0056040 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833c410 con 0x55aa78337790 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833c830 con 0x55aa78337790 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 started sync module instance, tier type = 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.450+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833b8b0 con 0x55aa78332630 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.454+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.454+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94190649457824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833d140 con 0x55aa78332630 2024-04-27T03:19:57.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.454+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833d990 con 0x55aa7832e330 2024-04-27T03:19:57.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.454+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.454+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94190649463104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833da90 con 0x55aa7832e330 2024-04-27T03:19:57.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.454+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833e590 con 0x55aa78332630 2024-04-27T03:19:57.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.454+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94190649457824] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94190649463104] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94190649464160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833cc90 con 0x55aa78332630 2024-04-27T03:19:57.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833ea40 con 0x7fe6d0056040 2024-04-27T03:19:57.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94190649468240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833e970 con 0x7fe6d0056040 2024-04-27T03:19:57.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833f900 con 0x55aa78332630 2024-04-27T03:19:57.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94190649464160] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.458+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94190649468240] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94190649470512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833f450 con 0x55aa78332630 2024-04-27T03:19:57.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa783401a0 con 0x55aa78337790 2024-04-27T03:19:57.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94190649474288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7833f450 con 0x55aa78337790 2024-04-27T03:19:57.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78341010 con 0x55aa78337790 2024-04-27T03:19:57.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94190649470512] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94190649474288] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94190649467008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78341010 con 0x55aa78337790 2024-04-27T03:19:57.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.462+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78341860 con 0x7fe6d0056040 2024-04-27T03:19:57.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'243 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94190649467008] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94190649479664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78341be0 con 0x7fe6d0056040 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=0 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=1 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=2 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=3 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=4 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=5 2024-04-27T03:19:57.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=6 2024-04-27T03:19:57.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94190649479664] v31'244 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 20 add_watcher() i=7 2024-04-27T03:19:57.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:57.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78340c50 con 0x55aa7832e330 2024-04-27T03:19:57.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.466+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94190649480848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78340c50 con 0x55aa7832e330 2024-04-27T03:19:57.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94190649480848] v31'247 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe5957e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:57.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:57.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 5 note: GC not initialized 2024-04-27T03:19:57.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe58f7fe640 20 reqs_thread_entry: start 2024-04-27T03:19:57.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:57.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 20 Filter name: none 2024-04-27T03:19:57.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe58d7fa640 20 reqs_thread_entry: start 2024-04-27T03:19:57.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:57.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa78368440 con 0x55aa7832e330 2024-04-27T03:19:57.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~473 out=473b,getxattrs out=75b] v0'0 uv3 ondisk = 0) ==== 231+0+596 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 20 rados_obj.operate() r=0 bl.length=473 2024-04-27T03:19:57.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:57.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:57.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:57.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~460 in=460b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa7836ed40 con 0x55aa7832e330 2024-04-27T03:19:57.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~460] v31'5 uv5 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:57.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:57.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.470+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94190649665456 in=648b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55aa7836a450 con 0x55aa78332630 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94190649470512 notify 133143986228 ret 0) ==== 678+0+0 (crc 0 0 0) 0x55aa783261a0 con 0x55aa78332630 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe5c1ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986228 cookie 94190649470512 notifier 4828 bl.length()=636 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe5c1ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe5c1ffb640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe5c1ffb640 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fe5a4003eb0 con 0x55aa78332630 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94190649665456 out=8b] v0'0 uv243 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94190649665456 notify 133143986228 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55aa783234c0 con 0x55aa78332630 2024-04-27T03:19:57.485 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:57.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe5957e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.474+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94190649480848] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x55aa7832e330 2024-04-27T03:19:57.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.478+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94190649480848] v31'248 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.478+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94190649457824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x55aa78332630 2024-04-27T03:19:57.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.478+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94190649457824] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.478+0000 7fe6e0ba4980 20 remove_watcher() i=0 2024-04-27T03:19:57.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.478+0000 7fe6e0ba4980 2 removed watcher, disabling cache 2024-04-27T03:19:57.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.478+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94190649463104] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x55aa7832e330 2024-04-27T03:19:57.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94190649463104] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x55aa7832e330 2024-04-27T03:19:57.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6e0ba4980 20 remove_watcher() i=1 2024-04-27T03:19:57.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94190649464160] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x55aa78332630 2024-04-27T03:19:57.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94190649464160] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6e0ba4980 20 remove_watcher() i=2 2024-04-27T03:19:57.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94190649468240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x7fe6d0056040 2024-04-27T03:19:57.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94190649468240] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6e0ba4980 20 remove_watcher() i=3 2024-04-27T03:19:57.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.482+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94190649470512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x55aa78332630 2024-04-27T03:19:57.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.486+0000 7fe6deec0640 1 -- 172.21.15.110:0/239968153 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94190649470512] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d0033030 con 0x55aa78332630 2024-04-27T03:19:57.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.486+0000 7fe6e0ba4980 20 remove_watcher() i=4 2024-04-27T03:19:57.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.486+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94190649474288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x55aa78337790 2024-04-27T03:19:57.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.486+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94190649474288] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.486+0000 7fe6e0ba4980 20 remove_watcher() i=5 2024-04-27T03:19:57.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.486+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94190649467008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x55aa78337790 2024-04-27T03:19:57.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6de6bf640 1 -- 172.21.15.110:0/239968153 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94190649467008] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d8049040 con 0x55aa78337790 2024-04-27T03:19:57.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 20 remove_watcher() i=6 2024-04-27T03:19:57.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94190649479664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55aa78366140 con 0x7fe6d0056040 2024-04-27T03:19:57.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94190649479664] v31'245 uv243 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe6d40b2040 con 0x7fe6d0056040 2024-04-27T03:19:57.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 20 remove_watcher() i=7 2024-04-27T03:19:57.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa78332630 msgr2=0x55aa78334a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa78332630 0x55aa78334a60 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa7832e330 msgr2=0x55aa783307c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa7832e330 0x55aa783307c0 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe6d0056040 msgr2=0x7fe6d003b620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe6d0056040 0x7fe6d003b620 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55aa78337790 msgr2=0x55aa78339b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55aa78337790 0x55aa78339b80 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5a80346a0 msgr2=0x7fe5a8036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5a80346a0 0x7fe5a8036b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80127b0 tx=0x7fe6d8038000 comp rx=0 tx=0).stop 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 msgr2=0x55aa78322cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6ddebe640 1 -- 172.21.15.110:0/239968153 reap_dead start 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa78322cf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0049930 tx=0x7fe6d00498c0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 shutdown_connections 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/239968153 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7831b810 0x55aa78322cf0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 >> 172.21.15.110:0/239968153 conn(0x55aa78317b50 msgr2=0x55aa783151b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 shutdown_connections 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/239968153 wait complete. 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5b8034650 msgr2=0x7fe5b8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5b8034650 0x7fe5b8036b10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55aa782fa650 tx=0x7fe6d80480b0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7830e610 msgr2=0x55aa7830ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7830e610 0x55aa7830ea30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000aab0 tx=0x7fe6d000af70 comp rx=0 tx=0).stop 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 shutdown_connections 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5b8034650 0x7fe5b8036b10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/1854328075 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa7830e610 0x55aa7830ea30 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 >> 172.21.15.110:0/1854328075 conn(0x55aa782ff010 msgr2=0x55aa782ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 shutdown_connections 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/1854328075 wait complete. 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa782f7620 msgr2=0x55aa782f9a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa782f7620 0x55aa782f9a10 crc :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa782026c0 msgr2=0x55aa782f6250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa782026c0 0x55aa782f6250 crc :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5c4034650 msgr2=0x7fe5c4036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5c4034650 0x7fe5c4036b10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40ac000 tx=0x7fe6d40a5000 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 msgr2=0x55aa782ef320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782ef320 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8001d60 tx=0x7fe6d8000f50 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 shutdown_connections 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55aa782f7620 0x55aa782f9a10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55aa782026c0 0x55aa782f6250 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe5c4034650 0x7fe5c4036b10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 --2- 172.21.15.110:0/3911251471 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55aa781fe160 0x55aa782ef320 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 >> 172.21.15.110:0/3911251471 conn(0x55aa781f9a30 msgr2=0x55aa782e2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 shutdown_connections 2024-04-27T03:19:57.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.490+0000 7fe6e0ba4980 1 -- 172.21.15.110:0/3911251471 wait complete. 2024-04-27T03:19:57.513 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:57.513 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:57.513 ERROR:root:0 2024-04-27T03:19:57.513 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'system_user', '--display-name', 'System User', '--access-key', 'system_akey', '--secret', 'system_skey', '--caps', 'user-info-without-keys=read', '--system'] 2024-04-27T03:19:57.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'system_user', '--display-name', 'System User', '--access-key', 'system_akey', '--secret', 'system_skey', '--caps', 'user-info-without-keys=read', '--system'] 2024-04-27T03:19:57.513 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid system_user --display-name 'System User' --access-key system_akey --secret system_skey --caps user-info-without-keys=read --system 2024-04-27T03:19:57.601 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:57.601 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:57.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fce99980 1 Processor -- start 2024-04-27T03:19:57.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fce99980 1 -- start start 2024-04-27T03:19:57.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fce99980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ce501336d0 0x55ce50133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fce99980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55ce4fff40d0 con 0x55ce50133ff0 2024-04-27T03:19:57.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fce99980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55ce4ffe9430 con 0x55ce501336d0 2024-04-27T03:19:57.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fa9b4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55ce501336d0 0x55ce50133ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fa1b3640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55ce50133ff0 0x55ce5013a7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44558/0 (socket says 172.21.15.110:44558) 2024-04-27T03:19:57.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.590+0000 7f75fa1b3640 1 -- 172.21.15.110:0/2163580130 learned_addr learned my addr 172.21.15.110:0/2163580130 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:57.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fa9b4640 1 -- 172.21.15.110:0/2163580130 >> v1:172.21.15.77:6789/0 conn(0x55ce50133ff0 legacy=0x55ce5013a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fa9b4640 1 -- 172.21.15.110:0/2163580130 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce4ffe2510 con 0x55ce501336d0 2024-04-27T03:19:57.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/2163580130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50133ab0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f75f400a480 tx=0x7f75f401b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cae54ac3fb980e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75f99b2640 1 -- 172.21.15.110:0/2163580130 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f40b0070 con 0x55ce501336d0 2024-04-27T03:19:57.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75f99b2640 1 -- 172.21.15.110:0/2163580130 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f40ab540 con 0x55ce501336d0 2024-04-27T03:19:57.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75f99b2640 1 -- 172.21.15.110:0/2163580130 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f40ab840 con 0x55ce501336d0 2024-04-27T03:19:57.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fce99980 1 -- 172.21.15.110:0/2163580130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 msgr2=0x55ce50133ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fce99980 1 --2- 172.21.15.110:0/2163580130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50133ab0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f75f400a480 tx=0x7f75f401b280 comp rx=0 tx=0).stop 2024-04-27T03:19:57.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fce99980 1 -- 172.21.15.110:0/2163580130 shutdown_connections 2024-04-27T03:19:57.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fce99980 1 --2- 172.21.15.110:0/2163580130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50133ab0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fce99980 1 -- 172.21.15.110:0/2163580130 >> 172.21.15.110:0/2163580130 conn(0x55ce5012f160 msgr2=0x55ce50131580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fce99980 1 -- 172.21.15.110:0/2163580130 shutdown_connections 2024-04-27T03:19:57.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.594+0000 7f75fce99980 1 -- 172.21.15.110:0/2163580130 wait complete. 2024-04-27T03:19:57.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.610+0000 7f75fce99980 1 Processor -- start 2024-04-27T03:19:57.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.610+0000 7f75fce99980 1 -- start start 2024-04-27T03:19:57.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.610+0000 7f75fce99980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50220400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.610+0000 7f75fce99980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce500382d0 con 0x55ce501336d0 2024-04-27T03:19:57.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.610+0000 7f75fa9b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50220400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.610+0000 7f75fa9b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50220400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51560/0 (socket says 172.21.15.110:51560) 2024-04-27T03:19:57.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.610+0000 7f75fa9b4640 1 -- 172.21.15.110:0/3676945625 learned_addr learned my addr 172.21.15.110:0/3676945625 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fa9b4640 1 -- 172.21.15.110:0/3676945625 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce4ffbeae0 con 0x55ce501336d0 2024-04-27T03:19:57.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/3676945625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50220400 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f75f40ae040 tx=0x7f75f4003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa848f0c3fad7fa server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f74e8e09640 1 -- 172.21.15.110:0/3676945625 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f40b0040 con 0x55ce501336d0 2024-04-27T03:19:57.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f74e8e09640 1 -- 172.21.15.110:0/3676945625 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f40ab9b0 con 0x55ce501336d0 2024-04-27T03:19:57.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f74e8e09640 1 -- 172.21.15.110:0/3676945625 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f40abc90 con 0x55ce501336d0 2024-04-27T03:19:57.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- 172.21.15.110:0/3676945625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 msgr2=0x55ce50220400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 --2- 172.21.15.110:0/3676945625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50220400 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f75f40ae040 tx=0x7f75f4003030 comp rx=0 tx=0).stop 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- 172.21.15.110:0/3676945625 shutdown_connections 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 --2- 172.21.15.110:0/3676945625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50220400 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- 172.21.15.110:0/3676945625 >> 172.21.15.110:0/3676945625 conn(0x55ce5012f160 msgr2=0x55ce5021bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- 172.21.15.110:0/3676945625 shutdown_connections 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- 172.21.15.110:0/3676945625 wait complete. 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 Processor -- start 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- start start 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce5013ca10 con 0x55ce501336d0 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fa9b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fa9b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50135e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51574/0 (socket says 172.21.15.110:51574) 2024-04-27T03:19:57.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fa9b4640 1 -- 172.21.15.110:0/821424461 learned_addr learned my addr 172.21.15.110:0/821424461 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fa9b4640 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce50136390 con 0x55ce501336d0 2024-04-27T03:19:57.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50135e50 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f75f401bb20 tx=0x7f75f4018020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f74e7606640 1 -- 172.21.15.110:0/821424461 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f40c5070 con 0x55ce501336d0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ce501346f0 con 0x55ce501336d0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f74e7606640 1 -- 172.21.15.110:0/821424461 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f40b6970 con 0x55ce501336d0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.614+0000 7f74e7606640 1 -- 172.21.15.110:0/821424461 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75f40b0050 con 0x55ce501336d0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce50134be0 con 0x55ce501336d0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f74e7606640 1 -- 172.21.15.110:0/821424461 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f75f40c8070 con 0x55ce501336d0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f74e7606640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74e0034650 0x7f74e0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f74e7606640 1 -- 172.21.15.110:0/821424461 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f75f40bc070 con 0x55ce501336d0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fa1b3640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74e0034650 0x7f74e0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50133ff0 0x55ce502308a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50230de0 con 0x55ce50133ff0 2024-04-27T03:19:57.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50133ff0 0x55ce502308a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fa1b3640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74e0034650 0x7f74e0036b10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f75e4001da0 tx=0x7f75e4001300 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50133ff0 0x55ce502308a0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fb1b5640 1 -- 172.21.15.110:0/821424461 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f75ec00a4d0 con 0x55ce50133ff0 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50231690 0x55ce50233a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50234180 con 0x55ce50231690 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50231690 0x55ce50233a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.618+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50231690 0x55ce50233a80 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fa9b4640 1 -- 172.21.15.110:0/821424461 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f75f40c7030 con 0x55ce50231690 2024-04-27T03:19:57.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50235a30 con 0x55ce50133ff0 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fb1b5640 1 -- 172.21.15.110:0/821424461 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f75ec00a4d0 con 0x55ce50133ff0 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50234c60 con 0x55ce50231690 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fa9b4640 1 -- 172.21.15.110:0/821424461 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce50231690 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50235050 con 0x55ce50133ff0 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fb1b5640 1 -- 172.21.15.110:0/821424461 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f75ec00a4d0 con 0x55ce50133ff0 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 Processor -- start 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 -- start start 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce50244350 0x55ce50244770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce5003c310 con 0x55ce50244350 2024-04-27T03:19:57.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fb1b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce50244350 0x55ce50244770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fb1b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce50244350 0x55ce50244770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51588/0 (socket says 172.21.15.110:51588) 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fb1b5640 1 -- 172.21.15.110:0/1944237467 learned_addr learned my addr 172.21.15.110:0/1944237467 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fb1b5640 1 -- 172.21.15.110:0/1944237467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce50244cb0 con 0x55ce50244350 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce50244350 0x55ce50244770 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f75ec017680 tx=0x7f75ec017a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f74e4e01640 1 -- 172.21.15.110:0/1944237467 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75ec0157d0 con 0x55ce50244350 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f74e4e01640 1 -- 172.21.15.110:0/1944237467 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec015970 con 0x55ce50244350 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ce50245750 con 0x55ce50244350 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f74e4e01640 1 -- 172.21.15.110:0/1944237467 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75ec015c50 con 0x55ce50244350 2024-04-27T03:19:57.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.622+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce50245c20 con 0x55ce50244350 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f74e4e01640 1 -- 172.21.15.110:0/1944237467 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f75ec028440 con 0x55ce50244350 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f74e4e01640 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74d40346a0 0x7f74d4036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f74e4e01640 1 -- 172.21.15.110:0/1944237467 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f75ec00e040 con 0x55ce50244350 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74d40346a0 0x7f74d4036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 Processor -- start 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 -- start start 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce502518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce50220a90 con 0x55ce502514b0 2024-04-27T03:19:57.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74d40346a0 0x7f74d4036b60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55ce50234780 tx=0x7f75f4000bd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fb1b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce502518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fb1b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce502518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51594/0 (socket says 172.21.15.110:51594) 2024-04-27T03:19:57.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fb1b5640 1 -- 172.21.15.110:0/2357326674 learned_addr learned my addr 172.21.15.110:0/2357326674 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fb1b5640 1 -- 172.21.15.110:0/2357326674 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce50251e80 con 0x55ce502514b0 2024-04-27T03:19:57.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/2357326674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce502518b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f75ec027450 tx=0x7f75ec04a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f442e939d8b3e574 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f74de7fc640 1 -- 172.21.15.110:0/2357326674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75ec01ee50 con 0x55ce502514b0 2024-04-27T03:19:57.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f74de7fc640 1 -- 172.21.15.110:0/2357326674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec027bc0 con 0x55ce502514b0 2024-04-27T03:19:57.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f74de7fc640 1 -- 172.21.15.110:0/2357326674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75ec027ea0 con 0x55ce502514b0 2024-04-27T03:19:57.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 -- 172.21.15.110:0/2357326674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 msgr2=0x55ce502518b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 --2- 172.21.15.110:0/2357326674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce502518b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f75ec027450 tx=0x7f75ec04a580 comp rx=0 tx=0).stop 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 -- 172.21.15.110:0/2357326674 shutdown_connections 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 --2- 172.21.15.110:0/2357326674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce502518b0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 -- 172.21.15.110:0/2357326674 >> 172.21.15.110:0/2357326674 conn(0x55ce5024d7f0 msgr2=0x55ce5024b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 -- 172.21.15.110:0/2357326674 shutdown_connections 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.626+0000 7f75fce99980 1 -- 172.21.15.110:0/2357326674 wait complete. 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 Processor -- start 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 -- start start 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce50258970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55ce50252620 con 0x55ce502514b0 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fb1b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce50258970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fb1b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce50258970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51600/0 (socket says 172.21.15.110:51600) 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 learned_addr learned my addr 172.21.15.110:0/119457159 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce50258eb0 con 0x55ce502514b0 2024-04-27T03:19:57.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce50258970 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f75ec027920 tx=0x7f75ec04a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f74dcff9640 1 -- 172.21.15.110:0/119457159 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75ec039070 con 0x55ce502514b0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f74dcff9640 1 -- 172.21.15.110:0/119457159 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec01ee50 con 0x55ce502514b0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f74dcff9640 1 -- 172.21.15.110:0/119457159 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75ec011830 con 0x55ce502514b0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ce50259140 con 0x55ce502514b0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce5025be20 con 0x55ce502514b0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f74dcff9640 1 -- 172.21.15.110:0/119457159 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f75ec028730 con 0x55ce502514b0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f74dcff9640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74c40346a0 0x7f74c4036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f74dcff9640 1 -- 172.21.15.110:0/119457159 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f75ec03e020 con 0x55ce502514b0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74c40346a0 0x7f74c4036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50263f90 0x55ce50266450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50266b50 con 0x55ce50263f90 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.630+0000 7f75fa1b3640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50263f90 0x55ce50266450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74c40346a0 0x7f74c4036b60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f75f40acf80 tx=0x7f75f40df000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa1b3640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50263f90 0x55ce50266450 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f75e400e640 con 0x55ce50263f90 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 realm 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50263430 con 0x55ce50263f90 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f75e400e640 con 0x55ce50263f90 2024-04-27T03:19:57.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50267650 con 0x55ce50263f90 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f75e400e640 con 0x55ce50263f90 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50268430 0x55ce5026a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026aea0 con 0x55ce50268430 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50268430 0x55ce5026a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50268430 0x55ce5026a820 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026b6f0 con 0x55ce50263f90 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f75e400e640 con 0x55ce50263f90 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 20 searching for the correct realm 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ce5026d460 0x55ce5026f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce5026d460 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ce5026d460 0x55ce5026f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.634+0000 7f75fa9b4640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ce5026d460 0x55ce5026f850 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fb1b5640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f75ec014030 0x7f75ec042410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x7f75ec014030 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f75ec014030 0x7f75ec042410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f75ec014030 0x7f75ec042410 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e400e640 con 0x7f75ec014030 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e400e640 con 0x55ce50263f90 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce5026d460 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.638+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x7f75ec014030 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x7f75ec014030 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x7f75ec014030 2024-04-27T03:19:57.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f75e4029040 con 0x7f75ec014030 2024-04-27T03:19:57.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce5026d460 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce5026d460 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x7f75ec014030 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f75e4029040 con 0x7f75ec014030 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.642+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce5026d460 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x7f75ec014030 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x7f75ec014030 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50263f90 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x7f75ec014030 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x7f75ec014030 2024-04-27T03:19:57.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce5026d460 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x7f75ec014030 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75e4029040 con 0x7f75ec014030 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce5026d460 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.646+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026fed0 con 0x55ce50268430 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026c070 con 0x55ce50263f90 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f75e4029040 con 0x55ce50263f90 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026bc80 con 0x55ce50268430 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5026bcd0 con 0x55ce50263f90 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 zone default found 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 4 Realm: () 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:57.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 zonegroup default 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50270330 con 0x7f75ec014030 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x7f75ec014030 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50271da0 con 0x55ce5026d460 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce502721e0 con 0x55ce5026d460 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 started sync module instance, tier type = 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.650+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce502711b0 con 0x55ce50268430 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94344596362928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce502711b0 con 0x55ce50268430 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50272ce0 con 0x55ce50263f90 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94344596367280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50272ca0 con 0x55ce50263f90 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50273bf0 con 0x55ce50268430 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94344596362928] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.654+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94344596367280] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94344596369616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50272ca0 con 0x55ce50268430 2024-04-27T03:19:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50274020 con 0x7f75ec014030 2024-04-27T03:19:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x7f75ec014030 2024-04-27T03:19:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94344596360448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50273fe0 con 0x7f75ec014030 2024-04-27T03:19:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50274f00 con 0x55ce50268430 2024-04-27T03:19:57.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94344596369616] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94344596360448] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x7f75ec014030 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94344596374544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50274f00 con 0x55ce50268430 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.658+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce502752f0 con 0x55ce5026d460 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94344596371232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50275b90 con 0x55ce5026d460 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50276550 con 0x55ce5026d460 2024-04-27T03:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94344596374544] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94344596371232] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94344596379360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50276550 con 0x55ce5026d460 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50276e20 con 0x7f75ec014030 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'246 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x7f75ec014030 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94344596384112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50277240 con 0x7f75ec014030 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 20 add_watcher() i=0 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 20 add_watcher() i=1 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 20 add_watcher() i=2 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 20 add_watcher() i=3 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 20 add_watcher() i=4 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.662+0000 7f75fce99980 20 add_watcher() i=5 2024-04-27T03:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94344596379360] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 20 add_watcher() i=6 2024-04-27T03:19:57.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94344596384112] v31'247 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x7f75ec014030 2024-04-27T03:19:57.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 20 add_watcher() i=7 2024-04-27T03:19:57.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:57.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce50279ee0 con 0x55ce50263f90 2024-04-27T03:19:57.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94344596381856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce50277240 con 0x55ce50263f90 2024-04-27T03:19:57.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94344596381856] v31'249 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f74b17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:57.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:57.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.666+0000 7f75fce99980 5 note: GC not initialized 2024-04-27T03:19:57.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f742bfff640 20 reqs_thread_entry: start 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 20 Filter name: none 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f7429ffb640 20 reqs_thread_entry: start 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 cache get: name=default.rgw.meta+users.uid+system_user : miss 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:62 8.1e 8:78ce4db0:users.uid::system_user:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029d820 con 0x55ce5026d460 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(62 system_user [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 239+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 cache put: name=default.rgw.meta+users.uid+system_user info.flags=0x0 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 adding default.rgw.meta+users.uid+system_user to cache LRU end 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 cache get: name=default.rgw.meta+users.keys+system_akey : miss 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 8.0 8:01223cbf:users.keys::system_akey:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029be60 con 0x55ce5026d460 2024-04-27T03:19:57.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(63 system_akey [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 197+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 cache put: name=default.rgw.meta+users.keys+system_akey info.flags=0x0 2024-04-27T03:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 adding default.rgw.meta+users.keys+system_akey to cache LRU end 2024-04-27T03:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 cache get: name=default.rgw.meta+users.keys+system_akey : hit (negative entry) 2024-04-27T03:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 cache get: name=default.rgw.meta+users.keys+system_akey : hit (negative entry) 2024-04-27T03:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 8.1e 8:78ce4db0:users.uid::system_user:head [delete,create,call version.set in=58b,writefull 0~349 in=349b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029c700 con 0x55ce5026d460 2024-04-27T03:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(64 system_user [delete,create,call,writefull 0~349] v31'1 uv1 ondisk = 0) ==== 281+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 cache put: name=default.rgw.meta+users.uid+system_user info.flags=0x17 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 moving default.rgw.meta+users.uid+system_user to cache LRU end 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:system_user, ofs0, ns] 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.670+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94344596537936 in=545b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029c8f0 con 0x55ce50268430 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94344596362928 notify 133143986229 ret 0) ==== 575+0+0 (crc 0 0 0) 0x55ce5025be20 con 0x55ce50268430 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(65 notify.0 [notify cookie 94344596537936 out=8b] v0'0 uv246 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f74de7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986229 cookie 94344596362928 notifier 4835 bl.length()=533 2024-04-27T03:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f74de7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+system_user info.flags=0x17 2024-04-27T03:19:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f74de7fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+system_user to cache LRU end 2024-04-27T03:19:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f74de7fc640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f74c0003eb0 con 0x55ce50268430 2024-04-27T03:19:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(66 notify.0 [notify-ack] v0'0 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94344596537936 notify 133143986229 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55ce50259140 con 0x55ce50268430 2024-04-27T03:19:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 8.0 8:01223cbf:users.keys::system_akey:head [delete,create,writefull 0~15 in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029c250 con 0x55ce5026d460 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.674+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(67 system_akey [delete,create,writefull 0~15] v31'1 uv1 ondisk = 0) ==== 239+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fce99980 10 cache put: name=default.rgw.meta+users.keys+system_akey info.flags=0x7 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fce99980 10 moving default.rgw.meta+users.keys+system_akey to cache LRU end 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fce99980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:system_akey, ofs0, ns] 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [notify cookie 94344596537824 in=188b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55ce502a1330 con 0x55ce50263f90 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== watch-notify(notify (1) cookie 94344596367280 notify 133143986178 ret 0) ==== 218+0+0 (crc 0 0 0) 0x55ce4fff40d0 con 0x55ce50263f90 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f74de7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986178 cookie 94344596367280 notifier 4835 bl.length()=176 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f74de7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+system_akey info.flags=0x7 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f74de7fc640 10 rgw watcher librados: moving default.rgw.meta+users.keys+system_akey to cache LRU end 2024-04-27T03:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f74de7fc640 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 7.0 7:05bf5b68:::notify.1:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f74c0003eb0 con 0x55ce50263f90 2024-04-27T03:19:57.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(68 notify.1 [notify cookie 94344596537824 out=8b] v0'0 uv246 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(69 notify.1 [notify-ack] v0'0 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== watch-notify(notify_complete (2) cookie 94344596537824 notify 133143986178 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f75e4012810 con 0x55ce50263f90 2024-04-27T03:19:57.688 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"system_user","display_name":"System User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"system_user","access_key":"system_akey","secret_key":"system_skey","active":true,"create_date":"2024-04-27T03:19:57.674826Z"}],"swift_keys":[],"caps":[{"type":"user-info-without-keys","perm":"read"}],"op_mask":"read, write, delete","system":true,"default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.674817Z","tags":[],"group_ids":[]} 2024-04-27T03:19:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f74b17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94344596381856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x55ce50263f90 2024-04-27T03:19:57.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94344596381856] v31'250 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.678+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94344596362928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x55ce50268430 2024-04-27T03:19:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.682+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94344596362928] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.682+0000 7f75fce99980 20 remove_watcher() i=0 2024-04-27T03:19:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.682+0000 7f75fce99980 2 removed watcher, disabling cache 2024-04-27T03:19:57.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.682+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94344596367280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x55ce50263f90 2024-04-27T03:19:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.682+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94344596367280] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x55ce50263f90 2024-04-27T03:19:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.682+0000 7f75fce99980 20 remove_watcher() i=1 2024-04-27T03:19:57.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.682+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94344596369616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x55ce50268430 2024-04-27T03:19:57.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.686+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94344596369616] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.686+0000 7f75fce99980 20 remove_watcher() i=2 2024-04-27T03:19:57.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.686+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94344596360448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x7f75ec014030 2024-04-27T03:19:57.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.686+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94344596360448] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x7f75ec014030 2024-04-27T03:19:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.686+0000 7f75fce99980 20 remove_watcher() i=3 2024-04-27T03:19:57.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.686+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94344596374544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x55ce50268430 2024-04-27T03:19:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fb1b5640 1 -- 172.21.15.110:0/119457159 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94344596374544] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75ec025020 con 0x55ce50268430 2024-04-27T03:19:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fce99980 20 remove_watcher() i=4 2024-04-27T03:19:57.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94344596371232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x55ce5026d460 2024-04-27T03:19:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94344596371232] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fce99980 20 remove_watcher() i=5 2024-04-27T03:19:57.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94344596379360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x55ce5026d460 2024-04-27T03:19:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fa9b4640 1 -- 172.21.15.110:0/119457159 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94344596379360] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75f40d8040 con 0x55ce5026d460 2024-04-27T03:19:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fce99980 20 remove_watcher() i=6 2024-04-27T03:19:57.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.690+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94344596384112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55ce5029b520 con 0x7f75ec014030 2024-04-27T03:19:57.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94344596384112] v31'248 uv246 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f75e403c1c0 con 0x7f75ec014030 2024-04-27T03:19:57.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 20 remove_watcher() i=7 2024-04-27T03:19:57.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50268430 msgr2=0x55ce5026a820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50268430 0x55ce5026a820 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50263f90 msgr2=0x55ce50266450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50263f90 0x55ce50266450 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f75ec014030 msgr2=0x7f75ec042410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f75ec014030 0x7f75ec042410 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ce5026d460 msgr2=0x55ce5026f850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55ce5026d460 0x55ce5026f850 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74c40346a0 msgr2=0x7f74c4036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74c40346a0 0x7f74c4036b60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f75f40acf80 tx=0x7f75f40df000 comp rx=0 tx=0).stop 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fa1b3640 1 -- 172.21.15.110:0/119457159 reap_dead start 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 msgr2=0x55ce50258970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce50258970 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f75ec027920 tx=0x7f75ec04a3d0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 shutdown_connections 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/119457159 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce502514b0 0x55ce50258970 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 >> 172.21.15.110:0/119457159 conn(0x55ce5024d7f0 msgr2=0x55ce5024ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 shutdown_connections 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/119457159 wait complete. 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74d40346a0 msgr2=0x7f74d4036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74d40346a0 0x7f74d4036b60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55ce50234780 tx=0x7f75f4000bd0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce50244350 msgr2=0x55ce50244770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce50244350 0x55ce50244770 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f75ec017680 tx=0x7f75ec017a80 comp rx=0 tx=0).stop 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 shutdown_connections 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74d40346a0 0x7f74d4036b60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/1944237467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce50244350 0x55ce50244770 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 >> 172.21.15.110:0/1944237467 conn(0x55ce50239160 msgr2=0x55ce50239c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 shutdown_connections 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/1944237467 wait complete. 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50231690 msgr2=0x55ce50233a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50231690 0x55ce50233a80 crc :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50133ff0 msgr2=0x55ce502308a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50133ff0 0x55ce502308a0 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74e0034650 msgr2=0x7f74e0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74e0034650 0x7f74e0036b10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f75e4001da0 tx=0x7f75e4001300 comp rx=0 tx=0).stop 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 msgr2=0x55ce50135e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50135e50 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f75f401bb20 tx=0x7f75f4018020 comp rx=0 tx=0).stop 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 shutdown_connections 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55ce50231690 0x55ce50233a80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55ce50133ff0 0x55ce502308a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f74e0034650 0x7f74e0036b10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 --2- 172.21.15.110:0/821424461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55ce501336d0 0x55ce50135e50 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 >> 172.21.15.110:0/821424461 conn(0x55ce5012f160 msgr2=0x55ce5021aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 shutdown_connections 2024-04-27T03:19:57.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.694+0000 7f75fce99980 1 -- 172.21.15.110:0/821424461 wait complete. 2024-04-27T03:19:57.716 DEBUG:tasks.util.rgw: json result: {'user_id': 'system_user', 'display_name': 'System User', 'email': '', 'suspended': 0, 'max_buckets': 1000, 'subusers': [], 'keys': [{'user': 'system_user', 'access_key': 'system_akey', 'secret_key': 'system_skey', 'active': True, 'create_date': '2024-04-27T03:19:57.674826Z'}], 'swift_keys': [], 'caps': [{'type': 'user-info-without-keys', 'perm': 'read'}], 'op_mask': 'read, write, delete', 'system': True, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.674817Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:57.716 ERROR:root:{'user_id': 'system_user', 'display_name': 'System User', 'email': '', 'suspended': 0, 'max_buckets': 1000, 'subusers': [], 'keys': [{'user': 'system_user', 'access_key': 'system_akey', 'secret_key': 'system_skey', 'active': True, 'create_date': '2024-04-27T03:19:57.674826Z'}], 'swift_keys': [], 'caps': [{'type': 'user-info-without-keys', 'perm': 'read'}], 'op_mask': 'read, write, delete', 'system': True, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.674817Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:57.716 ERROR:root:0 2024-04-27T03:19:57.717 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:57.722 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:57.722 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'system_user'] 2024-04-27T03:19:57.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'system_user'] 2024-04-27T03:19:57.722 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid system_user 2024-04-27T03:19:57.753 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:57.753 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:57.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.742+0000 7fe1b9d97980 1 Processor -- start 2024-04-27T03:19:57.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.742+0000 7fe1b9d97980 1 -- start start 2024-04-27T03:19:57.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.742+0000 7fe1b9d97980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55cb83de25e0 0x55cb83e762a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.742+0000 7fe1b9d97980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55cb83d2c0d0 con 0x55cb83de3920 2024-04-27T03:19:57.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.742+0000 7fe1b9d97980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55cb83d21430 con 0x55cb83de25e0 2024-04-27T03:19:57.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b70b1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55cb83de25e0 0x55cb83e762a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b78b2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55cb83de3920 0x55cb83de1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44560/0 (socket says 172.21.15.110:44560) 2024-04-27T03:19:57.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b78b2640 1 -- 172.21.15.110:0/3456441450 learned_addr learned my addr 172.21.15.110:0/3456441450 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:57.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b70b1640 1 -- 172.21.15.110:0/3456441450 >> v1:172.21.15.77:6789/0 conn(0x55cb83de3920 legacy=0x55cb83de1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b70b1640 1 -- 172.21.15.110:0/3456441450 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb83d1a510 con 0x55cb83de25e0 2024-04-27T03:19:57.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b70b1640 1 --2- 172.21.15.110:0/3456441450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83e762a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000a480 tx=0x7fe1a001b280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1671694e85abf4dd server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b68b0640 1 -- 172.21.15.110:0/3456441450 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a00b0070 con 0x55cb83de25e0 2024-04-27T03:19:57.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b68b0640 1 -- 172.21.15.110:0/3456441450 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a00ab540 con 0x55cb83de25e0 2024-04-27T03:19:57.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b68b0640 1 -- 172.21.15.110:0/3456441450 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a00ab840 con 0x55cb83de25e0 2024-04-27T03:19:57.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3456441450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 msgr2=0x55cb83e762a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/3456441450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83e762a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000a480 tx=0x7fe1a001b280 comp rx=0 tx=0).stop 2024-04-27T03:19:57.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3456441450 shutdown_connections 2024-04-27T03:19:57.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/3456441450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83e762a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.746+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3456441450 >> 172.21.15.110:0/3456441450 conn(0x55cb83e68b60 msgr2=0x55cb83e6af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.750+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3456441450 shutdown_connections 2024-04-27T03:19:57.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.750+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3456441450 wait complete. 2024-04-27T03:19:57.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b9d97980 1 Processor -- start 2024-04-27T03:19:57.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b9d97980 1 -- start start 2024-04-27T03:19:57.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b9d97980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f5aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b9d97980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cb83d702d0 con 0x55cb83de25e0 2024-04-27T03:19:57.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b78b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f5aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b78b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f5aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51618/0 (socket says 172.21.15.110:51618) 2024-04-27T03:19:57.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b78b2640 1 -- 172.21.15.110:0/876718722 learned_addr learned my addr 172.21.15.110:0/876718722 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.762+0000 7fe1b78b2640 1 -- 172.21.15.110:0/876718722 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb83cf6ae0 con 0x55cb83de25e0 2024-04-27T03:19:57.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/876718722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f5aad0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0001d90 tx=0x7fe1b0001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf47c60ba535f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe0a5e0b640 1 -- 172.21.15.110:0/876718722 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1b0013030 con 0x55cb83de25e0 2024-04-27T03:19:57.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe0a5e0b640 1 -- 172.21.15.110:0/876718722 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b000e690 con 0x55cb83de25e0 2024-04-27T03:19:57.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe0a5e0b640 1 -- 172.21.15.110:0/876718722 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1b000ea10 con 0x55cb83de25e0 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- 172.21.15.110:0/876718722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 msgr2=0x55cb83f5aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/876718722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f5aad0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0001d90 tx=0x7fe1b0001e80 comp rx=0 tx=0).stop 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- 172.21.15.110:0/876718722 shutdown_connections 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/876718722 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f5aad0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- 172.21.15.110:0/876718722 >> 172.21.15.110:0/876718722 conn(0x55cb83e68b60 msgr2=0x55cb83f56660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- 172.21.15.110:0/876718722 shutdown_connections 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- 172.21.15.110:0/876718722 wait complete. 2024-04-27T03:19:57.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 Processor -- start 2024-04-27T03:19:57.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- start start 2024-04-27T03:19:57.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f61b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cb83e792d0 con 0x55cb83de25e0 2024-04-27T03:19:57.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b78b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f61b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b78b2640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f61b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51630/0 (socket says 172.21.15.110:51630) 2024-04-27T03:19:57.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b78b2640 1 -- 172.21.15.110:0/1836473357 learned_addr learned my addr 172.21.15.110:0/1836473357 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b78b2640 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb83f62050 con 0x55cb83de25e0 2024-04-27T03:19:57.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f61b10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55cb83f5b960 tx=0x7fe1b0000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe09ffff640 1 -- 172.21.15.110:0/1836473357 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1b0027070 con 0x55cb83de25e0 2024-04-27T03:19:57.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cb83f622e0 con 0x55cb83de25e0 2024-04-27T03:19:57.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe09ffff640 1 -- 172.21.15.110:0/1836473357 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b0019900 con 0x55cb83de25e0 2024-04-27T03:19:57.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe09ffff640 1 -- 172.21.15.110:0/1836473357 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1b0019c20 con 0x55cb83de25e0 2024-04-27T03:19:57.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.766+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb83f64f60 con 0x55cb83de25e0 2024-04-27T03:19:57.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe09ffff640 1 -- 172.21.15.110:0/1836473357 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe1b002a070 con 0x55cb83de25e0 2024-04-27T03:19:57.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe09ffff640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe0a0034650 0x7fe0a0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe09ffff640 1 -- 172.21.15.110:0/1836473357 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fe1b001f070 con 0x55cb83de25e0 2024-04-27T03:19:57.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b70b1640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe0a0034650 0x7fe0a0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83de3920 0x55cb83f68a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f69060 con 0x55cb83de3920 2024-04-27T03:19:57.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83de3920 0x55cb83f68a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b70b1640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe0a0034650 0x7fe0a0036b10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00ad000 tx=0x7fe1a00a5040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83de3920 0x55cb83f68a40 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b80b3640 1 -- 172.21.15.110:0/1836473357 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe1a800a4d0 con 0x55cb83de3920 2024-04-27T03:19:57.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83f69e50 0x55cb83f6c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f6c870 con 0x55cb83f69e50 2024-04-27T03:19:57.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83f69e50 0x55cb83f6c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83f69e50 0x55cb83f6c240 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b78b2640 1 -- 172.21.15.110:0/1836473357 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83f69e50 2024-04-27T03:19:57.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f6e120 con 0x55cb83de3920 2024-04-27T03:19:57.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.770+0000 7fe1b80b3640 1 -- 172.21.15.110:0/1836473357 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fe1a800a4d0 con 0x55cb83de3920 2024-04-27T03:19:57.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f6d350 con 0x55cb83f69e50 2024-04-27T03:19:57.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b78b2640 1 -- 172.21.15.110:0/1836473357 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83f69e50 2024-04-27T03:19:57.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f6d740 con 0x55cb83de3920 2024-04-27T03:19:57.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b80b3640 1 -- 172.21.15.110:0/1836473357 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fe1a800a4d0 con 0x55cb83de3920 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 Processor -- start 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 -- start start 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f7ca20 0x55cb83f7ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cb83d74310 con 0x55cb83f7ca20 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b80b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f7ca20 0x55cb83f7ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b80b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f7ca20 0x55cb83f7ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51640/0 (socket says 172.21.15.110:51640) 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b80b3640 1 -- 172.21.15.110:0/3904407421 learned_addr learned my addr 172.21.15.110:0/3904407421 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b80b3640 1 -- 172.21.15.110:0/3904407421 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb83f7d380 con 0x55cb83f7ca20 2024-04-27T03:19:57.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f7ca20 0x55cb83f7ce40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800dfc0 tx=0x7fe1a80178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe09d7fa640 1 -- 172.21.15.110:0/3904407421 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a80157d0 con 0x55cb83f7ca20 2024-04-27T03:19:57.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe09d7fa640 1 -- 172.21.15.110:0/3904407421 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a8015970 con 0x55cb83f7ca20 2024-04-27T03:19:57.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cb83f7de20 con 0x55cb83f7ca20 2024-04-27T03:19:57.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe09d7fa640 1 -- 172.21.15.110:0/3904407421 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a8015c70 con 0x55cb83f7ca20 2024-04-27T03:19:57.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.774+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb83f7e2f0 con 0x55cb83f7ca20 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe09d7fa640 1 -- 172.21.15.110:0/3904407421 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe1a8029440 con 0x55cb83f7ca20 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe09d7fa640 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe090034650 0x7fe090036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe09d7fa640 1 -- 172.21.15.110:0/3904407421 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fe1a800e040 con 0x55cb83f7ca20 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe090034650 0x7fe090036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 Processor -- start 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- start start 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe090034650 0x7fe090036b10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55cb83f6ce70 tx=0x7fe1b00483c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f8a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cb83f5b220 con 0x55cb83f89c20 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f8a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f8a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51654/0 (socket says 172.21.15.110:51654) 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 -- 172.21.15.110:0/2128808948 learned_addr learned my addr 172.21.15.110:0/2128808948 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 -- 172.21.15.110:0/2128808948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb83f8a610 con 0x55cb83f89c20 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/2128808948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f8a040 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe1a804a8e0 tx=0x7fe1a8015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519946b90583075 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe096ffd640 1 -- 172.21.15.110:0/2128808948 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a8012440 con 0x55cb83f89c20 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe096ffd640 1 -- 172.21.15.110:0/2128808948 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a80113a0 con 0x55cb83f89c20 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe096ffd640 1 -- 172.21.15.110:0/2128808948 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a80125e0 con 0x55cb83f89c20 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- 172.21.15.110:0/2128808948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 msgr2=0x55cb83f8a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/2128808948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f8a040 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe1a804a8e0 tx=0x7fe1a8015fd0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- 172.21.15.110:0/2128808948 shutdown_connections 2024-04-27T03:19:57.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/2128808948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f8a040 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- 172.21.15.110:0/2128808948 >> 172.21.15.110:0/2128808948 conn(0x55cb83f85f60 msgr2=0x55cb83f83fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- 172.21.15.110:0/2128808948 shutdown_connections 2024-04-27T03:19:57.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- 172.21.15.110:0/2128808948 wait complete. 2024-04-27T03:19:57.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 Processor -- start 2024-04-27T03:19:57.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- start start 2024-04-27T03:19:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f91140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b9d97980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55cb83f8ad70 con 0x55cb83f89c20 2024-04-27T03:19:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f91140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f91140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51670/0 (socket says 172.21.15.110:51670) 2024-04-27T03:19:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.778+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 learned_addr learned my addr 172.21.15.110:0/456893555 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:57.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb83f91680 con 0x55cb83f89c20 2024-04-27T03:19:57.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f91140 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8016010 tx=0x7fe1a804a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe0957fa640 1 -- 172.21.15.110:0/456893555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a804c110 con 0x55cb83f89c20 2024-04-27T03:19:57.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe0957fa640 1 -- 172.21.15.110:0/456893555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a804c2b0 con 0x55cb83f89c20 2024-04-27T03:19:57.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cb83f91910 con 0x55cb83f89c20 2024-04-27T03:19:57.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe0957fa640 1 -- 172.21.15.110:0/456893555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a80491b0 con 0x55cb83f89c20 2024-04-27T03:19:57.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb83f945f0 con 0x55cb83f89c20 2024-04-27T03:19:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe0957fa640 1 -- 172.21.15.110:0/456893555 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe1a8044070 con 0x55cb83f89c20 2024-04-27T03:19:57.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe0957fa640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe080034650 0x7fe080036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe0957fa640 1 -- 172.21.15.110:0/456893555 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fe1a8041050 con 0x55cb83f89c20 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe080034650 0x7fe080036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83f9c5b0 0x55cb83f9ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f9f1c0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b70b1640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83f9c5b0 0x55cb83f9ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe080034650 0x7fe080036b10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000c010 tx=0x7fe1b003a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b70b1640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83f9c5b0 0x55cb83f9ea70 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 20 realm 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f9f5f0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.782+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83f9fec0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83fa08e0 0x55cb83fa2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa3480 con 0x55cb83fa08e0 2024-04-27T03:19:57.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83fa08e0 0x55cb83fa2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83fa08e0 0x55cb83fa2d10 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa3cd0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:57.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 20 searching for the correct realm 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55cb83fa5a40 0x55cb83fa7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55cb83fa5a40 0x55cb83fa7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b78b2640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55cb83fa5a40 0x55cb83fa7e30 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b80b3640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe1a8034070 0x7fe1a804c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x7fe1a8034070 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.786+0000 7fe1b70b1640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe1a8034070 0x7fe1a804c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe1a8034070 0x7fe1a804c9d0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:57.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x7fe1a8034070 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.790+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x7fe1a8034070 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x7fe1a8034070 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x7fe1a8034070 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.794+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x7fe1a8034070 2024-04-27T03:19:57.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x7fe1a8034070 2024-04-27T03:19:57.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa84b0 con 0x55cb83fa08e0 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b9d97980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b9d97980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b9d97980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b9d97980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:57.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b9d97980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.798+0000 7fe1b9d97980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa4680 con 0x55cb83f9c5b0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa4f20 con 0x55cb83fa08e0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa42c0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 zone default found 2024-04-27T03:19:57.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 4 Realm: () 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 zonegroup default 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fa8dc0 con 0x7fe1a8034070 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83faab90 con 0x55cb83fa5a40 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83faaf60 con 0x55cb83fa5a40 2024-04-27T03:19:57.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:57.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 started sync module instance, tier type = 2024-04-27T03:19:57.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:57.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.802+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83faa030 con 0x55cb83fa08e0 2024-04-27T03:19:57.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94332580963360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fab870 con 0x55cb83fa08e0 2024-04-27T03:19:57.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fac0c0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94332580968560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fac1c0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83faccc0 con 0x55cb83fa08e0 2024-04-27T03:19:57.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94332580963360] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.806+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94332580968560] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94332580969616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83facc80 con 0x55cb83fa08e0 2024-04-27T03:19:57.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fad590 con 0x7fe1a8034070 2024-04-27T03:19:57.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94332580974816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83facc80 con 0x7fe1a8034070 2024-04-27T03:19:57.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fae010 con 0x55cb83fa08e0 2024-04-27T03:19:57.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94332580969616] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94332580974816] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94332580975872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fadb20 con 0x55cb83fa08e0 2024-04-27T03:19:57.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.810+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fae8b0 con 0x55cb83fa5a40 2024-04-27T03:19:57.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94332580979712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fadb20 con 0x55cb83fa5a40 2024-04-27T03:19:57.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83faf720 con 0x55cb83fa5a40 2024-04-27T03:19:57.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94332580975872] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94332580979712] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94332580981840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83faf270 con 0x55cb83fa5a40 2024-04-27T03:19:57.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fafc30 con 0x7fe1a8034070 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'249 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94332580981840] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94332580984704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fb0050 con 0x7fe1a8034070 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 20 add_watcher() i=0 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 20 add_watcher() i=1 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 20 add_watcher() i=2 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 20 add_watcher() i=3 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 20 add_watcher() i=4 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.814+0000 7fe1b9d97980 20 add_watcher() i=5 2024-04-27T03:19:57.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 20 add_watcher() i=6 2024-04-27T03:19:57.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94332580984704] v31'250 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 20 add_watcher() i=7 2024-04-27T03:19:57.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:57.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fb0050 con 0x55cb83f9c5b0 2024-04-27T03:19:57.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94332580985760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fb0920 con 0x55cb83f9c5b0 2024-04-27T03:19:57.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94332580985760] v31'251 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe06dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:57.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:57.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.818+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbba90 con 0x55cb83fa08e0 2024-04-27T03:19:57.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.822+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v31'59 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.822+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbbb10 con 0x7fe1a8034070 2024-04-27T03:19:57.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.822+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 gc.1 [create,call,call,call] v31'37 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.822+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbc200 con 0x55cb83fa5a40 2024-04-27T03:19:57.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.826+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v31'36 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.826+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbc770 con 0x55cb83f9c5b0 2024-04-27T03:19:57.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.826+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(65 gc.3 [create,call,call,call] v31'252 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.826+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbcc20 con 0x55cb83fa08e0 2024-04-27T03:19:57.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.830+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v31'85 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.830+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbcc20 con 0x55cb83fa5a40 2024-04-27T03:19:57.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.830+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v31'63 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.830+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fb7780 con 0x55cb83fa08e0 2024-04-27T03:19:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.834+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v31'39 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.834+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fb7780 con 0x55cb83f9c5b0 2024-04-27T03:19:57.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.834+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 gc.7 [create,call,call,call] v31'59 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.834+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fb7860 con 0x55cb83fa08e0 2024-04-27T03:19:57.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.834+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v31'29 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.834+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.838+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v31'35 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.838+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.838+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v31'86 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.838+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa5a40 2024-04-27T03:19:57.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.842+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v31'35 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.842+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.842+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v31'35 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.842+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83f9c5b0 2024-04-27T03:19:57.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.846+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(75 gc.13 [create,call,call,call] v31'69 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.846+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83f9c5b0 2024-04-27T03:19:57.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.846+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(76 gc.14 [create,call,call,call] v31'253 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.846+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.850+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v31'77 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.850+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.850+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v31'87 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.850+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa5a40 2024-04-27T03:19:57.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.854+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v31'42 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.854+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.854+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v31'37 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.854+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83f9c5b0 2024-04-27T03:19:57.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.858+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(81 gc.19 [create,call,call,call] v31'254 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.858+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.858+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v31'78 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.858+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.858+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v31'45 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.858+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa5a40 2024-04-27T03:19:57.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.862+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v31'37 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.862+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83f9c5b0 2024-04-27T03:19:57.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.862+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 gc.23 [create,call,call,call] v31'47 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.862+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa5a40 2024-04-27T03:19:57.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.862+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v31'31 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.862+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa5a40 2024-04-27T03:19:57.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.866+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v31'64 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.876 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.866+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa5a40 2024-04-27T03:19:57.877 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.866+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v31'38 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.866+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.870+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(89 gc.27 [create,call,call,call] v31'42 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.870+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83f9c5b0 2024-04-27T03:19:57.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.870+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v31'60 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.870+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83fa08e0 2024-04-27T03:19:57.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(91 gc.29 [create,call,call,call] v31'79 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x55cb83f9c5b0 2024-04-27T03:19:57.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v31'70 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbd530 con 0x7fe1a8034070 2024-04-27T03:19:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(93 gc.31 [create,call,call,call] v31'35 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe06cfe1640 20 reqs_thread_entry: start 2024-04-27T03:19:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 20 Filter name: none 2024-04-27T03:19:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fdfe67fc640 20 reqs_thread_entry: start 2024-04-27T03:19:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 10 cache get: name=default.rgw.meta+users.uid+system_user : miss 2024-04-27T03:19:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 20 rados->read ofs=0 len=0 2024-04-27T03:19:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:94 8.1e 8:78ce4db0:users.uid::system_user:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fd85a0 con 0x55cb83fa5a40 2024-04-27T03:19:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(94 system_user [call out=48b,read 0~349 out=349b,getxattrs out=75b] v0'0 uv1 ondisk = 0) ==== 239+0+472 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 20 rados_obj.operate() r=0 bl.length=349 2024-04-27T03:19:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 10 cache put: name=default.rgw.meta+users.uid+system_user info.flags=0x13 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 10 adding default.rgw.meta+users.uid+system_user to cache LRU end 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:95 8.1e 8:79f34a7d:users.uid::system_user.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fbadd0 con 0x55cb83fa5a40 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.874+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(95 system_user.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 163+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b9d97980 10 removing key index: system_akey 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:96 8.0 8:01223cbf:users.keys::system_akey:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fd9010 con 0x55cb83fa5a40 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(96 system_akey [delete] v31'2 uv2 ondisk = 0) ==== 155+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b9d97980 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:system_akey, ofs0, ns] 2024-04-27T03:19:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:97 7.0 7:05bf5b68:::notify.1:head [notify cookie 94332581157664 in=173b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb6f0 con 0x55cb83f9c5b0 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== watch-notify(notify (1) cookie 94332580968560 notify 133143986179 ret 0) ==== 203+0+0 (crc 0 0 0) 0x55cb83fb7860 con 0x55cb83f9c5b0 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe096ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986179 cookie 94332580968560 notifier 4842 bl.length()=161 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(97 notify.1 [notify cookie 94332581157664 out=8b] v0'0 uv249 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe096ffd640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:98 7.0 7:05bf5b68:::notify.1:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fe07c003d70 con 0x55cb83f9c5b0 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(98 notify.1 [notify-ack] v0'0 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== watch-notify(notify_complete (2) cookie 94332581157664 notify 133143986179 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55cb83fbc330 con 0x55cb83f9c5b0 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b9d97980 10 removing email index: 2024-04-27T03:19:57.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b9d97980 10 removing user buckets index 2024-04-27T03:19:57.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.878+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:99 8.1e 8:79f34a7d:users.uid::system_user.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x55cb83fa5a40 2024-04-27T03:19:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(99 system_user.buckets [delete] v31'2 uv1 ondisk = -2 ((2) No such file or directory)) ==== 163+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b9d97980 10 removing user index: system_user 2024-04-27T03:19:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:100 8.1e 8:78ce4db0:users.uid::system_user:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x55cb83fa5a40 2024-04-27T03:19:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(100 system_user [call,call,delete] v31'3 uv3 ondisk = 0) ==== 239+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b9d97980 10 removing default.rgw.meta+users.uid+system_user from cache 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b9d97980 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:system_user, ofs0, ns] 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 7.1f 7:f95f44c2:::notify.0:head [notify cookie 94332581164656 in=172b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fd9b20 con 0x55cb83fa08e0 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== watch-notify(notify (1) cookie 94332580963360 notify 133143986230 ret 0) ==== 202+0+0 (crc 0 0 0) 0x55cb83fb7640 con 0x55cb83fa08e0 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe096ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986230 cookie 94332580963360 notifier 4842 bl.length()=160 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe096ffd640 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:102 7.1f 7:f95f44c2:::notify.0:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fe07c003d70 con 0x55cb83fa08e0 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(101 notify.0 [notify cookie 94332581164656 out=8b] v0'0 uv249 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(102 notify.0 [notify-ack] v0'0 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.882+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== watch-notify(notify_complete (2) cookie 94332581164656 notify 133143986230 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55cb83f91680 con 0x55cb83fa08e0 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.886+0000 7fe06dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.886+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94332580985760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x55cb83f9c5b0 2024-04-27T03:19:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.886+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(103 data_loggenerations_metadata [watch unwatch cookie 94332580985760] v31'255 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.886+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94332580963360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x55cb83fa08e0 2024-04-27T03:19:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.890+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 94332580963360] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.890+0000 7fe1b9d97980 20 remove_watcher() i=0 2024-04-27T03:19:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.890+0000 7fe1b9d97980 2 removed watcher, disabling cache 2024-04-27T03:19:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.890+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94332580968560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x55cb83f9c5b0 2024-04-27T03:19:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.890+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 94332580968560] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x55cb83f9c5b0 2024-04-27T03:19:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.890+0000 7fe1b9d97980 20 remove_watcher() i=1 2024-04-27T03:19:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.890+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94332580969616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fab870 con 0x55cb83fa08e0 2024-04-27T03:19:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 94332580969616] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b9d97980 20 remove_watcher() i=2 2024-04-27T03:19:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94332580974816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x7fe1a8034070 2024-04-27T03:19:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 94332580974816] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b9d97980 20 remove_watcher() i=3 2024-04-27T03:19:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94332580975872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fad0e0 con 0x55cb83fa08e0 2024-04-27T03:19:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b80b3640 1 -- 172.21.15.110:0/456893555 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 94332580975872] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a8031070 con 0x55cb83fa08e0 2024-04-27T03:19:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b9d97980 20 remove_watcher() i=4 2024-04-27T03:19:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.894+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94332580979712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x55cb83fa5a40 2024-04-27T03:19:57.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 94332580979712] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 20 remove_watcher() i=5 2024-04-27T03:19:57.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94332580981840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x55cb83fa5a40 2024-04-27T03:19:57.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b78b2640 1 -- 172.21.15.110:0/456893555 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 94332580981840] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1b0029030 con 0x55cb83fa5a40 2024-04-27T03:19:57.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 20 remove_watcher() i=6 2024-04-27T03:19:57.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94332580984704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55cb83fdb670 con 0x7fe1a8034070 2024-04-27T03:19:57.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 94332580984704] v31'251 uv249 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fe1a00b3040 con 0x7fe1a8034070 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 20 remove_watcher() i=7 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83fa08e0 msgr2=0x55cb83fa2d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83fa08e0 0x55cb83fa2d10 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83f9c5b0 msgr2=0x55cb83f9ea70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83f9c5b0 0x55cb83f9ea70 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe1a8034070 msgr2=0x7fe1a804c9d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fe1a8034070 0x7fe1a804c9d0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55cb83fa5a40 msgr2=0x55cb83fa7e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55cb83fa5a40 0x55cb83fa7e30 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe080034650 msgr2=0x7fe080036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe080034650 0x7fe080036b10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000c010 tx=0x7fe1b003a000 comp rx=0 tx=0).stop 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 msgr2=0x55cb83f91140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b70b1640 1 -- 172.21.15.110:0/456893555 reap_dead start 2024-04-27T03:19:57.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f91140 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8016010 tx=0x7fe1a804a870 comp rx=0 tx=0).stop 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 shutdown_connections 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/456893555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f89c20 0x55cb83f91140 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 >> 172.21.15.110:0/456893555 conn(0x55cb83f85f60 msgr2=0x55cb83f835c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 shutdown_connections 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/456893555 wait complete. 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe090034650 msgr2=0x7fe090036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe090034650 0x7fe090036b10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55cb83f6ce70 tx=0x7fe1b00483c0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f7ca20 msgr2=0x55cb83f7ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f7ca20 0x55cb83f7ce40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800dfc0 tx=0x7fe1a80178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 shutdown_connections 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe090034650 0x7fe090036b10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/3904407421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83f7ca20 0x55cb83f7ce40 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 >> 172.21.15.110:0/3904407421 conn(0x55cb83f71830 msgr2=0x55cb83f72340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 shutdown_connections 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/3904407421 wait complete. 2024-04-27T03:19:57.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83f69e50 msgr2=0x55cb83f6c240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83f69e50 0x55cb83f6c240 crc :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83de3920 msgr2=0x55cb83f68a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83de3920 0x55cb83f68a40 crc :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe0a0034650 msgr2=0x7fe0a0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe0a0034650 0x7fe0a0036b10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe1a00ad000 tx=0x7fe1a00a5040 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 msgr2=0x55cb83f61b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f61b10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x55cb83f5b960 tx=0x7fe1b0000bd0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 shutdown_connections 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55cb83f69e50 0x55cb83f6c240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55cb83de3920 0x55cb83f68a40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fe0a0034650 0x7fe0a0036b10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 --2- 172.21.15.110:0/1836473357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55cb83de25e0 0x55cb83f61b10 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 >> 172.21.15.110:0/1836473357 conn(0x55cb83e68b60 msgr2=0x55cb83f55170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 shutdown_connections 2024-04-27T03:19:57.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.898+0000 7fe1b9d97980 1 -- 172.21.15.110:0/1836473357 wait complete. 2024-04-27T03:19:57.920 ERROR:root:None 2024-04-27T03:19:57.920 ERROR:root:0 2024-04-27T03:19:57.920 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'a_user', '--display-name', 'True Admin User', '--access-key', 'true_admin_akey', '--secret', 'true_admin_skey', '--caps', 'user-info-without-keys=read', '--admin'] 2024-04-27T03:19:57.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'create', '--uid', 'a_user', '--display-name', 'True Admin User', '--access-key', 'true_admin_akey', '--secret', 'true_admin_skey', '--caps', 'user-info-without-keys=read', '--admin'] 2024-04-27T03:19:57.920 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user create --uid a_user --display-name 'True Admin User' --access-key true_admin_akey --secret true_admin_skey --caps user-info-without-keys=read --admin 2024-04-27T03:19:57.998 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:57.999 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:57.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e5b69980 1 Processor -- start 2024-04-27T03:19:58.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e5b69980 1 -- start start 2024-04-27T03:19:58.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e5b69980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560a6d282830 0x560a6d3132a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e5b69980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x560a6d1cd0d0 con 0x560a6d3137e0 2024-04-27T03:19:58.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e5b69980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x560a6d1c2430 con 0x560a6d282830 2024-04-27T03:19:58.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e3684640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x560a6d282830 0x560a6d3132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e2e83640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x560a6d3137e0 0x560a6d315bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44566/0 (socket says 172.21.15.110:44566) 2024-04-27T03:19:58.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e2e83640 1 -- 172.21.15.110:0/122342344 learned_addr learned my addr 172.21.15.110:0/122342344 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:58.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e3684640 1 -- 172.21.15.110:0/122342344 >> v1:172.21.15.77:6789/0 conn(0x560a6d3137e0 legacy=0x560a6d315bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.990+0000 7f81e3684640 1 -- 172.21.15.110:0/122342344 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6d1bb510 con 0x560a6d282830 2024-04-27T03:19:58.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e3684640 1 --2- 172.21.15.110:0/122342344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3132a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00a420 tx=0x7f81dc01afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20caacf7314524e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e2682640 1 -- 172.21.15.110:0/122342344 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81dc0af070 con 0x560a6d282830 2024-04-27T03:19:58.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e2682640 1 -- 172.21.15.110:0/122342344 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc0aa530 con 0x560a6d282830 2024-04-27T03:19:58.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e2682640 1 -- 172.21.15.110:0/122342344 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81dc0aa850 con 0x560a6d282830 2024-04-27T03:19:58.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e5b69980 1 -- 172.21.15.110:0/122342344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 msgr2=0x560a6d3132a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e5b69980 1 --2- 172.21.15.110:0/122342344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3132a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00a420 tx=0x7f81dc01afd0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e5b69980 1 -- 172.21.15.110:0/122342344 shutdown_connections 2024-04-27T03:19:58.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e5b69980 1 --2- 172.21.15.110:0/122342344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3132a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e5b69980 1 -- 172.21.15.110:0/122342344 >> 172.21.15.110:0/122342344 conn(0x560a6d3089c0 msgr2=0x560a6d30ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e5b69980 1 -- 172.21.15.110:0/122342344 shutdown_connections 2024-04-27T03:19:58.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:57.994+0000 7f81e5b69980 1 -- 172.21.15.110:0/122342344 wait complete. 2024-04-27T03:19:58.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e5b69980 1 Processor -- start 2024-04-27T03:19:58.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e5b69980 1 -- start start 2024-04-27T03:19:58.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e5b69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3fb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e5b69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560a6d2112d0 con 0x560a6d282830 2024-04-27T03:19:58.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e3684640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3fb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e3684640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3fb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51692/0 (socket says 172.21.15.110:51692) 2024-04-27T03:19:58.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e3684640 1 -- 172.21.15.110:0/3078087969 learned_addr learned my addr 172.21.15.110:0/3078087969 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e3684640 1 -- 172.21.15.110:0/3078087969 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6d197ae0 con 0x560a6d282830 2024-04-27T03:19:58.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f81e3684640 1 --2- 172.21.15.110:0/3078087969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3fb690 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0ad040 tx=0x7f81dc0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa388ac7437821f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f80d160a640 1 -- 172.21.15.110:0/3078087969 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81dc0af050 con 0x560a6d282830 2024-04-27T03:19:58.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f80d160a640 1 -- 172.21.15.110:0/3078087969 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc0b0040 con 0x560a6d282830 2024-04-27T03:19:58.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.006+0000 7f80d160a640 1 -- 172.21.15.110:0/3078087969 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81dc0aaca0 con 0x560a6d282830 2024-04-27T03:19:58.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/3078087969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 msgr2=0x560a6d3fb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3078087969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3fb690 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0ad040 tx=0x7f81dc0033b0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/3078087969 shutdown_connections 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3078087969 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d3fb690 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/3078087969 >> 172.21.15.110:0/3078087969 conn(0x560a6d3089c0 msgr2=0x560a6d3f7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/3078087969 shutdown_connections 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/3078087969 wait complete. 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 Processor -- start 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- start start 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d402670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560a6d319db0 con 0x560a6d282830 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e3684640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d402670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e3684640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d402670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51706/0 (socket says 172.21.15.110:51706) 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e3684640 1 -- 172.21.15.110:0/535242923 learned_addr learned my addr 172.21.15.110:0/535242923 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e3684640 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6d402bb0 con 0x560a6d282830 2024-04-27T03:19:58.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e3684640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d402670 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0b8040 tx=0x7f81dc019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f80cfe07640 1 -- 172.21.15.110:0/535242923 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81dc0c5070 con 0x560a6d282830 2024-04-27T03:19:58.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560a6d402e40 con 0x560a6d282830 2024-04-27T03:19:58.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f80cfe07640 1 -- 172.21.15.110:0/535242923 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc0b6d40 con 0x560a6d282830 2024-04-27T03:19:58.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f80cfe07640 1 -- 172.21.15.110:0/535242923 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81dc0c15f0 con 0x560a6d282830 2024-04-27T03:19:58.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6d405a70 con 0x560a6d282830 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f80cfe07640 1 -- 172.21.15.110:0/535242923 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f81dc0c8070 con 0x560a6d282830 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f80cfe07640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c8034650 0x7f80c8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f80cfe07640 1 -- 172.21.15.110:0/535242923 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f81dc0bb070 con 0x560a6d282830 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e2e83640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c8034650 0x7f80c8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d3137e0 0x560a6d409550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d409a90 con 0x560a6d3137e0 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.010+0000 7f81e3e85640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d3137e0 0x560a6d409550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e2e83640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c8034650 0x7f80c8036b10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f81cc001da0 tx=0x7f81cc001300 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e3e85640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d3137e0 0x560a6d409550 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e3e85640 1 -- 172.21.15.110:0/535242923 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81d40083f0 con 0x560a6d3137e0 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d40a820 0x560a6d40cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d40d270 con 0x560a6d40a820 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e3684640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d40a820 0x560a6d40cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e3684640 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d40a820 0x560a6d40cc10 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e3684640 1 -- 172.21.15.110:0/535242923 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d40a820 2024-04-27T03:19:58.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d40eb20 con 0x560a6d3137e0 2024-04-27T03:19:58.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e3e85640 1 -- 172.21.15.110:0/535242923 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f81d40083f0 con 0x560a6d3137e0 2024-04-27T03:19:58.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.014+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d40dd50 con 0x560a6d40a820 2024-04-27T03:19:58.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e3684640 1 -- 172.21.15.110:0/535242923 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d40a820 2024-04-27T03:19:58.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d40e140 con 0x560a6d3137e0 2024-04-27T03:19:58.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e3e85640 1 -- 172.21.15.110:0/535242923 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f81d40083f0 con 0x560a6d3137e0 2024-04-27T03:19:58.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e5b69980 1 Processor -- start 2024-04-27T03:19:58.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e5b69980 1 -- start start 2024-04-27T03:19:58.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e5b69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d41d440 0x560a6d41d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e5b69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560a6d215310 con 0x560a6d41d440 2024-04-27T03:19:58.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e3e85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d41d440 0x560a6d41d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e3e85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d41d440 0x560a6d41d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51720/0 (socket says 172.21.15.110:51720) 2024-04-27T03:19:58.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e3e85640 1 -- 172.21.15.110:0/3715089040 learned_addr learned my addr 172.21.15.110:0/3715089040 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e3e85640 1 -- 172.21.15.110:0/3715089040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6d41dda0 con 0x560a6d41d440 2024-04-27T03:19:58.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e3e85640 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d41d440 0x560a6d41d860 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f81d400aab0 tx=0x7f81d400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f80cd602640 1 -- 172.21.15.110:0/3715089040 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4007d30 con 0x560a6d41d440 2024-04-27T03:19:58.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f80cd602640 1 -- 172.21.15.110:0/3715089040 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d401d070 con 0x560a6d41d440 2024-04-27T03:19:58.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560a6d41e840 con 0x560a6d41d440 2024-04-27T03:19:58.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f80cd602640 1 -- 172.21.15.110:0/3715089040 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4018440 con 0x560a6d41d440 2024-04-27T03:19:58.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.018+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6d41ed10 con 0x560a6d41d440 2024-04-27T03:19:58.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f80cd602640 1 -- 172.21.15.110:0/3715089040 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f81d4013070 con 0x560a6d41d440 2024-04-27T03:19:58.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f80cd602640 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80bc034650 0x7f80bc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f80cd602640 1 -- 172.21.15.110:0/3715089040 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f81d404b660 con 0x560a6d41d440 2024-04-27T03:19:58.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3684640 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80bc034650 0x7f80bc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 Processor -- start 2024-04-27T03:19:58.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- start start 2024-04-27T03:19:58.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3684640 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80bc034650 0x7f80bc036b10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x560a6d40d870 tx=0x7f81dc0b4f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d42a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560a6d3fbd20 con 0x560a6d42a5c0 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d42a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d42a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51730/0 (socket says 172.21.15.110:51730) 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 -- 172.21.15.110:0/3690143254 learned_addr learned my addr 172.21.15.110:0/3690143254 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 -- 172.21.15.110:0/3690143254 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6d42afb0 con 0x560a6d42a5c0 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 --2- 172.21.15.110:0/3690143254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d42a9e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f81d4049930 tx=0x7f81d4018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e64ad428dc52843 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f80c6ffd640 1 -- 172.21.15.110:0/3690143254 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4047070 con 0x560a6d42a5c0 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f80c6ffd640 1 -- 172.21.15.110:0/3690143254 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d4018bc0 con 0x560a6d42a5c0 2024-04-27T03:19:58.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f80c6ffd640 1 -- 172.21.15.110:0/3690143254 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4020360 con 0x560a6d42a5c0 2024-04-27T03:19:58.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- 172.21.15.110:0/3690143254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 msgr2=0x560a6d42a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3690143254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d42a9e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f81d4049930 tx=0x7f81d4018950 comp rx=0 tx=0).stop 2024-04-27T03:19:58.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- 172.21.15.110:0/3690143254 shutdown_connections 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3690143254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d42a9e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- 172.21.15.110:0/3690143254 >> 172.21.15.110:0/3690143254 conn(0x560a6d426950 msgr2=0x560a6d4249b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- 172.21.15.110:0/3690143254 shutdown_connections 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- 172.21.15.110:0/3690143254 wait complete. 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 Processor -- start 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- start start 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d431a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e5b69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x560a6d42b6a0 con 0x560a6d42a5c0 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d431a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d431a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51740/0 (socket says 172.21.15.110:51740) 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.022+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 learned_addr learned my addr 172.21.15.110:0/2356270827 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a6d431fd0 con 0x560a6d42a5c0 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e3e85640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d431a90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f81d4018aa0 tx=0x7f81d40498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f80c57fa640 1 -- 172.21.15.110:0/2356270827 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4031070 con 0x560a6d42a5c0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560a6d432260 con 0x560a6d42a5c0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f80c57fa640 1 -- 172.21.15.110:0/2356270827 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d404b2e0 con 0x560a6d42a5c0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a6d434f40 con 0x560a6d42a5c0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f80c57fa640 1 -- 172.21.15.110:0/2356270827 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81d4049a70 con 0x560a6d42a5c0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f80c57fa640 1 -- 172.21.15.110:0/2356270827 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f81d4034070 con 0x560a6d42a5c0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f80c57fa640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80ac034650 0x7f80ac036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f80c57fa640 1 -- 172.21.15.110:0/2356270827 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f81d400c0b0 con 0x560a6d42a5c0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e3684640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80ac034650 0x7f80ac036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d43d120 0x560a6d43f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d43fd10 con 0x560a6d43d120 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e2e83640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d43d120 0x560a6d43f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e3684640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80ac034650 0x7f80ac036b10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0b4fd0 tx=0x7f81dc0e0000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e2e83640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d43d120 0x560a6d43f5c0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81cc00e640 con 0x560a6d43d120 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 realm 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d43c630 con 0x560a6d43d120 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81cc00e640 con 0x560a6d43d120 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d4407e0 con 0x560a6d43d120 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81cc00e640 con 0x560a6d43d120 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d4415c0 0x560a6d4439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.026+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d444040 con 0x560a6d4415c0 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3e85640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d4415c0 0x560a6d4439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3e85640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d4415c0 0x560a6d4439b0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e5b69980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d444890 con 0x560a6d43d120 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f81cc00e640 con 0x560a6d43d120 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e5b69980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e5b69980 20 searching for the correct realm 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560a6d446600 0x560a6d4489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d446600 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3684640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560a6d446600 0x560a6d4489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3684640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560a6d446600 0x560a6d4489f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3e85640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d4021700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x7f81d4036070 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e2e83640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d4021700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e2e83640 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d4021700 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc00e640 con 0x7f81d4036070 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc00e640 con 0x560a6d43d120 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.030+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d446600 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.047 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x7f81d4036070 2024-04-27T03:19:58.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x7f81d4036070 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x7f81d4036070 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f81cc029040 con 0x7f81d4036070 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.034+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d446600 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d446600 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x7f81d4036070 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f81cc029040 con 0x7f81d4036070 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d446600 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x7f81d4036070 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x7f81d4036070 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d43d120 2024-04-27T03:19:58.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x7f81d4036070 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x7f81d4036070 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d446600 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x7f81d4036070 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81cc029040 con 0x7f81d4036070 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.038+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d446600 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x560a6d449070 con 0x560a6d4415c0 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:58.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d445210 con 0x560a6d43d120 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f81cc029040 con 0x560a6d43d120 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.042+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d444e50 con 0x560a6d4415c0 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d445b70 con 0x560a6d43d120 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f81cc03c1c0 con 0x560a6d43d120 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 zone default found 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 4 Realm: () 2024-04-27T03:19:58.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 zonegroup default 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d4456b0 con 0x7f81d4036070 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x7f81d4036070 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44b3b0 con 0x560a6d446600 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44b750 con 0x560a6d446600 2024-04-27T03:19:58.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 started sync module instance, tier type = 2024-04-27T03:19:58.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:58.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44a850 con 0x560a6d4415c0 2024-04-27T03:19:58.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94602782878784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44a4b0 con 0x560a6d4415c0 2024-04-27T03:19:58.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.046+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44c630 con 0x560a6d43d120 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x560a6d43d120 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94602782884224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44c5a0 con 0x560a6d43d120 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44d530 con 0x560a6d4415c0 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94602782878784] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94602782884224] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x560a6d43d120 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94602782886432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44d040 con 0x560a6d4415c0 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44de00 con 0x7f81d4036070 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x7f81d4036070 2024-04-27T03:19:58.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94602782890320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44d040 con 0x7f81d4036070 2024-04-27T03:19:58.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.050+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44e8a0 con 0x560a6d4415c0 2024-04-27T03:19:58.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94602782890320] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x7f81d4036070 2024-04-27T03:19:58.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94602782886432] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94602782882656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44e860 con 0x560a6d4415c0 2024-04-27T03:19:58.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44ec90 con 0x560a6d446600 2024-04-27T03:19:58.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94602782891376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44f4e0 con 0x560a6d446600 2024-04-27T03:19:58.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44fea0 con 0x560a6d446600 2024-04-27T03:19:58.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94602782882656] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94602782891376] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.054+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94602782896176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d44fea0 con 0x560a6d446600 2024-04-27T03:19:58.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d450770 con 0x7f81d4036070 2024-04-27T03:19:58.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'252 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x7f81d4036070 2024-04-27T03:19:58.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94602782896176] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94602782900928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d450b90 con 0x7f81d4036070 2024-04-27T03:19:58.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=0 2024-04-27T03:19:58.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=1 2024-04-27T03:19:58.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=2 2024-04-27T03:19:58.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=3 2024-04-27T03:19:58.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=4 2024-04-27T03:19:58.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=5 2024-04-27T03:19:58.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=6 2024-04-27T03:19:58.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94602782900928] v31'253 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x7f81d4036070 2024-04-27T03:19:58.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 20 add_watcher() i=7 2024-04-27T03:19:58.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:58.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d453bc0 con 0x560a6d43d120 2024-04-27T03:19:58.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f81cc03c1c0 con 0x560a6d43d120 2024-04-27T03:19:58.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.058+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94602782901984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d450fb0 con 0x560a6d43d120 2024-04-27T03:19:58.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94602782901984] v31'256 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x560a6d43d120 2024-04-27T03:19:58.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f8099fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:58.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:58.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 5 note: GC not initialized 2024-04-27T03:19:58.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f8098fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:58.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:58.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 Filter name: none 2024-04-27T03:19:58.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f80127fc640 20 reqs_thread_entry: start 2024-04-27T03:19:58.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 cache get: name=default.rgw.meta+users.uid+a_user : miss 2024-04-27T03:19:58.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 8.14 8:2efb937b:users.uid::a_user:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d477100 con 0x560a6d4415c0 2024-04-27T03:19:58.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 a_user [call,read 0~0,getxattrs] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 234+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 cache put: name=default.rgw.meta+users.uid+a_user info.flags=0x0 2024-04-27T03:19:58.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 adding default.rgw.meta+users.uid+a_user to cache LRU end 2024-04-27T03:19:58.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 cache get: name=default.rgw.meta+users.keys+true_admin_akey : miss 2024-04-27T03:19:58.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:63 8.15 8:afe09604:users.keys::true_admin_akey:head [stat,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d475740 con 0x560a6d446600 2024-04-27T03:19:58.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(63 true_admin_akey [stat,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 201+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 cache put: name=default.rgw.meta+users.keys+true_admin_akey info.flags=0x0 2024-04-27T03:19:58.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 adding default.rgw.meta+users.keys+true_admin_akey to cache LRU end 2024-04-27T03:19:58.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 cache get: name=default.rgw.meta+users.keys+true_admin_akey : hit (negative entry) 2024-04-27T03:19:58.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 10 cache get: name=default.rgw.meta+users.keys+true_admin_akey : hit (negative entry) 2024-04-27T03:19:58.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.062+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 8.14 8:2efb937b:users.uid::a_user:head [delete,create,call version.set in=58b,writefull 0~363 in=363b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d475fe0 con 0x560a6d4415c0 2024-04-27T03:19:58.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 a_user [delete,create,call,writefull 0~363] v31'3 uv3 ondisk = 0) ==== 276+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 10 cache put: name=default.rgw.meta+users.uid+a_user info.flags=0x17 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 10 moving default.rgw.meta+users.uid+a_user to cache LRU end 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:a_user, ofs0, ns] 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.15 7:a93a5511:::notify.2:head [notify cookie 94602783054640 in=554b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d47a6f0 con 0x560a6d4415c0 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== watch-notify(notify (1) cookie 94602782886432 notify 133143986231 ret 0) ==== 584+0+0 (crc 0 0 0) 0x560a6d476b50 con 0x560a6d4415c0 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986231 cookie 94602782886432 notifier 4849 bl.length()=542 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c6ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+a_user info.flags=0x17 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c6ffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+a_user to cache LRU end 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(65 notify.2 [notify cookie 94602783054640 out=8b] v0'0 uv252 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c6ffd640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f80a8003eb0 con 0x560a6d4415c0 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== watch-notify(notify_complete (2) cookie 94602783054640 notify 133143986231 ret 0) ==== 42+0+48 (crc 0 0 0) 0x560a6d432260 con 0x560a6d4415c0 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 8.15 8:afe09604:users.keys::true_admin_akey:head [delete,create,writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d476050 con 0x560a6d446600 2024-04-27T03:19:58.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(66 notify.2 [notify-ack] v0'0 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 true_admin_akey [delete,create,writefull 0~10] v31'3 uv3 ondisk = 0) ==== 243+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 10 cache put: name=default.rgw.meta+users.keys+true_admin_akey info.flags=0x7 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 10 moving default.rgw.meta+users.keys+true_admin_akey to cache LRU end 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:true_admin_akey, ofs0, ns] 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:68 7.3 7:c609908c:::notify.5:head [notify cookie 94602783054640 in=187b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x560a6d47a800 con 0x560a6d446600 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== watch-notify(notify (1) cookie 94602782891376 notify 133143986212 ret 0) ==== 217+0+0 (crc 0 0 0) 0x7f81dc0da040 con 0x560a6d446600 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986212 cookie 94602782891376 notifier 4849 bl.length()=175 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(68 notify.5 [notify cookie 94602783054640 out=8b] v0'0 uv252 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c67fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+true_admin_akey info.flags=0x7 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c67fc640 10 rgw watcher librados: moving default.rgw.meta+users.keys+true_admin_akey to cache LRU end 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.066+0000 7f80c67fc640 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:69 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f8018003410 con 0x560a6d446600 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.070+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(69 notify.5 [notify-ack] v0'0 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.070+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== watch-notify(notify_complete (2) cookie 94602783054640 notify 133143986212 ret 0) ==== 42+0+48 (crc 0 0 0) 0x560a6d405a70 con 0x560a6d446600 2024-04-27T03:19:58.082 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"a_user","display_name":"True Admin User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"a_user","access_key":"true_admin_akey","secret_key":"true_admin_skey","active":true,"create_date":"2024-04-27T03:19:58.067885Z"}],"swift_keys":[],"caps":[{"type":"user-info-without-keys","perm":"read"}],"op_mask":"read, write, delete","admin":true,"default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:58.067876Z","tags":[],"group_ids":[]} 2024-04-27T03:19:58.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.070+0000 7f8099fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.070+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:70 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94602782901984] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x560a6d43d120 2024-04-27T03:19:58.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.070+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(70 data_loggenerations_metadata [watch unwatch cookie 94602782901984] v31'257 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x560a6d43d120 2024-04-27T03:19:58.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.070+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94602782878784] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x560a6d4415c0 2024-04-27T03:19:58.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.074+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.0 [watch unwatch cookie 94602782878784] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.074+0000 7f81e5b69980 20 remove_watcher() i=0 2024-04-27T03:19:58.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.074+0000 7f81e5b69980 2 removed watcher, disabling cache 2024-04-27T03:19:58.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.074+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:72 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94602782884224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x560a6d43d120 2024-04-27T03:19:58.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.074+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(72 notify.1 [watch unwatch cookie 94602782884224] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x560a6d43d120 2024-04-27T03:19:58.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.074+0000 7f81e5b69980 20 remove_watcher() i=1 2024-04-27T03:19:58.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.074+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:73 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94602782886432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x560a6d4415c0 2024-04-27T03:19:58.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(73 notify.2 [watch unwatch cookie 94602782886432] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e5b69980 20 remove_watcher() i=2 2024-04-27T03:19:58.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94602782890320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x7f81d4036070 2024-04-27T03:19:58.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(74 notify.3 [watch unwatch cookie 94602782890320] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x7f81d4036070 2024-04-27T03:19:58.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e5b69980 20 remove_watcher() i=3 2024-04-27T03:19:58.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:75 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94602782882656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x560a6d4415c0 2024-04-27T03:19:58.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e3e85640 1 -- 172.21.15.110:0/2356270827 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(75 notify.4 [watch unwatch cookie 94602782882656] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81d4033030 con 0x560a6d4415c0 2024-04-27T03:19:58.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e5b69980 20 remove_watcher() i=4 2024-04-27T03:19:58.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.078+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:76 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94602782891376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x560a6d446600 2024-04-27T03:19:58.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(76 notify.5 [watch unwatch cookie 94602782891376] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 20 remove_watcher() i=5 2024-04-27T03:19:58.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:77 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94602782896176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x560a6d446600 2024-04-27T03:19:58.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e3684640 1 -- 172.21.15.110:0/2356270827 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(77 notify.6 [watch unwatch cookie 94602782896176] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81dc0d1070 con 0x560a6d446600 2024-04-27T03:19:58.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 20 remove_watcher() i=6 2024-04-27T03:19:58.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:78 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94602782900928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x560a6d474e00 con 0x7f81d4036070 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(78 notify.7 [watch unwatch cookie 94602782900928] v31'254 uv252 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f81cc03c1c0 con 0x7f81d4036070 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 20 remove_watcher() i=7 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d4415c0 msgr2=0x560a6d4439b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d4415c0 0x560a6d4439b0 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d43d120 msgr2=0x560a6d43f5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d43d120 0x560a6d43f5c0 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 msgr2=0x7f81d4021700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f81d4036070 0x7f81d4021700 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560a6d446600 msgr2=0x560a6d4489f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x560a6d446600 0x560a6d4489f0 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80ac034650 msgr2=0x7f80ac036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80ac034650 0x7f80ac036b10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0b4fd0 tx=0x7f81dc0e0000 comp rx=0 tx=0).stop 2024-04-27T03:19:58.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 msgr2=0x560a6d431a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d431a90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f81d4018aa0 tx=0x7f81d40498c0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e2e83640 1 -- 172.21.15.110:0/2356270827 reap_dead start 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 shutdown_connections 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/2356270827 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d42a5c0 0x560a6d431a90 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 >> 172.21.15.110:0/2356270827 conn(0x560a6d426950 msgr2=0x560a6d423fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 shutdown_connections 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/2356270827 wait complete. 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80bc034650 msgr2=0x7f80bc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80bc034650 0x7f80bc036b10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x560a6d40d870 tx=0x7f81dc0b4f60 comp rx=0 tx=0).stop 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d41d440 msgr2=0x560a6d41d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d41d440 0x560a6d41d860 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f81d400aab0 tx=0x7f81d400af70 comp rx=0 tx=0).stop 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 shutdown_connections 2024-04-27T03:19:58.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80bc034650 0x7f80bc036b10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/3715089040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d41d440 0x560a6d41d860 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 >> 172.21.15.110:0/3715089040 conn(0x560a6d412250 msgr2=0x560a6d412d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 shutdown_connections 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/3715089040 wait complete. 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d40a820 msgr2=0x560a6d40cc10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d40a820 0x560a6d40cc10 crc :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d3137e0 msgr2=0x560a6d409550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d3137e0 0x560a6d409550 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c8034650 msgr2=0x7f80c8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c8034650 0x7f80c8036b10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f81cc001da0 tx=0x7f81cc001300 comp rx=0 tx=0).stop 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 msgr2=0x560a6d402670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d402670 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0b8040 tx=0x7f81dc019810 comp rx=0 tx=0).stop 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 shutdown_connections 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x560a6d40a820 0x560a6d40cc10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x560a6d3137e0 0x560a6d409550 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.082+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f80c8034650 0x7f80c8036b10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.086+0000 7f81e5b69980 1 --2- 172.21.15.110:0/535242923 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x560a6d282830 0x560a6d402670 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.086+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 >> 172.21.15.110:0/535242923 conn(0x560a6d3089c0 msgr2=0x560a6d3f5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.086+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 shutdown_connections 2024-04-27T03:19:58.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.086+0000 7f81e5b69980 1 -- 172.21.15.110:0/535242923 wait complete. 2024-04-27T03:19:58.105 DEBUG:tasks.util.rgw: json result: {'user_id': 'a_user', 'display_name': 'True Admin User', 'email': '', 'suspended': 0, 'max_buckets': 1000, 'subusers': [], 'keys': [{'user': 'a_user', 'access_key': 'true_admin_akey', 'secret_key': 'true_admin_skey', 'active': True, 'create_date': '2024-04-27T03:19:58.067885Z'}], 'swift_keys': [], 'caps': [{'type': 'user-info-without-keys', 'perm': 'read'}], 'op_mask': 'read, write, delete', 'admin': True, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:58.067876Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.105 ERROR:root:{'user_id': 'a_user', 'display_name': 'True Admin User', 'email': '', 'suspended': 0, 'max_buckets': 1000, 'subusers': [], 'keys': [{'user': 'a_user', 'access_key': 'true_admin_akey', 'secret_key': 'true_admin_skey', 'active': True, 'create_date': '2024-04-27T03:19:58.067885Z'}], 'swift_keys': [], 'caps': [{'type': 'user-info-without-keys', 'perm': 'read'}], 'op_mask': 'read, write, delete', 'admin': True, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:58.067876Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.105 ERROR:root:0 2024-04-27T03:19:58.106 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:58.110 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:58.111 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'rm', '--uid', 'a_user'] 2024-04-27T03:19:58.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'rm', '--uid', 'a_user'] 2024-04-27T03:19:58.111 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user rm --uid a_user 2024-04-27T03:19:58.176 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:58.177 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:58.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f76860bf980 1 Processor -- start 2024-04-27T03:19:58.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f76860bf980 1 -- start start 2024-04-27T03:19:58.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f76860bf980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559806fa3000 0x559807034320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f76860bf980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x559806eee0d0 con 0x559807034860 2024-04-27T03:19:58.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f76860bf980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x559806ee3430 con 0x559806fa3000 2024-04-27T03:19:58.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f7683bda640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559806fa3000 0x559807034320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f76833d9640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x559807034860 0x559807036c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44576/0 (socket says 172.21.15.110:44576) 2024-04-27T03:19:58.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f7683bda640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x559806fa3000 0x559807034320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51746/0 (socket says 172.21.15.110:51746) 2024-04-27T03:19:58.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f76833d9640 1 -- 172.21.15.110:0/418024174 learned_addr learned my addr 172.21.15.110:0/418024174 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-04-27T03:19:58.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f7683bda640 1 -- 172.21.15.110:0/418024174 >> v1:172.21.15.77:6789/0 conn(0x559807034860 legacy=0x559807036c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.166+0000 7f7683bda640 1 -- 172.21.15.110:0/418024174 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x559806edc510 con 0x559806fa3000 2024-04-27T03:19:58.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f7683bda640 1 --2- 172.21.15.110:0/418024174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807034320 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f767c00a420 tx=0x7f767c01afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7128b7620b9de4a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f7682bd8640 1 -- 172.21.15.110:0/418024174 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f767c0af070 con 0x559806fa3000 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f7682bd8640 1 -- 172.21.15.110:0/418024174 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f767c0aa530 con 0x559806fa3000 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f7682bd8640 1 -- 172.21.15.110:0/418024174 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f767c0aa850 con 0x559806fa3000 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f76860bf980 1 -- 172.21.15.110:0/418024174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 msgr2=0x559807034320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f76860bf980 1 --2- 172.21.15.110:0/418024174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807034320 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f767c00a420 tx=0x7f767c01afd0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f76860bf980 1 -- 172.21.15.110:0/418024174 shutdown_connections 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f76860bf980 1 --2- 172.21.15.110:0/418024174 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807034320 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f76860bf980 1 -- 172.21.15.110:0/418024174 >> 172.21.15.110:0/418024174 conn(0x559807029a40 msgr2=0x55980702be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f76860bf980 1 -- 172.21.15.110:0/418024174 shutdown_connections 2024-04-27T03:19:58.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.170+0000 7f76860bf980 1 -- 172.21.15.110:0/418024174 wait complete. 2024-04-27T03:19:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.182+0000 7f76860bf980 1 Processor -- start 2024-04-27T03:19:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.182+0000 7f76860bf980 1 -- start start 2024-04-27T03:19:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.182+0000 7f76860bf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x55980711c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.182+0000 7f76860bf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559806f322d0 con 0x559806fa3000 2024-04-27T03:19:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x55980711c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x55980711c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51758/0 (socket says 172.21.15.110:51758) 2024-04-27T03:19:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 -- 172.21.15.110:0/4037558607 learned_addr learned my addr 172.21.15.110:0/4037558607 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 -- 172.21.15.110:0/4037558607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559806eb8ae0 con 0x559806fa3000 2024-04-27T03:19:58.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 --2- 172.21.15.110:0/4037558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x55980711c6b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f767c0ad040 tx=0x7f767c0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71cd2ef2e4850c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7571e0b640 1 -- 172.21.15.110:0/4037558607 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f767c0af050 con 0x559806fa3000 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7571e0b640 1 -- 172.21.15.110:0/4037558607 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f767c0b0040 con 0x559806fa3000 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7571e0b640 1 -- 172.21.15.110:0/4037558607 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f767c0aaca0 con 0x559806fa3000 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 -- 172.21.15.110:0/4037558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 msgr2=0x55980711c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 --2- 172.21.15.110:0/4037558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x55980711c6b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f767c0ad040 tx=0x7f767c0033b0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 -- 172.21.15.110:0/4037558607 shutdown_connections 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 --2- 172.21.15.110:0/4037558607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x55980711c6b0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 -- 172.21.15.110:0/4037558607 >> 172.21.15.110:0/4037558607 conn(0x559807029a40 msgr2=0x559807118240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 -- 172.21.15.110:0/4037558607 shutdown_connections 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 -- 172.21.15.110:0/4037558607 wait complete. 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 Processor -- start 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 -- start start 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807123540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f76860bf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55980703ae30 con 0x559806fa3000 2024-04-27T03:19:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807123540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807123540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51772/0 (socket says 172.21.15.110:51772) 2024-04-27T03:19:58.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 -- 172.21.15.110:0/72755436 learned_addr learned my addr 172.21.15.110:0/72755436 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559807123a80 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.186+0000 7f7683bda640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807123540 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f767c018c00 tx=0x7f767c019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f756bfff640 1 -- 172.21.15.110:0/72755436 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f767c0c5070 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559807123d10 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559807126990 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f756bfff640 1 -- 172.21.15.110:0/72755436 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f767c0b6d60 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f756bfff640 1 -- 172.21.15.110:0/72755436 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f767c0cf060 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f756bfff640 1 -- 172.21.15.110:0/72755436 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f767c0c8070 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f756bfff640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c0346a0 0x7f756c036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f756bfff640 1 -- 172.21.15.110:0/72755436 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f767c0bb070 con 0x559806fa3000 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76833d9640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c0346a0 0x7f756c036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559807034860 0x55980712a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980712a9b0 con 0x559807034860 2024-04-27T03:19:58.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76843db640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559807034860 0x55980712a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76833d9640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c0346a0 0x7f756c036b60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f766c001da0 tx=0x7f766c001300 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76843db640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559807034860 0x55980712a470 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76843db640 1 -- 172.21.15.110:0/72755436 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807034860 2024-04-27T03:19:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55980712b710 0x55980712db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980712e130 con 0x55980712b710 2024-04-27T03:19:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f7683bda640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55980712b710 0x55980712db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.190+0000 7f7683bda640 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55980712b710 0x55980712db00 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f7683bda640 1 -- 172.21.15.110:0/72755436 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f767c0c7030 con 0x55980712b710 2024-04-27T03:19:58.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980712f9e0 con 0x559807034860 2024-04-27T03:19:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76843db640 1 -- 172.21.15.110:0/72755436 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f76740083f0 con 0x559807034860 2024-04-27T03:19:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980712ec40 con 0x55980712b710 2024-04-27T03:19:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f7683bda640 1 -- 172.21.15.110:0/72755436 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f767c0c7030 con 0x55980712b710 2024-04-27T03:19:58.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980712f030 con 0x559807034860 2024-04-27T03:19:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76843db640 1 -- 172.21.15.110:0/72755436 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f76740083f0 con 0x559807034860 2024-04-27T03:19:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 Processor -- start 2024-04-27T03:19:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 -- start start 2024-04-27T03:19:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980713e2c0 0x55980713e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x559806f36310 con 0x55980713e2c0 2024-04-27T03:19:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76843db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980713e2c0 0x55980713e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76843db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980713e2c0 0x55980713e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51782/0 (socket says 172.21.15.110:51782) 2024-04-27T03:19:58.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76843db640 1 -- 172.21.15.110:0/3945681089 learned_addr learned my addr 172.21.15.110:0/3945681089 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76843db640 1 -- 172.21.15.110:0/3945681089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55980713ec20 con 0x55980713e2c0 2024-04-27T03:19:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76843db640 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980713e2c0 0x55980713e6e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f767400aab0 tx=0x7f767400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f75697fa640 1 -- 172.21.15.110:0/3945681089 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7674007d30 con 0x55980713e2c0 2024-04-27T03:19:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55980713f6c0 con 0x55980713e2c0 2024-04-27T03:19:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f75697fa640 1 -- 172.21.15.110:0/3945681089 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f767401d070 con 0x55980713e2c0 2024-04-27T03:19:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55980713fb90 con 0x55980713e2c0 2024-04-27T03:19:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.194+0000 7f75697fa640 1 -- 172.21.15.110:0/3945681089 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7674022460 con 0x55980713e2c0 2024-04-27T03:19:58.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f75697fa640 1 -- 172.21.15.110:0/3945681089 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7674013070 con 0x55980713e2c0 2024-04-27T03:19:58.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f75697fa640 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f755c034650 0x7f755c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f75697fa640 1 -- 172.21.15.110:0/3945681089 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f767404b640 con 0x55980713e2c0 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f7683bda640 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f755c034650 0x7f755c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 Processor -- start 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f7683bda640 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f755c034650 0x7f755c036b10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55980712e760 tx=0x7f767c0e3950 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- start start 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x55980714b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55980711cd40 con 0x55980714b490 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76843db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x55980714b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76843db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x55980714b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51786/0 (socket says 172.21.15.110:51786) 2024-04-27T03:19:58.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76843db640 1 -- 172.21.15.110:0/2622154913 learned_addr learned my addr 172.21.15.110:0/2622154913 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76843db640 1 -- 172.21.15.110:0/2622154913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55980714be80 con 0x55980714b490 2024-04-27T03:19:58.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76843db640 1 --2- 172.21.15.110:0/2622154913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x55980714b8b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f76740163e0 tx=0x7f76740173a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ba2a764d8c559f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f7562ffd640 1 -- 172.21.15.110:0/2622154913 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7674047070 con 0x55980714b490 2024-04-27T03:19:58.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f7562ffd640 1 -- 172.21.15.110:0/2622154913 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76740227e0 con 0x55980714b490 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f7562ffd640 1 -- 172.21.15.110:0/2622154913 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7674018570 con 0x55980714b490 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- 172.21.15.110:0/2622154913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 msgr2=0x55980714b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 --2- 172.21.15.110:0/2622154913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x55980714b8b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f76740163e0 tx=0x7f76740173a0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- 172.21.15.110:0/2622154913 shutdown_connections 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 --2- 172.21.15.110:0/2622154913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x55980714b8b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- 172.21.15.110:0/2622154913 >> 172.21.15.110:0/2622154913 conn(0x5598071477d0 msgr2=0x559807145830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- 172.21.15.110:0/2622154913 shutdown_connections 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- 172.21.15.110:0/2622154913 wait complete. 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 Processor -- start 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.198+0000 7f76860bf980 1 -- start start 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x559807152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55980714c600 con 0x55980714b490 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76843db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x559807152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76843db640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x559807152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51794/0 (socket says 172.21.15.110:51794) 2024-04-27T03:19:58.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 learned_addr learned my addr 172.21.15.110:0/4013976590 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559807152e50 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76843db640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x559807152910 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7674016ae0 tx=0x7f76740498a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f75617fa640 1 -- 172.21.15.110:0/4013976590 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7674030020 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5598071530e0 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f75617fa640 1 -- 172.21.15.110:0/4013976590 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7674020c80 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559807155dc0 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f75617fa640 1 -- 172.21.15.110:0/4013976590 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7674047030 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f75617fa640 1 -- 172.21.15.110:0/4013976590 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7674033020 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f75617fa640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f754c034650 0x7f754c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f75617fa640 1 -- 172.21.15.110:0/4013976590 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f7674048070 con 0x55980714b490 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f7683bda640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f754c034650 0x7f754c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55980715e060 0x559807160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559807160c70 con 0x55980715e060 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76833d9640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55980715e060 0x559807160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f7683bda640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f754c034650 0x7f754c036b10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f767c0c0a30 tx=0x7f767c0e8000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76833d9640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55980715e060 0x559807160520 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f766c00e640 con 0x55980715e060 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 20 realm 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980715d4f0 con 0x55980715e060 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.202+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f766c00e640 con 0x55980715e060 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559807161740 con 0x55980715e060 2024-04-27T03:19:58.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f766c00e640 con 0x55980715e060 2024-04-27T03:19:58.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559807162520 0x559807164910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559807164fa0 con 0x559807162520 2024-04-27T03:19:58.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76843db640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559807162520 0x559807164910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76843db640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559807162520 0x559807164910 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5598071657f0 con 0x55980715e060 2024-04-27T03:19:58.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f766c00e640 con 0x55980715e060 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 20 searching for the correct realm 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559807167560 0x559807169950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559807169fd0 con 0x559807167560 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f7683bda640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559807167560 0x559807169950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f7683bda640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559807167560 0x559807169950 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559807169fd0 con 0x559807162520 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76843db640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7674035020 0x7f7674013a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559807169fd0 con 0x7f7674035020 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76833d9640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7674035020 0x7f7674013a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.206+0000 7f76833d9640 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7674035020 0x7f7674013a00 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c00e640 con 0x7f7674035020 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559807169fd0 con 0x55980715e060 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c00e640 con 0x55980715e060 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559807169fd0 con 0x55980715e060 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x559807169fd0 con 0x559807167560 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x55980715e060 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x55980715e060 2024-04-27T03:19:58.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x7f7674035020 2024-04-27T03:19:58.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x7f7674035020 2024-04-27T03:19:58.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x55980715e060 2024-04-27T03:19:58.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x7f7674035020 2024-04-27T03:19:58.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f766c029040 con 0x7f7674035020 2024-04-27T03:19:58.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.210+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807167560 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x55980715e060 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807167560 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x55980715e060 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x7f7674035020 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f766c029040 con 0x7f7674035020 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x55980715e060 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807167560 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x7f7674035020 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x7f7674035020 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x55980715e060 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x7f7674035020 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x7f7674035020 2024-04-27T03:19:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807167560 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x7f7674035020 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f766c029040 con 0x7f7674035020 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.214+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807167560 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f767c0fe040 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559807166140 con 0x55980715e060 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f766c029040 con 0x55980715e060 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5598071669e0 con 0x559807162520 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.218+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x559807165d80 con 0x55980715e060 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 zone default found 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 4 Realm: () 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:58.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 zonegroup default 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5598071665c0 con 0x7f7674035020 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980716c310 con 0x559807167560 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980716c6b0 con 0x559807167560 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 started sync module instance, tier type = 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716b7b0 con 0x559807162520 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94111442320288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716cff0 con 0x559807162520 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.222+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716d840 con 0x55980715e060 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94111442325488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716d920 con 0x55980715e060 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716e440 con 0x559807162520 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94111442320288] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94111442325488] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94111442326544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716e400 con 0x559807162520 2024-04-27T03:19:58.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716ece0 con 0x7f7674035020 2024-04-27T03:19:58.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94111442331696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716e400 con 0x7f7674035020 2024-04-27T03:19:58.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716f760 con 0x559807162520 2024-04-27T03:19:58.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.226+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94111442331696] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94111442326544] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94111442329360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716f760 con 0x559807162520 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980716fb50 con 0x559807167560 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94111442332752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5598071703a0 con 0x559807167560 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807170da0 con 0x559807167560 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94111442329360] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(56 notify.6 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94111442334320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807170d60 con 0x559807167560 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807171830 con 0x7f7674035020 2024-04-27T03:19:58.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(55 notify.5 [watch watch cookie 94111442332752] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'255 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94111442342752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807171830 con 0x7f7674035020 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 20 add_watcher() i=0 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 20 add_watcher() i=1 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 20 add_watcher() i=2 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 20 add_watcher() i=3 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 20 add_watcher() i=4 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.230+0000 7f76860bf980 20 add_watcher() i=5 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94111442334320] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 20 add_watcher() i=6 2024-04-27T03:19:58.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94111442342752] v31'256 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 20 add_watcher() i=7 2024-04-27T03:19:58.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:58.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5598071708b0 con 0x55980715e060 2024-04-27T03:19:58.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94111442340016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807171cb0 con 0x55980715e060 2024-04-27T03:19:58.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94111442340016] v31'258 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f7539fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:58.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:58.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.234+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717d250 con 0x559807162520 2024-04-27T03:19:58.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.238+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v31'60 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.238+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717d300 con 0x7f7674035020 2024-04-27T03:19:58.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.238+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 gc.1 [create,call,call,call] v31'38 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.238+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717d9a0 con 0x559807167560 2024-04-27T03:19:58.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.242+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v31'37 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.242+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ddd0 con 0x55980715e060 2024-04-27T03:19:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.242+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(65 gc.3 [create,call,call,call] v31'259 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.242+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807178ee0 con 0x559807162520 2024-04-27T03:19:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.246+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v31'88 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.246+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807178ee0 con 0x559807167560 2024-04-27T03:19:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.246+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v31'65 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.246+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807178ee0 con 0x559807162520 2024-04-27T03:19:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.246+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v31'40 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.250+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x559807178ee0 con 0x55980715e060 2024-04-27T03:19:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.250+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 gc.7 [create,call,call,call] v31'61 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.250+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717e930 con 0x559807162520 2024-04-27T03:19:58.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.254+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v31'30 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.254+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717e930 con 0x559807162520 2024-04-27T03:19:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.254+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v31'36 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.254+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807162520 2024-04-27T03:19:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.258+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v31'89 uv61 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.258+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807167560 2024-04-27T03:19:58.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.258+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v31'36 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.258+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807162520 2024-04-27T03:19:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.258+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v31'36 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.258+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x55980715e060 2024-04-27T03:19:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.262+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(75 gc.13 [create,call,call,call] v31'71 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.262+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x55980715e060 2024-04-27T03:19:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.262+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(76 gc.14 [create,call,call,call] v31'260 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.262+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807162520 2024-04-27T03:19:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.266+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v31'80 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.266+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807162520 2024-04-27T03:19:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.266+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v31'90 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.266+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807167560 2024-04-27T03:19:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.270+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v31'43 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.270+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807162520 2024-04-27T03:19:58.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.270+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v31'38 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.270+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x55980715e060 2024-04-27T03:19:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.274+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(81 gc.19 [create,call,call,call] v31'261 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.274+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807162520 2024-04-27T03:19:58.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.274+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v31'81 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.274+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807162520 2024-04-27T03:19:58.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.278+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v31'46 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.278+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x559807167560 2024-04-27T03:19:58.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.278+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v31'38 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.278+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717ede0 con 0x55980715e060 2024-04-27T03:19:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.278+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 gc.23 [create,call,call,call] v31'48 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.278+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x559807167560 2024-04-27T03:19:58.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.282+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v31'32 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.282+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x559807167560 2024-04-27T03:19:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.282+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v31'66 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.282+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x559807167560 2024-04-27T03:19:58.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.282+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v31'39 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.282+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x559807162520 2024-04-27T03:19:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.286+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(89 gc.27 [create,call,call,call] v31'43 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.286+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x55980715e060 2024-04-27T03:19:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.290+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v31'62 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.290+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x559807162520 2024-04-27T03:19:58.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.290+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(91 gc.29 [create,call,call,call] v31'82 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.290+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x55980715e060 2024-04-27T03:19:58.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.290+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v31'72 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.290+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980717efb0 con 0x7f7674035020 2024-04-27T03:19:58.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(93 gc.31 [create,call,call,call] v31'36 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f7538fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:58.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:58.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 20 Filter name: none 2024-04-27T03:19:58.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f74b27fc640 20 reqs_thread_entry: start 2024-04-27T03:19:58.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 10 cache get: name=default.rgw.meta+users.uid+a_user : miss 2024-04-27T03:19:58.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:94 8.14 8:2efb937b:users.uid::a_user:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980717c660 con 0x559807162520 2024-04-27T03:19:58.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(94 a_user [call out=48b,read 0~363 out=363b,getxattrs out=75b] v0'0 uv3 ondisk = 0) ==== 234+0+486 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 20 rados_obj.operate() r=0 bl.length=363 2024-04-27T03:19:58.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 10 cache put: name=default.rgw.meta+users.uid+a_user info.flags=0x13 2024-04-27T03:19:58.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 10 adding default.rgw.meta+users.uid+a_user to cache LRU end 2024-04-27T03:19:58.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:58.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 8.11 8:8d3ad08f:users.uid::a_user.buckets:head [call user.list_buckets in=34b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980719ca50 con 0x559807162520 2024-04-27T03:19:58.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(95 a_user.buckets [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 158+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 10 removing key index: true_admin_akey 2024-04-27T03:19:58.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.294+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:96 8.15 8:afe09604:users.keys::true_admin_akey:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719aa30 con 0x559807167560 2024-04-27T03:19:58.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(96 true_admin_akey [delete] v31'4 uv4 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76860bf980 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.keys:true_admin_akey, ofs0, ns] 2024-04-27T03:19:58.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:97 7.3 7:c609908c:::notify.5:head [notify cookie 94111442515344 in=177b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980719cf50 con 0x559807167560 2024-04-27T03:19:58.309 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== watch-notify(notify (1) cookie 94111442332752 notify 133143986213 ret 0) ==== 207+0+0 (crc 0 0 0) 0x55980717e930 con 0x559807167560 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f7562ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986213 cookie 94111442332752 notifier 4856 bl.length()=165 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f7562ffd640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:98 7.3 7:c609908c:::notify.5:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f7548003d70 con 0x559807167560 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(97 notify.5 [notify cookie 94111442515344 out=8b] v0'0 uv255 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(98 notify.5 [notify-ack] v0'0 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== watch-notify(notify_complete (2) cookie 94111442515344 notify 133143986213 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7f767c0ed020 con 0x559807167560 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76860bf980 10 removing email index: 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76860bf980 10 removing user buckets index 2024-04-27T03:19:58.310 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 8.11 8:8d3ad08f:users.uid::a_user.buckets:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719aa30 con 0x559807162520 2024-04-27T03:19:58.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(99 a_user.buckets [delete] v31'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 158+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76860bf980 10 removing user index: a_user 2024-04-27T03:19:58.311 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.298+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 8.14 8:2efb937b:users.uid::a_user:head [call version.check_conds in=122b,call version.set in=58b,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x559807162520 2024-04-27T03:19:58.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(100 a_user [call,call,delete] v31'4 uv4 ondisk = 0) ==== 234+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76860bf980 10 removing default.rgw.meta+users.uid+a_user from cache 2024-04-27T03:19:58.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76860bf980 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:a_user, ofs0, ns] 2024-04-27T03:19:58.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 7.15 7:a93a5511:::notify.2:head [notify cookie 94111442519856 in=167b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x559807162520 2024-04-27T03:19:58.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== watch-notify(notify (1) cookie 94111442326544 notify 133143986232 ret 0) ==== 197+0+0 (crc 0 0 0) 0x559807178a30 con 0x559807162520 2024-04-27T03:19:58.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f75627fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986232 cookie 94111442326544 notifier 4856 bl.length()=155 2024-04-27T03:19:58.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f75627fc640 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:102 7.15 7:a93a5511:::notify.2:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f74b80032d0 con 0x559807162520 2024-04-27T03:19:58.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(101 notify.2 [notify cookie 94111442519856 out=8b] v0'0 uv255 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(102 notify.2 [notify-ack] v0'0 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== watch-notify(notify_complete (2) cookie 94111442519856 notify 133143986232 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5598071530e0 con 0x559807162520 2024-04-27T03:19:58.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f7539fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.302+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:103 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94111442340016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x55980715e060 2024-04-27T03:19:58.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.306+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(103 data_loggenerations_metadata [watch unwatch cookie 94111442340016] v31'262 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.306+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:104 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94111442320288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x559807162520 2024-04-27T03:19:58.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.306+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(104 notify.0 [watch unwatch cookie 94111442320288] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.306+0000 7f76860bf980 20 remove_watcher() i=0 2024-04-27T03:19:58.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.306+0000 7f76860bf980 2 removed watcher, disabling cache 2024-04-27T03:19:58.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.306+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94111442325488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x55980715e060 2024-04-27T03:19:58.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(105 notify.1 [watch unwatch cookie 94111442325488] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x55980715e060 2024-04-27T03:19:58.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76860bf980 20 remove_watcher() i=1 2024-04-27T03:19:58.319 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:106 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94111442326544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x559807162520 2024-04-27T03:19:58.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(106 notify.2 [watch unwatch cookie 94111442326544] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76860bf980 20 remove_watcher() i=2 2024-04-27T03:19:58.321 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:107 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94111442331696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x7f7674035020 2024-04-27T03:19:58.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(107 notify.3 [watch unwatch cookie 94111442331696] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76860bf980 20 remove_watcher() i=3 2024-04-27T03:19:58.322 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.310+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94111442329360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x559807162520 2024-04-27T03:19:58.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f76843db640 1 -- 172.21.15.110:0/4013976590 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(108 notify.4 [watch unwatch cookie 94111442329360] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f76740083f0 con 0x559807162520 2024-04-27T03:19:58.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f76860bf980 20 remove_watcher() i=4 2024-04-27T03:19:58.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:109 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94111442332752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x559807167560 2024-04-27T03:19:58.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(109 notify.5 [watch unwatch cookie 94111442332752] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f76860bf980 20 remove_watcher() i=5 2024-04-27T03:19:58.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:110 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94111442334320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x559807167560 2024-04-27T03:19:58.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f7683bda640 1 -- 172.21.15.110:0/4013976590 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(110 notify.6 [watch unwatch cookie 94111442334320] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f767c0c7030 con 0x559807167560 2024-04-27T03:19:58.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f76860bf980 20 remove_watcher() i=6 2024-04-27T03:19:58.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.314+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:111 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94111442342752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55980719cc50 con 0x7f7674035020 2024-04-27T03:19:58.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(111 notify.7 [watch unwatch cookie 94111442342752] v31'257 uv255 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f766c03c1c0 con 0x7f7674035020 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 20 remove_watcher() i=7 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559807162520 msgr2=0x559807164910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x559807162520 0x559807164910 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55980715e060 msgr2=0x559807160520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55980715e060 0x559807160520 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7674035020 msgr2=0x7f7674013a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f7674035020 0x7f7674013a00 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559807167560 msgr2=0x559807169950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x559807167560 0x559807169950 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f754c034650 msgr2=0x7f754c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f754c034650 0x7f754c036b10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f767c0c0a30 tx=0x7f767c0e8000 comp rx=0 tx=0).stop 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76833d9640 1 -- 172.21.15.110:0/4013976590 reap_dead start 2024-04-27T03:19:58.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 msgr2=0x559807152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x559807152910 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7674016ae0 tx=0x7f76740498a0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 shutdown_connections 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/4013976590 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980714b490 0x559807152910 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 >> 172.21.15.110:0/4013976590 conn(0x5598071477d0 msgr2=0x559807144e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 shutdown_connections 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/4013976590 wait complete. 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f755c034650 msgr2=0x7f755c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f755c034650 0x7f755c036b10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55980712e760 tx=0x7f767c0e3950 comp rx=0 tx=0).stop 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980713e2c0 msgr2=0x55980713e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980713e2c0 0x55980713e6e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f767400aab0 tx=0x7f767400af70 comp rx=0 tx=0).stop 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 shutdown_connections 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f755c034650 0x7f755c036b10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/3945681089 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55980713e2c0 0x55980713e6e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 >> 172.21.15.110:0/3945681089 conn(0x5598071330d0 msgr2=0x559807133be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 shutdown_connections 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/3945681089 wait complete. 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55980712b710 msgr2=0x55980712db00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55980712b710 0x55980712db00 crc :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559807034860 msgr2=0x55980712a470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559807034860 0x55980712a470 crc :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c0346a0 msgr2=0x7f756c036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c0346a0 0x7f756c036b60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f766c001da0 tx=0x7f766c001300 comp rx=0 tx=0).stop 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 msgr2=0x559807123540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807123540 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f767c018c00 tx=0x7f767c019810 comp rx=0 tx=0).stop 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 shutdown_connections 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55980712b710 0x55980712db00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x559807034860 0x55980712a470 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f756c0346a0 0x7f756c036b60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 --2- 172.21.15.110:0/72755436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x559806fa3000 0x559807123540 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 >> 172.21.15.110:0/72755436 conn(0x559807029a40 msgr2=0x559807116d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 shutdown_connections 2024-04-27T03:19:58.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.318+0000 7f76860bf980 1 -- 172.21.15.110:0/72755436 wait complete. 2024-04-27T03:19:58.338 ERROR:root:None 2024-04-27T03:19:58.338 ERROR:root:0 2024-04-27T03:19:58.338 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'ada', '--caps', 'user-info-without-keys=read'] 2024-04-27T03:19:58.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'ada', '--caps', 'user-info-without-keys=read'] 2024-04-27T03:19:58.338 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid ada --caps user-info-without-keys=read 2024-04-27T03:19:58.418 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:58.419 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:58.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c2f0f980 1 Processor -- start 2024-04-27T03:19:58.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c2f0f980 1 -- start start 2024-04-27T03:19:58.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c2f0f980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b88a3e6c0 0x556b88a40ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c2f0f980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x556b888fa0d0 con 0x556b88a3a160 2024-04-27T03:19:58.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c2f0f980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x556b888ef430 con 0x556b88a3e6c0 2024-04-27T03:19:58.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c0229640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b88a3e6c0 0x556b88a40ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c0229640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556b88a3e6c0 0x556b88a40ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51800/0 (socket says 172.21.15.110:51800) 2024-04-27T03:19:58.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c0229640 1 -- 172.21.15.110:0/4001216674 learned_addr learned my addr 172.21.15.110:0/4001216674 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c0229640 1 -- 172.21.15.110:0/4001216674 >> v1:172.21.15.77:6789/0 conn(0x556b88a3a160 legacy=0x556b88a3a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c0229640 1 -- 172.21.15.110:0/4001216674 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b888e8510 con 0x556b88a3e6c0 2024-04-27T03:19:58.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.410+0000 7fc6c0229640 1 --2- 172.21.15.110:0/4001216674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3e6c0 0x556b88a40ab0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800a5c0 tx=0x7fc6a801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cdb3238c2107133 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6bfa28640 1 -- 172.21.15.110:0/4001216674 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6a80a6070 con 0x556b88a3e6c0 2024-04-27T03:19:58.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6bfa28640 1 -- 172.21.15.110:0/4001216674 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a80af070 con 0x556b88a3e6c0 2024-04-27T03:19:58.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6bfa28640 1 -- 172.21.15.110:0/4001216674 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6a80aa5c0 con 0x556b88a3e6c0 2024-04-27T03:19:58.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/4001216674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3e6c0 msgr2=0x556b88a40ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/4001216674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3e6c0 0x556b88a40ab0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800a5c0 tx=0x7fc6a801b340 comp rx=0 tx=0).stop 2024-04-27T03:19:58.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/4001216674 shutdown_connections 2024-04-27T03:19:58.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/4001216674 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3e6c0 0x556b88a40ab0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/4001216674 >> 172.21.15.110:0/4001216674 conn(0x556b88a35a30 msgr2=0x556b88a37e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/4001216674 shutdown_connections 2024-04-27T03:19:58.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.414+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/4001216674 wait complete. 2024-04-27T03:19:58.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c2f0f980 1 Processor -- start 2024-04-27T03:19:58.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c2f0f980 1 -- start start 2024-04-27T03:19:58.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c2f0f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b889aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c2f0f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b8893e2d0 con 0x556b88a3a160 2024-04-27T03:19:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c0a2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b889aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c0a2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b889aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51810/0 (socket says 172.21.15.110:51810) 2024-04-27T03:19:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/2282013541 learned_addr learned my addr 172.21.15.110:0/2282013541 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.438 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/2282013541 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b888c4ae0 con 0x556b88a3a160 2024-04-27T03:19:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/2282013541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b889aef70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc6b00015f0 tx=0x7fc6b0001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b198f5289eee04 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc5aee0d640 1 -- 172.21.15.110:0/2282013541 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b0016070 con 0x556b88a3a160 2024-04-27T03:19:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc5aee0d640 1 -- 172.21.15.110:0/2282013541 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b000f640 con 0x556b88a3a160 2024-04-27T03:19:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.426+0000 7fc5aee0d640 1 -- 172.21.15.110:0/2282013541 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b000f960 con 0x556b88a3a160 2024-04-27T03:19:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/2282013541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 msgr2=0x556b889aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/2282013541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b889aef70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc6b00015f0 tx=0x7fc6b0001ab0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/2282013541 shutdown_connections 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/2282013541 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b889aef70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/2282013541 >> 172.21.15.110:0/2282013541 conn(0x556b88a35a30 msgr2=0x556b88b20f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/2282013541 shutdown_connections 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/2282013541 wait complete. 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 Processor -- start 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- start start 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b88b2d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b88a43ae0 con 0x556b88a3a160 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c0a2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b88b2d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c0a2a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b88b2d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51818/0 (socket says 172.21.15.110:51818) 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/1361327906 learned_addr learned my addr 172.21.15.110:0/1361327906 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b88b2da10 con 0x556b88a3a160 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b88b2d4d0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0001af0 tx=0x7fc6b0000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc5ad60a640 1 -- 172.21.15.110:0/1361327906 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b0016040 con 0x556b88a3a160 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b88b2dca0 con 0x556b88a3a160 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc5ad60a640 1 -- 172.21.15.110:0/1361327906 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b0022070 con 0x556b88a3a160 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b88b30920 con 0x556b88a3a160 2024-04-27T03:19:58.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc5ad60a640 1 -- 172.21.15.110:0/1361327906 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b0006180 con 0x556b88a3a160 2024-04-27T03:19:58.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc5ad60a640 1 -- 172.21.15.110:0/1361327906 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc6b00117c0 con 0x556b88a3a160 2024-04-27T03:19:58.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc5ad60a640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc5a8034650 0x7fc5a8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc5ad60a640 1 -- 172.21.15.110:0/1361327906 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc6b0049a70 con 0x556b88a3a160 2024-04-27T03:19:58.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c0229640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc5a8034650 0x7fc5a8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88a3e6c0 0x556b88b34400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.430+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b34a20 con 0x556b88a3e6c0 2024-04-27T03:19:58.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c122b640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88a3e6c0 0x556b88b34400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c0229640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc5a8034650 0x7fc5a8036b10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80ae000 tx=0x7fc6a80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c122b640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88a3e6c0 0x556b88b34400 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c122b640 1 -- 172.21.15.110:0/1361327906 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc6b80083f0 con 0x556b88a3e6c0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b357f0 0x556b88b37be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b38220 con 0x556b88b357f0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b357f0 0x556b88b37be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b357f0 0x556b88b37be0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/1361327906 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b357f0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b39ad0 con 0x556b88a3e6c0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c122b640 1 -- 172.21.15.110:0/1361327906 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fc6b80083f0 con 0x556b88a3e6c0 2024-04-27T03:19:58.447 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b38d00 con 0x556b88b357f0 2024-04-27T03:19:58.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/1361327906 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b357f0 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b390f0 con 0x556b88a3e6c0 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.434+0000 7fc6c122b640 1 -- 172.21.15.110:0/1361327906 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fc6b80083f0 con 0x556b88a3e6c0 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 Processor -- start 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 -- start start 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b48410 0x556b88b48830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b88942310 con 0x556b88b48410 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c122b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b48410 0x556b88b48830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c122b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b48410 0x556b88b48830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51828/0 (socket says 172.21.15.110:51828) 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c122b640 1 -- 172.21.15.110:0/3157158925 learned_addr learned my addr 172.21.15.110:0/3157158925 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c122b640 1 -- 172.21.15.110:0/3157158925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b88b48d70 con 0x556b88b48410 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c122b640 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b48410 0x556b88b48830 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800aab0 tx=0x7fc6b800af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc5a67fc640 1 -- 172.21.15.110:0/3157158925 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b8007d30 con 0x556b88b48410 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b88b49810 con 0x556b88b48410 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc5a67fc640 1 -- 172.21.15.110:0/3157158925 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b801d070 con 0x556b88b48410 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc5a67fc640 1 -- 172.21.15.110:0/3157158925 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b8018440 con 0x556b88b48410 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b88b49ce0 con 0x556b88b48410 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc5a67fc640 1 -- 172.21.15.110:0/3157158925 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc6b8013070 con 0x556b88b48410 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc5a67fc640 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc598034650 0x7fc598036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc5a67fc640 1 -- 172.21.15.110:0/3157158925 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc6b804b670 con 0x556b88b48410 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc598034650 0x7fc598036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 Processor -- start 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 -- start start 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc598034650 0x7fc598036b10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556b88b38820 tx=0x7fc6b0047c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b26c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.438+0000 7fc6c2f0f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b889af6c0 con 0x556b88b26800 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b26c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b26c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51836/0 (socket says 172.21.15.110:51836) 2024-04-27T03:19:58.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 -- 172.21.15.110:0/931774258 learned_addr learned my addr 172.21.15.110:0/931774258 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 -- 172.21.15.110:0/931774258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b88b25050 con 0x556b88b26800 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 --2- 172.21.15.110:0/931774258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b26c20 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8049970 tx=0x7fc6b803a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c0b88014219a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc597fff640 1 -- 172.21.15.110:0/931774258 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b803f020 con 0x556b88b26800 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc597fff640 1 -- 172.21.15.110:0/931774258 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b803b310 con 0x556b88b26800 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc597fff640 1 -- 172.21.15.110:0/931774258 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b8022460 con 0x556b88b26800 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/931774258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 msgr2=0x556b88b26c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/931774258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b26c20 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8049970 tx=0x7fc6b803a110 comp rx=0 tx=0).stop 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/931774258 shutdown_connections 2024-04-27T03:19:58.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/931774258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b26c20 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/931774258 >> 172.21.15.110:0/931774258 conn(0x556b88b51920 msgr2=0x556b88b4f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/931774258 shutdown_connections 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/931774258 wait complete. 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 Processor -- start 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- start start 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b5ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556b88b25520 con 0x556b88b26800 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b5ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b5ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51838/0 (socket says 172.21.15.110:51838) 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 learned_addr learned my addr 172.21.15.110:0/560277078 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b88b5f1a0 con 0x556b88b26800 2024-04-27T03:19:58.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c122b640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b5ec60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x556b88b506b0 tx=0x7fc6b80498d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc5967fc640 1 -- 172.21.15.110:0/560277078 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b802f020 con 0x556b88b26800 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556b88b5f430 con 0x556b88b26800 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b88b62110 con 0x556b88b26800 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc5967fc640 1 -- 172.21.15.110:0/560277078 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6b8020c80 con 0x556b88b26800 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.442+0000 7fc5967fc640 1 -- 172.21.15.110:0/560277078 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc6b803f040 con 0x556b88b26800 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc5967fc640 1 -- 172.21.15.110:0/560277078 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc6b8032020 con 0x556b88b26800 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc5967fc640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc58c034650 0x7fc58c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc5967fc640 1 -- 172.21.15.110:0/560277078 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc6b8035020 con 0x556b88b26800 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc58c034650 0x7fc58c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88b6a300 0x556b88b6c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b6cef0 con 0x556b88b6a300 2024-04-27T03:19:58.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c0229640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88b6a300 0x556b88b6c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc58c034650 0x7fc58c036b10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc6b001cf70 tx=0x7fc6b0047f60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c0229640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88b6a300 0x556b88b6c7a0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 realm 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b6d320 con 0x556b88b6a300 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b6dbc0 con 0x556b88b6a300 2024-04-27T03:19:58.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b6e5e0 0x556b88b70a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b711a0 con 0x556b88b6e5e0 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c122b640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b6e5e0 0x556b88b70a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c122b640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b6e5e0 0x556b88b70a30 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.446+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b719f0 con 0x556b88b6a300 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c2f0f980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c2f0f980 20 searching for the correct realm 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b88b73760 0x556b88b75b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b73760 2024-04-27T03:19:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b88b73760 0x556b88b75b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0a2a640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b88b73760 0x556b88b75b50 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c122b640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc6b8034020 0x7fc6b803b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x7fc6b8034020 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc6b8034020 0x7fc6b803b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc6b8034020 0x7fc6b803b9c0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b73760 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.450+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x7fc6b8034020 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x7fc6b8034020 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b73760 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b73760 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.454+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x7fc6b8034020 2024-04-27T03:19:58.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b73760 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x7fc6b8034020 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6a300 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x7fc6b8034020 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b73760 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x7fc6b8034020 2024-04-27T03:19:58.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b73760 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.458+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76160 con 0x556b88b6e5e0 2024-04-27T03:19:58.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b72370 con 0x556b88b6a300 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b72c10 con 0x556b88b6e5e0 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b71fb0 con 0x556b88b6a300 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 zone default found 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 4 Realm: () 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 zonegroup default 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b76a70 con 0x7fc6b8034020 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b78d20 con 0x556b88b73760 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b790c0 con 0x556b88b73760 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 started sync module instance, tier type = 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.462+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b78160 con 0x556b88b6e5e0 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93920343590224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b78970 con 0x556b88b6e5e0 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b79e40 con 0x556b88b6a300 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93920343595280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7a6e0 con 0x556b88b6a300 2024-04-27T03:19:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7b0a0 con 0x556b88b6e5e0 2024-04-27T03:19:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93920343590224] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.466+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 93920343595280] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93920343597616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7b0a0 con 0x556b88b6e5e0 2024-04-27T03:19:58.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7a650 con 0x7fc6b8034020 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93920343602368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7b0a0 con 0x7fc6b8034020 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7c3f0 con 0x556b88b6e5e0 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93920343597616] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.470+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 93920343602368] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93920343603424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7abf0 con 0x556b88b6e5e0 2024-04-27T03:19:58.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7cc90 con 0x556b88b73760 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93920343607264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7abf0 con 0x556b88b73760 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7da10 con 0x556b88b73760 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93920343603424] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93920343607264] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93920343609392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7de00 con 0x556b88b73760 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7e650 con 0x7fc6b8034020 2024-04-27T03:19:58.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93920343609392] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'258 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93920343612928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7ea40 con 0x7fc6b8034020 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 20 add_watcher() i=0 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 20 add_watcher() i=1 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 20 add_watcher() i=2 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 20 add_watcher() i=3 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 20 add_watcher() i=4 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 20 add_watcher() i=5 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.474+0000 7fc6c2f0f980 20 add_watcher() i=6 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 93920343612928] v31'259 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 20 add_watcher() i=7 2024-04-27T03:19:58.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:58.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88b81aa0 con 0x556b88b6a300 2024-04-27T03:19:58.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93920343614448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88b7ee50 con 0x556b88b6a300 2024-04-27T03:19:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93920343614448] v31'263 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc576fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 5 note: GC not initialized 2024-04-27T03:19:58.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc575fe3640 20 reqs_thread_entry: start 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.478+0000 7fc6c2f0f980 20 Filter name: none 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc4e77fe640 20 reqs_thread_entry: start 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba5320 con 0x556b88b6a300 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~460 out=460b,getxattrs out=75b] v0'0 uv5 ondisk = 0) ==== 231+0+583 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 20 rados_obj.operate() r=0 bl.length=460 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~490 in=490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88bab930 con 0x556b88b6a300 2024-04-27T03:19:58.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~490] v31'6 uv6 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 93920343798688 in=678b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba7040 con 0x556b88b6e5e0 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 93920343603424 notify 133143986233 ret 0) ==== 708+0+0 (crc 0 0 0) 0x556b88b62110 con 0x556b88b6e5e0 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc597fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986233 cookie 93920343603424 notifier 4863 bl.length()=666 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc597fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc597fff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc597fff640 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fc584003eb0 con 0x556b88b6e5e0 2024-04-27T03:19:58.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 93920343798688 out=8b] v0'0 uv258 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.482+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 93920343798688 notify 133143986233 ret 0) ==== 42+0+48 (crc 0 0 0) 0x556b88b5f430 con 0x556b88b6e5e0 2024-04-27T03:19:58.496 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"user-info-without-keys","perm":"read"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:58.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.486+0000 7fc576fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.486+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93920343614448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x556b88b6a300 2024-04-27T03:19:58.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.486+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 93920343614448] v31'264 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.486+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93920343590224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x556b88b6e5e0 2024-04-27T03:19:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 93920343590224] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c2f0f980 20 remove_watcher() i=0 2024-04-27T03:19:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c2f0f980 2 removed watcher, disabling cache 2024-04-27T03:19:58.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93920343595280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x556b88b6a300 2024-04-27T03:19:58.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 93920343595280] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x556b88b6a300 2024-04-27T03:19:58.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c2f0f980 20 remove_watcher() i=1 2024-04-27T03:19:58.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93920343597616] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x556b88b6e5e0 2024-04-27T03:19:58.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 93920343597616] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c2f0f980 20 remove_watcher() i=2 2024-04-27T03:19:58.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.490+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93920343602368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x7fc6b8034020 2024-04-27T03:19:58.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 93920343602368] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c2f0f980 20 remove_watcher() i=3 2024-04-27T03:19:58.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93920343603424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x556b88b6e5e0 2024-04-27T03:19:58.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c122b640 1 -- 172.21.15.110:0/560277078 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 93920343603424] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b8041050 con 0x556b88b6e5e0 2024-04-27T03:19:58.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c2f0f980 20 remove_watcher() i=4 2024-04-27T03:19:58.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93920343607264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x556b88b73760 2024-04-27T03:19:58.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 93920343607264] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c2f0f980 20 remove_watcher() i=5 2024-04-27T03:19:58.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.494+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93920343609392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x556b88b73760 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c0a2a640 1 -- 172.21.15.110:0/560277078 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 93920343609392] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6b000f070 con 0x556b88b73760 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 20 remove_watcher() i=6 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93920343612928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556b88ba3020 con 0x7fc6b8034020 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 93920343612928] v31'260 uv258 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc6a80aaa20 con 0x7fc6b8034020 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 20 remove_watcher() i=7 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b6e5e0 msgr2=0x556b88b70a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b6e5e0 0x556b88b70a30 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88b6a300 msgr2=0x556b88b6c7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88b6a300 0x556b88b6c7a0 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc6b8034020 msgr2=0x7fc6b803b9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc6b8034020 0x7fc6b803b9c0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b88b73760 msgr2=0x556b88b75b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556b88b73760 0x556b88b75b50 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc58c034650 msgr2=0x7fc58c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc58c034650 0x7fc58c036b10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc6b001cf70 tx=0x7fc6b0047f60 comp rx=0 tx=0).stop 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 msgr2=0x556b88b5ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c0229640 1 -- 172.21.15.110:0/560277078 reap_dead start 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b5ec60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x556b88b506b0 tx=0x7fc6b80498d0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 shutdown_connections 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/560277078 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b26800 0x556b88b5ec60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 >> 172.21.15.110:0/560277078 conn(0x556b88b51920 msgr2=0x556b88b4ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 shutdown_connections 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/560277078 wait complete. 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc598034650 msgr2=0x7fc598036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc598034650 0x7fc598036b10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556b88b38820 tx=0x7fc6b0047c10 comp rx=0 tx=0).stop 2024-04-27T03:19:58.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b48410 msgr2=0x556b88b48830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b48410 0x556b88b48830 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc6b800aab0 tx=0x7fc6b800af70 comp rx=0 tx=0).stop 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 shutdown_connections 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc598034650 0x7fc598036b10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/3157158925 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88b48410 0x556b88b48830 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 >> 172.21.15.110:0/3157158925 conn(0x556b88b3d220 msgr2=0x556b88b3dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 shutdown_connections 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/3157158925 wait complete. 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b357f0 msgr2=0x556b88b37be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b357f0 0x556b88b37be0 crc :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88a3e6c0 msgr2=0x556b88b34400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88a3e6c0 0x556b88b34400 crc :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc5a8034650 msgr2=0x7fc5a8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc5a8034650 0x7fc5a8036b10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80ae000 tx=0x7fc6a80a6000 comp rx=0 tx=0).stop 2024-04-27T03:19:58.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 msgr2=0x556b88b2d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b88b2d4d0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0001af0 tx=0x7fc6b0000f50 comp rx=0 tx=0).stop 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 shutdown_connections 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556b88b357f0 0x556b88b37be0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556b88a3e6c0 0x556b88b34400 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc5a8034650 0x7fc5a8036b10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 --2- 172.21.15.110:0/1361327906 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556b88a3a160 0x556b88b2d4d0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 >> 172.21.15.110:0/1361327906 conn(0x556b88a35a30 msgr2=0x556b88b1fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 shutdown_connections 2024-04-27T03:19:58.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.498+0000 7fc6c2f0f980 1 -- 172.21.15.110:0/1361327906 wait complete. 2024-04-27T03:19:58.519 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': 'read'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.519 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': 'read'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.519 ERROR:root:0 2024-04-27T03:19:58.519 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:58.524 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:58.524 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'ada', '--caps', 'user-info-without-keys=*'] 2024-04-27T03:19:58.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'ada', '--caps', 'user-info-without-keys=*'] 2024-04-27T03:19:58.524 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid ada --caps 'user-info-without-keys=*' 2024-04-27T03:19:58.600 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:58.600 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:58.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a98ed980 1 Processor -- start 2024-04-27T03:19:58.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a98ed980 1 -- start start 2024-04-27T03:19:58.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a98ed980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556a0357f4c0 0x556a0357f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a98ed980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x556a0343c0d0 con 0x556a0357fde0 2024-04-27T03:19:58.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a98ed980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x556a03431430 con 0x556a0357f4c0 2024-04-27T03:19:58.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a7408640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556a0357f4c0 0x556a0357f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a7408640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x556a0357f4c0 0x556a0357f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51850/0 (socket says 172.21.15.110:51850) 2024-04-27T03:19:58.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a7408640 1 -- 172.21.15.110:0/4104691532 learned_addr learned my addr 172.21.15.110:0/4104691532 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a7408640 1 -- 172.21.15.110:0/4104691532 >> v1:172.21.15.77:6789/0 conn(0x556a0357fde0 legacy=0x556a035807a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.590+0000 7fc5a7408640 1 -- 172.21.15.110:0/4104691532 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a0342a510 con 0x556a0357f4c0 2024-04-27T03:19:58.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a7408640 1 --2- 172.21.15.110:0/4104691532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0357f8a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc59800a5c0 tx=0x7fc59801b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3168d2786c013650 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a6406640 1 -- 172.21.15.110:0/4104691532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5980a6070 con 0x556a0357f4c0 2024-04-27T03:19:58.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a6406640 1 -- 172.21.15.110:0/4104691532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5980af070 con 0x556a0357f4c0 2024-04-27T03:19:58.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a6406640 1 -- 172.21.15.110:0/4104691532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5980aa5c0 con 0x556a0357f4c0 2024-04-27T03:19:58.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a98ed980 1 -- 172.21.15.110:0/4104691532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 msgr2=0x556a0357f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/4104691532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0357f8a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc59800a5c0 tx=0x7fc59801b340 comp rx=0 tx=0).stop 2024-04-27T03:19:58.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a98ed980 1 -- 172.21.15.110:0/4104691532 shutdown_connections 2024-04-27T03:19:58.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/4104691532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0357f8a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a98ed980 1 -- 172.21.15.110:0/4104691532 >> 172.21.15.110:0/4104691532 conn(0x556a034f2ba0 msgr2=0x556a034f2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a98ed980 1 -- 172.21.15.110:0/4104691532 shutdown_connections 2024-04-27T03:19:58.606 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.594+0000 7fc5a98ed980 1 -- 172.21.15.110:0/4104691532 wait complete. 2024-04-27T03:19:58.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 Processor -- start 2024-04-27T03:19:58.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- start start 2024-04-27T03:19:58.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0366ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556a034802d0 con 0x556a0357f4c0 2024-04-27T03:19:58.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0366ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0366ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51854/0 (socket says 172.21.15.110:51854) 2024-04-27T03:19:58.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 -- 172.21.15.110:0/984405194 learned_addr learned my addr 172.21.15.110:0/984405194 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 -- 172.21.15.110:0/984405194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a03406ae0 con 0x556a0357f4c0 2024-04-27T03:19:58.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 --2- 172.21.15.110:0/984405194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0366ad80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc5980199f0 tx=0x7fc598003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ca794d169d2ff9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc49560a640 1 -- 172.21.15.110:0/984405194 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5980aaa00 con 0x556a0357f4c0 2024-04-27T03:19:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc49560a640 1 -- 172.21.15.110:0/984405194 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5980aaba0 con 0x556a0357f4c0 2024-04-27T03:19:58.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc49560a640 1 -- 172.21.15.110:0/984405194 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5980bd3f0 con 0x556a0357f4c0 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- 172.21.15.110:0/984405194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 msgr2=0x556a0366ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/984405194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0366ad80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc5980199f0 tx=0x7fc598003030 comp rx=0 tx=0).stop 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- 172.21.15.110:0/984405194 shutdown_connections 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/984405194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a0366ad80 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- 172.21.15.110:0/984405194 >> 172.21.15.110:0/984405194 conn(0x556a034f2ba0 msgr2=0x556a03666910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- 172.21.15.110:0/984405194 shutdown_connections 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- 172.21.15.110:0/984405194 wait complete. 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 Processor -- start 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- start start 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a03671d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a98ed980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556a03589510 con 0x556a0357f4c0 2024-04-27T03:19:58.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a03671d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a03671d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51860/0 (socket says 172.21.15.110:51860) 2024-04-27T03:19:58.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.610+0000 7fc5a7408640 1 -- 172.21.15.110:0/1806314841 learned_addr learned my addr 172.21.15.110:0/1806314841 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a7408640 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a03672240 con 0x556a0357f4c0 2024-04-27T03:19:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a7408640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a03671d00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc59801b170 tx=0x7fc5980180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc493e07640 1 -- 172.21.15.110:0/1806314841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5980bd7c0 con 0x556a0357f4c0 2024-04-27T03:19:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc493e07640 1 -- 172.21.15.110:0/1806314841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5980bd960 con 0x556a0357f4c0 2024-04-27T03:19:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc493e07640 1 -- 172.21.15.110:0/1806314841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5980bdc40 con 0x556a0357f4c0 2024-04-27T03:19:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a036724d0 con 0x556a0357f4c0 2024-04-27T03:19:58.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a036750b0 con 0x556a0357f4c0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc493e07640 1 -- 172.21.15.110:0/1806314841 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc5980ce020 con 0x556a0357f4c0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc493e07640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc48c034650 0x7fc48c036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc493e07640 1 -- 172.21.15.110:0/1806314841 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc5980cc020 con 0x556a0357f4c0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a6c07640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc48c034650 0x7fc48c036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a0357fde0 0x556a03678b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a03679190 con 0x556a0357fde0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a0357fde0 0x556a03678b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a6c07640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc48c034650 0x7fc48c036b10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc59c001e20 tx=0x7fc59c0013a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a0357fde0 0x556a03678b90 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a7c09640 1 -- 172.21.15.110:0/1806314841 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc5a000a4d0 con 0x556a0357fde0 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a03679f80 0x556a0367c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a0367c9b0 con 0x556a03679f80 2024-04-27T03:19:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.614+0000 7fc5a7408640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a03679f80 0x556a0367c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7408640 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a03679f80 0x556a0367c370 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7408640 1 -- 172.21.15.110:0/1806314841 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a03679f80 2024-04-27T03:19:58.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a0367e260 con 0x556a0357fde0 2024-04-27T03:19:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7c09640 1 -- 172.21.15.110:0/1806314841 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fc5a000a4d0 con 0x556a0357fde0 2024-04-27T03:19:58.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a0367d490 con 0x556a03679f80 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7408640 1 -- 172.21.15.110:0/1806314841 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a03679f80 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a0367d880 con 0x556a0357fde0 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7c09640 1 -- 172.21.15.110:0/1806314841 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fc5a000a4d0 con 0x556a0357fde0 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a98ed980 1 Processor -- start 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a98ed980 1 -- start start 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a98ed980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0368cbd0 0x556a0368cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a98ed980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556a03484310 con 0x556a0368cbd0 2024-04-27T03:19:58.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7c09640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0368cbd0 0x556a0368cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7c09640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0368cbd0 0x556a0368cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51876/0 (socket says 172.21.15.110:51876) 2024-04-27T03:19:58.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3953518918 learned_addr learned my addr 172.21.15.110:0/3953518918 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.618+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3953518918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a0368d530 con 0x556a0368cbd0 2024-04-27T03:19:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0368cbd0 0x556a0368cff0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000dfc0 tx=0x7fc5a00178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc491602640 1 -- 172.21.15.110:0/3953518918 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a00157d0 con 0x556a0368cbd0 2024-04-27T03:19:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc491602640 1 -- 172.21.15.110:0/3953518918 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a0015970 con 0x556a0368cbd0 2024-04-27T03:19:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a0368dfd0 con 0x556a0368cbd0 2024-04-27T03:19:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc491602640 1 -- 172.21.15.110:0/3953518918 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a0015c70 con 0x556a0368cbd0 2024-04-27T03:19:58.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a0368e4a0 con 0x556a0368cbd0 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc491602640 1 -- 172.21.15.110:0/3953518918 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc5a0029440 con 0x556a0368cbd0 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc491602640 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc480034640 0x7fc480036b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc491602640 1 -- 172.21.15.110:0/3953518918 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc5a000e040 con 0x556a0368cbd0 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7408640 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc480034640 0x7fc480036b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a98ed980 1 Processor -- start 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a98ed980 1 -- start start 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7408640 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc480034640 0x7fc480036b00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x556a03679a20 tx=0x7fc5980e3820 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a98ed980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a0369a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a98ed980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556a0366b410 con 0x556a03699d80 2024-04-27T03:19:58.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7c09640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a0369a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7c09640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a0369a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51892/0 (socket says 172.21.15.110:51892) 2024-04-27T03:19:58.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3965191992 learned_addr learned my addr 172.21.15.110:0/3965191992 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3965191992 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a0369a770 con 0x556a03699d80 2024-04-27T03:19:58.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/3965191992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a0369a1a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0015fa0 tx=0x7fc5a0015fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6eb45f7c4d3f11a server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc48affd640 1 -- 172.21.15.110:0/3965191992 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a00113a0 con 0x556a03699d80 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc48affd640 1 -- 172.21.15.110:0/3965191992 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a0011540 con 0x556a03699d80 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.622+0000 7fc48affd640 1 -- 172.21.15.110:0/3965191992 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a00284d0 con 0x556a03699d80 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3965191992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 msgr2=0x556a0369a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3965191992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a0369a1a0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0015fa0 tx=0x7fc5a0015fd0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3965191992 shutdown_connections 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3965191992 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a0369a1a0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3965191992 >> 172.21.15.110:0/3965191992 conn(0x556a03696110 msgr2=0x556a03694170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3965191992 shutdown_connections 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3965191992 wait complete. 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 Processor -- start 2024-04-27T03:19:58.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- start start 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a036a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x556a0369aee0 con 0x556a03699d80 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a7c09640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a036a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a7c09640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a036a12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51896/0 (socket says 172.21.15.110:51896) 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 learned_addr learned my addr 172.21.15.110:0/3415874383 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a036a17f0 con 0x556a03699d80 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a036a12b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc5a001eb40 tx=0x7fc5a004a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc4897fa640 1 -- 172.21.15.110:0/3415874383 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a00113a0 con 0x556a03699d80 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a036a1a80 con 0x556a03699d80 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a036a4760 con 0x556a03699d80 2024-04-27T03:19:58.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc4897fa640 1 -- 172.21.15.110:0/3415874383 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a004c410 con 0x556a03699d80 2024-04-27T03:19:58.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc4897fa640 1 -- 172.21.15.110:0/3415874383 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5a0013dd0 con 0x556a03699d80 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc4897fa640 1 -- 172.21.15.110:0/3415874383 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc5a004ac40 con 0x556a03699d80 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc4897fa640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4700345b0 0x7fc470036a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc4897fa640 1 -- 172.21.15.110:0/3415874383 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc5a0002b90 con 0x556a03699d80 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a7408640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4700345b0 0x7fc470036a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a036ac9b0 0x556a036aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.626+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036af5c0 con 0x556a036ac9b0 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a6c07640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a036ac9b0 0x556a036aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7408640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4700345b0 0x7fc470036a70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x556a036761f0 tx=0x7fc5980ed000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a6c07640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a036ac9b0 0x556a036aee70 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc59c00d040 con 0x556a036ac9b0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 realm 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036abe80 con 0x556a036ac9b0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc59c00d040 con 0x556a036ac9b0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036b00c0 con 0x556a036ac9b0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc59c00d040 con 0x556a036ac9b0 2024-04-27T03:19:58.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a036b0ea0 0x556a036b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036b3960 con 0x556a036b0ea0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a036b0ea0 0x556a036b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a036b0ea0 0x556a036b3290 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036b41b0 con 0x556a036ac9b0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fc59c00d040 con 0x556a036ac9b0 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 20 searching for the correct realm 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556a036b5f20 0x556a036b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b5f20 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7408640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556a036b5f20 0x556a036b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7408640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556a036b5f20 0x556a036b8310 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.630+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7c09640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5a0014030 0x7fc5a00208a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x7fc5a0014030 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5a0014030 0x7fc5a00208a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5a0014030 0x7fc5a00208a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c00d040 con 0x7fc5a0014030 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c00d040 con 0x556a036ac9b0 2024-04-27T03:19:58.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b5f20 2024-04-27T03:19:58.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.657 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x7fc5a0014030 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x7fc5a0014030 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.634+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x7fc5a0014030 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fc59c02a040 con 0x7fc5a0014030 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b5f20 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b5f20 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x7fc5a0014030 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fc59c02a040 con 0x7fc5a0014030 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b5f20 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.659 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x7fc5a0014030 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x7fc5a0014030 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.638+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036ac9b0 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c02a040 con 0x556a036ac9b0 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x7fc5a0014030 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b5f20 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x7fc5a0014030 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.660 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b5f20 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8920 con 0x556a036b0ea0 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:58.661 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036b4b30 con 0x556a036ac9b0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.642+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036b53d0 con 0x556a036b0ea0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036b4770 con 0x556a036ac9b0 2024-04-27T03:19:58.662 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 zone default found 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 4 Realm: () 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 zonegroup default 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036b8e10 con 0x7fc5a0014030 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.663 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036bac30 con 0x556a036b5f20 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036bb050 con 0x556a036b5f20 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 started sync module instance, tier type = 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:58.664 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036ba0d0 con 0x556a036b0ea0 2024-04-27T03:19:58.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 93913812280512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bb940 con 0x556a036b0ea0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.646+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bc190 con 0x556a036ac9b0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 93913812285760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bc2b0 con 0x556a036ac9b0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bcdd0 con 0x556a036b0ea0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 93913812280512] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 93913812285760] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 93913812286816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bcd90 con 0x556a036b0ea0 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.650+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bd670 con 0x7fc5a0014030 2024-04-27T03:19:58.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 93913812292032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bcd90 con 0x7fc5a0014030 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036be100 con 0x556a036b0ea0 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 93913812292032] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 93913812286816] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 93913812289696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036be100 con 0x556a036b0ea0 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036be520 con 0x556a036b5f20 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 93913812293088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bed70 con 0x556a036b5f20 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bf730 con 0x556a036b5f20 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.654+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 93913812289696] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 93913812293088] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 93913812297920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bf730 con 0x556a036b5f20 2024-04-27T03:19:58.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bed40 con 0x7fc5a0014030 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 93913812297920] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'261 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 93913812302672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036bf730 con 0x7fc5a0014030 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=0 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=1 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=2 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=3 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=4 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=5 2024-04-27T03:19:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=6 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 93913812302672] v31'262 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 20 add_watcher() i=7 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036c30e0 con 0x556a036ac9b0 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.658+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 93913812300880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036c0600 con 0x556a036ac9b0 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 93913812300880] v31'265 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc45e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 5 note: GC not initialized 2024-04-27T03:19:58.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc45d7e2640 20 reqs_thread_entry: start 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 20 Filter name: none 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc4567fc640 20 reqs_thread_entry: start 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036e6900 con 0x556a036ac9b0 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~490 out=490b,getxattrs out=75b] v0'0 uv6 ondisk = 0) ==== 231+0+613 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 20 rados_obj.operate() r=0 bl.length=490 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:58.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~490 in=490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036ed270 con 0x556a036ac9b0 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~490] v31'7 uv7 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 93913812487392 in=678b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x556a036e8940 con 0x556a036b0ea0 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 93913812289696 notify 133143986234 ret 0) ==== 708+0+0 (crc 0 0 0) 0x556a036a4760 con 0x556a036b0ea0 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc48a7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986234 cookie 93913812289696 notifier 4870 bl.length()=666 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc48a7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc48a7fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc48a7fc640 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fc46c003eb0 con 0x556a036b0ea0 2024-04-27T03:19:58.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.662+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 93913812487392 out=8b] v0'0 uv261 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.666+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.666+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 93913812487392 notify 133143986234 ret 0) ==== 42+0+48 (crc 0 0 0) 0x556a036a1a80 con 0x556a036b0ea0 2024-04-27T03:19:58.678 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"user-info-without-keys","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:58.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.666+0000 7fc45e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.666+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 93913812300880] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x556a036ac9b0 2024-04-27T03:19:58.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.666+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 93913812300880] v31'266 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 93913812280512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x556a036b0ea0 2024-04-27T03:19:58.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 93913812280512] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a98ed980 20 remove_watcher() i=0 2024-04-27T03:19:58.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a98ed980 2 removed watcher, disabling cache 2024-04-27T03:19:58.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 93913812285760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x556a036ac9b0 2024-04-27T03:19:58.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 93913812285760] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x556a036ac9b0 2024-04-27T03:19:58.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a98ed980 20 remove_watcher() i=1 2024-04-27T03:19:58.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.670+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 93913812286816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x556a036b0ea0 2024-04-27T03:19:58.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.674+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 93913812286816] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.674+0000 7fc5a98ed980 20 remove_watcher() i=2 2024-04-27T03:19:58.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.674+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 93913812292032] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x7fc5a0014030 2024-04-27T03:19:58.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.674+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 93913812292032] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.674+0000 7fc5a98ed980 20 remove_watcher() i=3 2024-04-27T03:19:58.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.674+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 93913812289696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x556a036b0ea0 2024-04-27T03:19:58.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a7c09640 1 -- 172.21.15.110:0/3415874383 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 93913812289696] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5a0026020 con 0x556a036b0ea0 2024-04-27T03:19:58.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a98ed980 20 remove_watcher() i=4 2024-04-27T03:19:58.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 93913812293088] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x556a036b5f20 2024-04-27T03:19:58.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 93913812293088] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a98ed980 20 remove_watcher() i=5 2024-04-27T03:19:58.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 93913812297920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x556a036b5f20 2024-04-27T03:19:58.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a7408640 1 -- 172.21.15.110:0/3415874383 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 93913812297920] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5980c6cd0 con 0x556a036b5f20 2024-04-27T03:19:58.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a98ed980 20 remove_watcher() i=6 2024-04-27T03:19:58.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 93913812302672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x556a036e4600 con 0x7fc5a0014030 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 93913812302672] v31'263 uv261 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc59c03d040 con 0x7fc5a0014030 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.678+0000 7fc5a98ed980 20 remove_watcher() i=7 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a036b0ea0 msgr2=0x556a036b3290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a036b0ea0 0x556a036b3290 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a036ac9b0 msgr2=0x556a036aee70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a036ac9b0 0x556a036aee70 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5a0014030 msgr2=0x7fc5a00208a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5a0014030 0x7fc5a00208a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556a036b5f20 msgr2=0x556a036b8310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x556a036b5f20 0x556a036b8310 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4700345b0 msgr2=0x7fc470036a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4700345b0 0x7fc470036a70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x556a036761f0 tx=0x7fc5980ed000 comp rx=0 tx=0).stop 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a6c07640 1 -- 172.21.15.110:0/3415874383 reap_dead start 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 msgr2=0x556a036a12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a036a12b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc5a001eb40 tx=0x7fc5a004a880 comp rx=0 tx=0).stop 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 shutdown_connections 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3415874383 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a03699d80 0x556a036a12b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 >> 172.21.15.110:0/3415874383 conn(0x556a03696110 msgr2=0x556a0369b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 shutdown_connections 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3415874383 wait complete. 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc480034640 msgr2=0x7fc480036b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc480034640 0x7fc480036b00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x556a03679a20 tx=0x7fc5980e3820 comp rx=0 tx=0).stop 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0368cbd0 msgr2=0x556a0368cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0368cbd0 0x556a0368cff0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000dfc0 tx=0x7fc5a00178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 shutdown_connections 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc480034640 0x7fc480036b00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/3953518918 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0368cbd0 0x556a0368cff0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 >> 172.21.15.110:0/3953518918 conn(0x556a036819e0 msgr2=0x556a036824f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 shutdown_connections 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/3953518918 wait complete. 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a03679f80 msgr2=0x556a0367c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a03679f80 0x556a0367c370 crc :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a0357fde0 msgr2=0x556a03678b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a0357fde0 0x556a03678b90 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc48c034650 msgr2=0x7fc48c036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc48c034650 0x7fc48c036b10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc59c001e20 tx=0x7fc59c0013a0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 msgr2=0x556a03671d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a03671d00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc59801b170 tx=0x7fc5980180e0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 shutdown_connections 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x556a03679f80 0x556a0367c370 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x556a0357fde0 0x556a03678b90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc48c034650 0x7fc48c036b10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 --2- 172.21.15.110:0/1806314841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x556a0357f4c0 0x556a03671d00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 >> 172.21.15.110:0/1806314841 conn(0x556a034f2ba0 msgr2=0x556a03665420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 shutdown_connections 2024-04-27T03:19:58.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.682+0000 7fc5a98ed980 1 -- 172.21.15.110:0/1806314841 wait complete. 2024-04-27T03:19:58.701 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.702 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.702 ERROR:root:0 2024-04-27T03:19:58.702 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:58.706 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:58.707 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:58.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:58.707 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid ada --caps 'users=read, write' 2024-04-27T03:19:58.773 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:58.774 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:58.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.762+0000 7fc5e5c31980 1 Processor -- start 2024-04-27T03:19:58.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.762+0000 7fc5e5c31980 1 -- start start 2024-04-27T03:19:58.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.762+0000 7fc5e5c31980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a9df664930 0x55a9df662ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.762+0000 7fc5e5c31980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a9df5ad0d0 con 0x55a9df663420 2024-04-27T03:19:58.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.762+0000 7fc5e5c31980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a9df5a2430 con 0x55a9df664930 2024-04-27T03:19:58.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e374c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a9df664930 0x55a9df662ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e374c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a9df664930 0x55a9df662ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51906/0 (socket says 172.21.15.110:51906) 2024-04-27T03:19:58.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e374c640 1 -- 172.21.15.110:0/803149334 learned_addr learned my addr 172.21.15.110:0/803149334 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e374c640 1 -- 172.21.15.110:0/803149334 >> v1:172.21.15.77:6789/0 conn(0x55a9df663420 legacy=0x55a9df6f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e374c640 1 -- 172.21.15.110:0/803149334 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9df59b510 con 0x55a9df664930 2024-04-27T03:19:58.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e374c640 1 --2- 172.21.15.110:0/803149334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df664930 0x55a9df662ee0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400a5c0 tx=0x7fc5d401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f96e01f564d49a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e274a640 1 -- 172.21.15.110:0/803149334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5d40a6070 con 0x55a9df664930 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e274a640 1 -- 172.21.15.110:0/803149334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d40af070 con 0x55a9df664930 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e274a640 1 -- 172.21.15.110:0/803149334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5d40aa5c0 con 0x55a9df664930 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e5c31980 1 -- 172.21.15.110:0/803149334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df664930 msgr2=0x55a9df662ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.766+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/803149334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df664930 0x55a9df662ee0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400a5c0 tx=0x7fc5d401b340 comp rx=0 tx=0).stop 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.770+0000 7fc5e5c31980 1 -- 172.21.15.110:0/803149334 shutdown_connections 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.770+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/803149334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df664930 0x55a9df662ee0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.770+0000 7fc5e5c31980 1 -- 172.21.15.110:0/803149334 >> 172.21.15.110:0/803149334 conn(0x55a9df6e9b70 msgr2=0x55a9df6ebf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.770+0000 7fc5e5c31980 1 -- 172.21.15.110:0/803149334 shutdown_connections 2024-04-27T03:19:58.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.770+0000 7fc5e5c31980 1 -- 172.21.15.110:0/803149334 wait complete. 2024-04-27T03:19:58.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e5c31980 1 Processor -- start 2024-04-27T03:19:58.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e5c31980 1 -- start start 2024-04-27T03:19:58.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e5c31980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7dbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e5c31980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9df5f12d0 con 0x55a9df663420 2024-04-27T03:19:58.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e374c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7dbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e374c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7dbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51912/0 (socket says 172.21.15.110:51912) 2024-04-27T03:19:58.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e374c640 1 -- 172.21.15.110:0/3718890064 learned_addr learned my addr 172.21.15.110:0/3718890064 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e374c640 1 -- 172.21.15.110:0/3718890064 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9df577ae0 con 0x55a9df663420 2024-04-27T03:19:58.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc5e374c640 1 --2- 172.21.15.110:0/3718890064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7dbd10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40199f0 tx=0x7fc5d4003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0650b12b1e26cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc4d160a640 1 -- 172.21.15.110:0/3718890064 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5d40aaa00 con 0x55a9df663420 2024-04-27T03:19:58.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc4d160a640 1 -- 172.21.15.110:0/3718890064 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d40aaba0 con 0x55a9df663420 2024-04-27T03:19:58.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.782+0000 7fc4d160a640 1 -- 172.21.15.110:0/3718890064 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5d40bd3f0 con 0x55a9df663420 2024-04-27T03:19:58.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3718890064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 msgr2=0x55a9df7dbd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3718890064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7dbd10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40199f0 tx=0x7fc5d4003030 comp rx=0 tx=0).stop 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3718890064 shutdown_connections 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3718890064 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7dbd10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3718890064 >> 172.21.15.110:0/3718890064 conn(0x55a9df6e9b70 msgr2=0x55a9df7d78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3718890064 shutdown_connections 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3718890064 wait complete. 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 Processor -- start 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- start start 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9df6fa4b0 con 0x55a9df663420 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e374c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e374c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7e2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51926/0 (socket says 172.21.15.110:51926) 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e374c640 1 -- 172.21.15.110:0/3184709245 learned_addr learned my addr 172.21.15.110:0/3184709245 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e374c640 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9df7e31d0 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e374c640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7e2c90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc5d401b170 tx=0x7fc5d40180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc4cfe07640 1 -- 172.21.15.110:0/3184709245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5d40bd7c0 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a9df7e3460 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc4cfe07640 1 -- 172.21.15.110:0/3184709245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d40bd960 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9df7e6040 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc4cfe07640 1 -- 172.21.15.110:0/3184709245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5d40bdc40 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc4cfe07640 1 -- 172.21.15.110:0/3184709245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc5d40ce030 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc4cfe07640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4c8034650 0x7fc4c8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc4cfe07640 1 -- 172.21.15.110:0/3184709245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc5d40cc020 con 0x55a9df663420 2024-04-27T03:19:58.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.786+0000 7fc5e2f4b640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4c8034650 0x7fc4c8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df664930 0x55a9df7e9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df7ea140 con 0x55a9df664930 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df664930 0x55a9df7e9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e2f4b640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4c8034650 0x7fc4c8036b10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8000f70 tx=0x7fc5d8001420 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df664930 0x55a9df7e9b20 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/3184709245 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc5dc00a4d0 con 0x55a9df664930 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df7eaf10 0x55a9df7ed300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df7ed960 con 0x55a9df7eaf10 2024-04-27T03:19:58.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e374c640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df7eaf10 0x55a9df7ed300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e374c640 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df7eaf10 0x55a9df7ed300 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e374c640 1 -- 172.21.15.110:0/3184709245 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df7eaf10 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df7ef210 con 0x55a9df664930 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/3184709245 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fc5dc00a4d0 con 0x55a9df664930 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df7ee440 con 0x55a9df7eaf10 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e374c640 1 -- 172.21.15.110:0/3184709245 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df7eaf10 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df7ee830 con 0x55a9df664930 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.790+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/3184709245 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fc5dc00a4d0 con 0x55a9df664930 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 Processor -- start 2024-04-27T03:19:58.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 -- start start 2024-04-27T03:19:58.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df7fdb30 0x55a9df7fdf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9df5f5310 con 0x55a9df7fdb30 2024-04-27T03:19:58.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e3f4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df7fdb30 0x55a9df7fdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e3f4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df7fdb30 0x55a9df7fdf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51936/0 (socket says 172.21.15.110:51936) 2024-04-27T03:19:58.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/307744551 learned_addr learned my addr 172.21.15.110:0/307744551 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/307744551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9df7fe490 con 0x55a9df7fdb30 2024-04-27T03:19:58.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df7fdb30 0x55a9df7fdf50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc017680 tx=0x7fc5dc017a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc4cd602640 1 -- 172.21.15.110:0/307744551 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc0157d0 con 0x55a9df7fdb30 2024-04-27T03:19:58.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc4cd602640 1 -- 172.21.15.110:0/307744551 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc015970 con 0x55a9df7fdb30 2024-04-27T03:19:58.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a9df7fef30 con 0x55a9df7fdb30 2024-04-27T03:19:58.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc4cd602640 1 -- 172.21.15.110:0/307744551 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc015c50 con 0x55a9df7fdb30 2024-04-27T03:19:58.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9df7ff400 con 0x55a9df7fdb30 2024-04-27T03:19:58.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc4cd602640 1 -- 172.21.15.110:0/307744551 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc5dc028440 con 0x55a9df7fdb30 2024-04-27T03:19:58.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc4cd602640 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4bc034650 0x7fc4bc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.807 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc4cd602640 1 -- 172.21.15.110:0/307744551 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc5dc00e040 con 0x55a9df7fdb30 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e374c640 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4bc034650 0x7fc4bc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 Processor -- start 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.794+0000 7fc5e5c31980 1 -- start start 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e374c640 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4bc034650 0x7fc4bc036b10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55a9df7edf60 tx=0x7fc5d40e36c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df80b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9df7dc3a0 con 0x55a9df80ace0 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df80b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df80b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51942/0 (socket says 172.21.15.110:51942) 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/216702986 learned_addr learned my addr 172.21.15.110:0/216702986 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.808 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/216702986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9df80b6d0 con 0x55a9df80ace0 2024-04-27T03:19:58.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/216702986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df80b100 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc04a480 tx=0x7fc5dc0274d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea1858702071c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/216702986 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc03d070 con 0x55a9df80ace0 2024-04-27T03:19:58.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/216702986 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc027bc0 con 0x55a9df80ace0 2024-04-27T03:19:58.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/216702986 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc011a40 con 0x55a9df80ace0 2024-04-27T03:19:58.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- 172.21.15.110:0/216702986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 msgr2=0x55a9df80b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/216702986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df80b100 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc04a480 tx=0x7fc5dc0274d0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- 172.21.15.110:0/216702986 shutdown_connections 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/216702986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df80b100 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- 172.21.15.110:0/216702986 >> 172.21.15.110:0/216702986 conn(0x55a9df807020 msgr2=0x55a9df805080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- 172.21.15.110:0/216702986 shutdown_connections 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- 172.21.15.110:0/216702986 wait complete. 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 Processor -- start 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- start start 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df8121a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e5c31980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a9df80bdf0 con 0x55a9df80ace0 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df8121a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.812 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df8121a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51952/0 (socket says 172.21.15.110:51952) 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 learned_addr learned my addr 172.21.15.110:0/98666417 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9df8126e0 con 0x55a9df80ace0 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.798+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df8121a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc027510 tx=0x7fc5dc04a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc4c57fa640 1 -- 172.21.15.110:0/98666417 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc039070 con 0x55a9df80ace0 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a9df812970 con 0x55a9df80ace0 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc4c57fa640 1 -- 172.21.15.110:0/98666417 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc01ee50 con 0x55a9df80ace0 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc4c57fa640 1 -- 172.21.15.110:0/98666417 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5dc02e040 con 0x55a9df80ace0 2024-04-27T03:19:58.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9df815650 con 0x55a9df80ace0 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc4c57fa640 1 -- 172.21.15.110:0/98666417 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc5dc028730 con 0x55a9df80ace0 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc4c57fa640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4ac034650 0x7fc4ac036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc4c57fa640 1 -- 172.21.15.110:0/98666417 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc5dc035df0 con 0x55a9df80ace0 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e374c640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4ac034650 0x7fc4ac036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df81d840 0x55a9df81fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df820430 con 0x55a9df81d840 2024-04-27T03:19:58.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e2f4b640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df81d840 0x55a9df81fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e374c640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4ac034650 0x7fc4ac036b10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40b37d0 tx=0x7fc5d40e9000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e2f4b640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df81d840 0x55a9df81fce0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc5d800d040 con 0x55a9df81d840 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 realm 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df81cd10 con 0x55a9df81d840 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc5d800d040 con 0x55a9df81d840 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df820f30 con 0x55a9df81d840 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc5d800d040 con 0x55a9df81d840 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df821d10 0x55a9df824100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8247d0 con 0x55a9df821d10 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.802+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df821d10 0x55a9df824100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df821d10 0x55a9df824100 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e5c31980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df825020 con 0x55a9df81d840 2024-04-27T03:19:58.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fc5d800d040 con 0x55a9df81d840 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e5c31980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e5c31980 20 searching for the correct realm 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9df826d90 0x55a9df829180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df826d90 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e374c640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9df826d90 0x55a9df829180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e374c640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9df826d90 0x55a9df829180 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e3f4d640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5dc058040 0x7fc5dc042460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x7fc5dc058040 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e2f4b640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5dc058040 0x7fc5dc042460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e2f4b640 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5dc058040 0x7fc5dc042460 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d800d040 con 0x7fc5dc058040 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d800d040 con 0x55a9df81d840 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.806+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df826d90 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x7fc5dc058040 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x7fc5dc058040 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x7fc5dc058040 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fc5d802a040 con 0x7fc5dc058040 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.810+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df826d90 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df826d90 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x7fc5dc058040 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fc5d802a040 con 0x7fc5dc058040 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df826d90 2024-04-27T03:19:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x7fc5dc058040 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x7fc5dc058040 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df81d840 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d802a040 con 0x55a9df81d840 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x7fc5dc058040 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df826d90 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.814+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x7fc5dc058040 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df826d90 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829800 con 0x55a9df821d10 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:58.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8259a0 con 0x55a9df81d840 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.818+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8255e0 con 0x55a9df821d10 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df826300 con 0x55a9df81d840 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 zone default found 2024-04-27T03:19:58.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 4 Realm: () 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 zonegroup default 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df829c80 con 0x7fc5dc058040 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82baa0 con 0x55a9df826d90 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82be70 con 0x55a9df826d90 2024-04-27T03:19:58.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 started sync module instance, tier type = 2024-04-27T03:19:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:58.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.822+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82af40 con 0x55a9df821d10 2024-04-27T03:19:58.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94188087718976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82af40 con 0x55a9df821d10 2024-04-27T03:19:58.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82c9f0 con 0x55a9df81d840 2024-04-27T03:19:58.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94188087723200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82c9b0 con 0x55a9df81d840 2024-04-27T03:19:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82d920 con 0x55a9df821d10 2024-04-27T03:19:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94188087718976] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.826+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94188087723200] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94188087725536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82d920 con 0x55a9df821d10 2024-04-27T03:19:58.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82dd50 con 0x7fc5dc058040 2024-04-27T03:19:58.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94188087716496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82dd10 con 0x7fc5dc058040 2024-04-27T03:19:58.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82ec30 con 0x55a9df821d10 2024-04-27T03:19:58.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94188087725536] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94188087716496] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94188087730496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82ec30 con 0x55a9df821d10 2024-04-27T03:19:58.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.830+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82f020 con 0x55a9df826d90 2024-04-27T03:19:58.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94188087727152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82f870 con 0x55a9df826d90 2024-04-27T03:19:58.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df830270 con 0x55a9df826d90 2024-04-27T03:19:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94188087730496] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94188087727152] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94188087732064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df830660 con 0x55a9df826d90 2024-04-27T03:19:58.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df830eb0 con 0x7fc5dc058040 2024-04-27T03:19:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'264 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94188087740480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df831300 con 0x7fc5dc058040 2024-04-27T03:19:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 20 add_watcher() i=0 2024-04-27T03:19:58.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 20 add_watcher() i=1 2024-04-27T03:19:58.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 20 add_watcher() i=2 2024-04-27T03:19:58.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 20 add_watcher() i=3 2024-04-27T03:19:58.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 20 add_watcher() i=4 2024-04-27T03:19:58.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 20 add_watcher() i=5 2024-04-27T03:19:58.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94188087732064] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.834+0000 7fc5e5c31980 20 add_watcher() i=6 2024-04-27T03:19:58.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94188087740480] v31'265 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 20 add_watcher() i=7 2024-04-27T03:19:58.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:58.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df82fd80 con 0x55a9df81d840 2024-04-27T03:19:58.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94188087737728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df831840 con 0x55a9df81d840 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94188087737728] v31'267 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc49a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 5 note: GC not initialized 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc4997e2640 20 reqs_thread_entry: start 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:58.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 20 Filter name: none 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc4927fc640 20 reqs_thread_entry: start 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 20 rados->read ofs=0 len=0 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.838+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df857ba0 con 0x55a9df81d840 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~490 out=490b,getxattrs out=75b] v0'0 uv7 ondisk = 0) ==== 231+0+613 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 20 rados_obj.operate() r=0 bl.length=490 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:58.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~503 in=503b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df85e500 con 0x55a9df81d840 2024-04-27T03:19:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~503] v31'8 uv8 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:58.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94188087926640 in=691b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9df859bb0 con 0x55a9df821d10 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94188087730496 notify 133143986235 ret 0) ==== 721+0+0 (crc 0 0 0) 0x55a9df815650 con 0x55a9df821d10 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc4c6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986235 cookie 94188087730496 notifier 4877 bl.length()=679 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94188087926640 out=8b] v0'0 uv264 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc4c6ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc4c6ffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:58.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc4c6ffd640 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fc4a8003eb0 con 0x55a9df821d10 2024-04-27T03:19:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.842+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94188087926640 notify 133143986235 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7fc5dc066020 con 0x55a9df821d10 2024-04-27T03:19:58.855 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"user-info-without-keys","perm":"*"},{"type":"users","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.846+0000 7fc49a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:58.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.846+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94188087737728] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x55a9df81d840 2024-04-27T03:19:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.846+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94188087737728] v31'268 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.846+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94188087718976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x55a9df821d10 2024-04-27T03:19:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94188087718976] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e5c31980 20 remove_watcher() i=0 2024-04-27T03:19:58.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e5c31980 2 removed watcher, disabling cache 2024-04-27T03:19:58.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94188087723200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x55a9df81d840 2024-04-27T03:19:58.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94188087723200] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x55a9df81d840 2024-04-27T03:19:58.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e5c31980 20 remove_watcher() i=1 2024-04-27T03:19:58.861 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94188087725536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x55a9df821d10 2024-04-27T03:19:58.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94188087725536] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e5c31980 20 remove_watcher() i=2 2024-04-27T03:19:58.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.850+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94188087716496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x7fc5dc058040 2024-04-27T03:19:58.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.854+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94188087716496] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.854+0000 7fc5e5c31980 20 remove_watcher() i=3 2024-04-27T03:19:58.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.854+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94188087730496] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x55a9df821d10 2024-04-27T03:19:58.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.854+0000 7fc5e3f4d640 1 -- 172.21.15.110:0/98666417 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94188087730496] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5dc031040 con 0x55a9df821d10 2024-04-27T03:19:58.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.854+0000 7fc5e5c31980 20 remove_watcher() i=4 2024-04-27T03:19:58.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.854+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94188087727152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x55a9df826d90 2024-04-27T03:19:58.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94188087727152] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 20 remove_watcher() i=5 2024-04-27T03:19:58.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94188087732064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x55a9df826d90 2024-04-27T03:19:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e374c640 1 -- 172.21.15.110:0/98666417 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94188087732064] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d40c6d20 con 0x55a9df826d90 2024-04-27T03:19:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 20 remove_watcher() i=6 2024-04-27T03:19:58.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94188087740480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9df8558a0 con 0x7fc5dc058040 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94188087740480] v31'266 uv264 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc5d803d040 con 0x7fc5dc058040 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 20 remove_watcher() i=7 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df821d10 msgr2=0x55a9df824100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df821d10 0x55a9df824100 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df81d840 msgr2=0x55a9df81fce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df81d840 0x55a9df81fce0 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5dc058040 msgr2=0x7fc5dc042460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc5dc058040 0x7fc5dc042460 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9df826d90 msgr2=0x55a9df829180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a9df826d90 0x55a9df829180 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4ac034650 msgr2=0x7fc4ac036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4ac034650 0x7fc4ac036b10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40b37d0 tx=0x7fc5d40e9000 comp rx=0 tx=0).stop 2024-04-27T03:19:58.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e2f4b640 1 -- 172.21.15.110:0/98666417 reap_dead start 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 msgr2=0x55a9df8121a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df8121a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc027510 tx=0x7fc5dc04a410 comp rx=0 tx=0).stop 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 shutdown_connections 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/98666417 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df80ace0 0x55a9df8121a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 >> 172.21.15.110:0/98666417 conn(0x55a9df807020 msgr2=0x55a9df804680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 shutdown_connections 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/98666417 wait complete. 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4bc034650 msgr2=0x7fc4bc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4bc034650 0x7fc4bc036b10 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55a9df7edf60 tx=0x7fc5d40e36c0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df7fdb30 msgr2=0x55a9df7fdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df7fdb30 0x55a9df7fdf50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc017680 tx=0x7fc5dc017a80 comp rx=0 tx=0).stop 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 shutdown_connections 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4bc034650 0x7fc4bc036b10 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/307744551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df7fdb30 0x55a9df7fdf50 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 >> 172.21.15.110:0/307744551 conn(0x55a9df7f2940 msgr2=0x55a9df7f3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 shutdown_connections 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.858+0000 7fc5e5c31980 1 -- 172.21.15.110:0/307744551 wait complete. 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df7eaf10 msgr2=0x55a9df7ed300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df7eaf10 0x55a9df7ed300 crc :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df664930 msgr2=0x55a9df7e9b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df664930 0x55a9df7e9b20 crc :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4c8034650 msgr2=0x7fc4c8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4c8034650 0x7fc4c8036b10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8000f70 tx=0x7fc5d8001420 comp rx=0 tx=0).stop 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 msgr2=0x55a9df7e2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7e2c90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc5d401b170 tx=0x7fc5d40180e0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 shutdown_connections 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a9df7eaf10 0x55a9df7ed300 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a9df664930 0x55a9df7e9b20 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc4c8034650 0x7fc4c8036b10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 --2- 172.21.15.110:0/3184709245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a9df663420 0x55a9df7e2c90 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 >> 172.21.15.110:0/3184709245 conn(0x55a9df6e9b70 msgr2=0x55a9df7d63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 shutdown_connections 2024-04-27T03:19:58.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.862+0000 7fc5e5c31980 1 -- 172.21.15.110:0/3184709245 wait complete. 2024-04-27T03:19:58.882 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.882 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:58.882 ERROR:root:0 2024-04-27T03:19:58.882 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:58.887 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:58.887 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'rm', '--uid', 'ada', '--caps', 'users=read, write; user-info-without-keys=*'] 2024-04-27T03:19:58.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'rm', '--uid', 'ada', '--caps', 'users=read, write; user-info-without-keys=*'] 2024-04-27T03:19:58.887 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps rm --uid ada --caps 'users=read, write; user-info-without-keys=*' 2024-04-27T03:19:58.959 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:58.959 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c43599980 1 Processor -- start 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c43599980 1 -- start start 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c43599980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55db9b842a20 0x55db9b842e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c43599980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55db9b7000d0 con 0x55db9b843340 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c43599980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55db9b6f5430 con 0x55db9b842a20 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c410b4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55db9b842a20 0x55db9b842e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c410b4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55db9b842a20 0x55db9b842e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51956/0 (socket says 172.21.15.110:51956) 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c410b4640 1 -- 172.21.15.110:0/3550436007 learned_addr learned my addr 172.21.15.110:0/3550436007 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c3bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55db9b843340 0x55db9b83f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44620/0 (socket says 172.21.15.110:44620) 2024-04-27T03:19:58.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c410b4640 1 -- 172.21.15.110:0/3550436007 >> v1:172.21.15.77:6789/0 conn(0x55db9b843340 legacy=0x55db9b83f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.950+0000 7f0c410b4640 1 -- 172.21.15.110:0/3550436007 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9b6ee510 con 0x55db9b842a20 2024-04-27T03:19:58.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c410b4640 1 --2- 172.21.15.110:0/3550436007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b842e00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400a5c0 tx=0x7f0c3401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d23dd7086b05ca0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c408b3640 1 -- 172.21.15.110:0/3550436007 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c340b0070 con 0x55db9b842a20 2024-04-27T03:19:58.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c408b3640 1 -- 172.21.15.110:0/3550436007 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c340ab520 con 0x55db9b842a20 2024-04-27T03:19:58.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c408b3640 1 -- 172.21.15.110:0/3550436007 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c340ab820 con 0x55db9b842a20 2024-04-27T03:19:58.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c43599980 1 -- 172.21.15.110:0/3550436007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 msgr2=0x55db9b842e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c43599980 1 --2- 172.21.15.110:0/3550436007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b842e00 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400a5c0 tx=0x7f0c3401b340 comp rx=0 tx=0).stop 2024-04-27T03:19:58.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c43599980 1 -- 172.21.15.110:0/3550436007 shutdown_connections 2024-04-27T03:19:58.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c43599980 1 --2- 172.21.15.110:0/3550436007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b842e00 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c43599980 1 -- 172.21.15.110:0/3550436007 >> 172.21.15.110:0/3550436007 conn(0x55db9b83b1d0 msgr2=0x55db9b83d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c43599980 1 -- 172.21.15.110:0/3550436007 shutdown_connections 2024-04-27T03:19:58.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.954+0000 7f0c43599980 1 -- 172.21.15.110:0/3550436007 wait complete. 2024-04-27T03:19:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 Processor -- start 2024-04-27T03:19:58.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- start start 2024-04-27T03:19:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b92cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db9b7442d0 con 0x55db9b842a20 2024-04-27T03:19:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b92cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b92cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51966/0 (socket says 172.21.15.110:51966) 2024-04-27T03:19:58.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 -- 172.21.15.110:0/3434560316 learned_addr learned my addr 172.21.15.110:0/3434560316 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 -- 172.21.15.110:0/3434560316 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9b6caae0 con 0x55db9b842a20 2024-04-27T03:19:58.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 --2- 172.21.15.110:0/3434560316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b92cdb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0c340199f0 tx=0x7f0c34003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb17bd6d1af9188 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0b2e614640 1 -- 172.21.15.110:0/3434560316 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c340b0040 con 0x55db9b842a20 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0b2e614640 1 -- 172.21.15.110:0/3434560316 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c340ab9b0 con 0x55db9b842a20 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0b2e614640 1 -- 172.21.15.110:0/3434560316 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c340abcb0 con 0x55db9b842a20 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- 172.21.15.110:0/3434560316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 msgr2=0x55db9b92cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 --2- 172.21.15.110:0/3434560316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b92cdb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0c340199f0 tx=0x7f0c34003030 comp rx=0 tx=0).stop 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- 172.21.15.110:0/3434560316 shutdown_connections 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 --2- 172.21.15.110:0/3434560316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b92cdb0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- 172.21.15.110:0/3434560316 >> 172.21.15.110:0/3434560316 conn(0x55db9b83b1d0 msgr2=0x55db9b928940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- 172.21.15.110:0/3434560316 shutdown_connections 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- 172.21.15.110:0/3434560316 wait complete. 2024-04-27T03:19:58.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 Processor -- start 2024-04-27T03:19:58.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- start start 2024-04-27T03:19:58.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b933d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c43599980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db9b84b520 con 0x55db9b842a20 2024-04-27T03:19:58.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b933d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b933d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51982/0 (socket says 172.21.15.110:51982) 2024-04-27T03:19:58.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.970+0000 7f0c410b4640 1 -- 172.21.15.110:0/2463236437 learned_addr learned my addr 172.21.15.110:0/2463236437 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c410b4640 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9b934270 con 0x55db9b842a20 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c410b4640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b933d30 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0c3401b380 tx=0x7f0c340180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0b2ce11640 1 -- 172.21.15.110:0/2463236437 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c340c3020 con 0x55db9b842a20 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55db9b934500 con 0x55db9b842a20 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0b2ce11640 1 -- 172.21.15.110:0/2463236437 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c340b6870 con 0x55db9b842a20 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0b2ce11640 1 -- 172.21.15.110:0/2463236437 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c340b0050 con 0x55db9b842a20 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db9b937130 con 0x55db9b842a20 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0b2ce11640 1 -- 172.21.15.110:0/2463236437 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0c340b6be0 con 0x55db9b842a20 2024-04-27T03:19:58.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0b2ce11640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b280346a0 0x7f0b28036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0b2ce11640 1 -- 172.21.15.110:0/2463236437 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f0c340bc070 con 0x55db9b842a20 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c3bfff640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b280346a0 0x7f0b28036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b843340 0x55db9b93ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b93b2c0 con 0x55db9b843340 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c418b5640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b843340 0x55db9b93ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c3bfff640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b280346a0 0x7f0b28036b60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c001e50 tx=0x7f0c2c0013b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c418b5640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b843340 0x55db9b93ad80 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c418b5640 1 -- 172.21.15.110:0/2463236437 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c3c00a4d0 con 0x55db9b843340 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b93bd70 0x55db9b93e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b93e980 con 0x55db9b93bd70 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.974+0000 7f0c410b4640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b93bd70 0x55db9b93e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c410b4640 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b93bd70 0x55db9b93e210 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c410b4640 1 -- 172.21.15.110:0/2463236437 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b93bd70 2024-04-27T03:19:58.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b940230 con 0x55db9b843340 2024-04-27T03:19:58.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c418b5640 1 -- 172.21.15.110:0/2463236437 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f0c3c00a4d0 con 0x55db9b843340 2024-04-27T03:19:58.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b93f460 con 0x55db9b93bd70 2024-04-27T03:19:58.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c410b4640 1 -- 172.21.15.110:0/2463236437 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b93bd70 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b93f850 con 0x55db9b843340 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c418b5640 1 -- 172.21.15.110:0/2463236437 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f0c3c00a4d0 con 0x55db9b843340 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c43599980 1 Processor -- start 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c43599980 1 -- start start 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c43599980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b94eb50 0x55db9b94ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c43599980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db9b748310 con 0x55db9b94eb50 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c418b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b94eb50 0x55db9b94ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c418b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b94eb50 0x55db9b94ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51996/0 (socket says 172.21.15.110:51996) 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c418b5640 1 -- 172.21.15.110:0/666304321 learned_addr learned my addr 172.21.15.110:0/666304321 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c418b5640 1 -- 172.21.15.110:0/666304321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9b94f4b0 con 0x55db9b94eb50 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.978+0000 7f0c418b5640 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b94eb50 0x55db9b94ef70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c017680 tx=0x7f0c3c017a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b25ffb640 1 -- 172.21.15.110:0/666304321 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c3c0157d0 con 0x55db9b94eb50 2024-04-27T03:19:58.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55db9b94ff50 con 0x55db9b94eb50 2024-04-27T03:19:58.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b25ffb640 1 -- 172.21.15.110:0/666304321 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3c015970 con 0x55db9b94eb50 2024-04-27T03:19:58.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db9b950420 con 0x55db9b94eb50 2024-04-27T03:19:58.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b25ffb640 1 -- 172.21.15.110:0/666304321 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c3c015c50 con 0x55db9b94eb50 2024-04-27T03:19:58.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b25ffb640 1 -- 172.21.15.110:0/666304321 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0c3c012440 con 0x55db9b94eb50 2024-04-27T03:19:58.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b25ffb640 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b180346a0 0x7f0b18036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b25ffb640 1 -- 172.21.15.110:0/666304321 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f0c3c00e040 con 0x55db9b94eb50 2024-04-27T03:19:58.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c410b4640 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b180346a0 0x7f0b18036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 Processor -- start 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- start start 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c410b4640 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b180346a0 0x7f0b18036b60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55db9b93ef80 tx=0x7f0c340bff90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b95c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db9b92d4a0 con 0x55db9b95bcb0 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c418b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b95c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c418b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b95c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:51998/0 (socket says 172.21.15.110:51998) 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c418b5640 1 -- 172.21.15.110:0/1225511657 learned_addr learned my addr 172.21.15.110:0/1225511657 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c418b5640 1 -- 172.21.15.110:0/1225511657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9b95c6a0 con 0x55db9b95bcb0 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c418b5640 1 --2- 172.21.15.110:0/1225511657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b95c0d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c04a6d0 tx=0x7f0c3c013870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc271414e2afc51 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:58.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b1f7fe640 1 -- 172.21.15.110:0/1225511657 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c3c028520 con 0x55db9b95bcb0 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b1f7fe640 1 -- 172.21.15.110:0/1225511657 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3c042070 con 0x55db9b95bcb0 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0b1f7fe640 1 -- 172.21.15.110:0/1225511657 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c3c011e10 con 0x55db9b95bcb0 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- 172.21.15.110:0/1225511657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 msgr2=0x55db9b95c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 --2- 172.21.15.110:0/1225511657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b95c0d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c04a6d0 tx=0x7f0c3c013870 comp rx=0 tx=0).stop 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- 172.21.15.110:0/1225511657 shutdown_connections 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 --2- 172.21.15.110:0/1225511657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b95c0d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- 172.21.15.110:0/1225511657 >> 172.21.15.110:0/1225511657 conn(0x55db9b958040 msgr2=0x55db9b9560a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.982+0000 7f0c43599980 1 -- 172.21.15.110:0/1225511657 shutdown_connections 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 -- 172.21.15.110:0/1225511657 wait complete. 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 Processor -- start 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 -- start start 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b9631d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55db9b95cdc0 con 0x55db9b95bcb0 2024-04-27T03:19:58.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c418b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b9631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c418b5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b9631d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52010/0 (socket says 172.21.15.110:52010) 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 learned_addr learned my addr 172.21.15.110:0/1949103662 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db9b963710 con 0x55db9b95bcb0 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c418b5640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b9631d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c01e910 tx=0x7f0c3c04a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0b1dffb640 1 -- 172.21.15.110:0/1949103662 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c3c036070 con 0x55db9b95bcb0 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55db9b9639a0 con 0x55db9b95bcb0 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0b1dffb640 1 -- 172.21.15.110:0/1949103662 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3c011730 con 0x55db9b95bcb0 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0b1dffb640 1 -- 172.21.15.110:0/1949103662 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c3c048030 con 0x55db9b95bcb0 2024-04-27T03:19:59.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db9b966680 con 0x55db9b95bcb0 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0b1dffb640 1 -- 172.21.15.110:0/1949103662 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0c3c039070 con 0x55db9b95bcb0 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0b1dffb640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b10034650 0x7f0b10036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0b1dffb640 1 -- 172.21.15.110:0/1949103662 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f0c3c029070 con 0x55db9b95bcb0 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c410b4640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b10034650 0x7f0b10036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b96e850 0x55db9b970cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b971410 con 0x55db9b96e850 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.986+0000 7f0c3bfff640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b96e850 0x55db9b970cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c410b4640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b10034650 0x7f0b10036b10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0c3401bbe0 tx=0x7f0c340d8000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c3bfff640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b96e850 0x55db9b970cc0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c2c00f6a0 con 0x55db9b96e850 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 realm 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b96ddb0 con 0x55db9b96e850 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c2c00f6a0 con 0x55db9b96e850 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b971ee0 con 0x55db9b96e850 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c2c00f6a0 con 0x55db9b96e850 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b972cc0 0x55db9b9750b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b975730 con 0x55db9b972cc0 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c418b5640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b972cc0 0x55db9b9750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c418b5640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b972cc0 0x55db9b9750b0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b975f80 con 0x55db9b96e850 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f0c2c00f6a0 con 0x55db9b96e850 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 20 searching for the correct realm 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db9b977cf0 0x55db9b97a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b977cf0 2024-04-27T03:19:59.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c410b4640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db9b977cf0 0x55db9b97a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c410b4640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db9b977cf0 0x55db9b97a0e0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.990+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c418b5640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c3c03b070 0x7f0c3c03b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x7f0c3c03b070 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c3c03b070 0x7f0c3c03b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c3c03b070 0x7f0c3c03b870 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c00f6a0 con 0x7f0c3c03b070 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c00f6a0 con 0x55db9b96e850 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b977cf0 2024-04-27T03:19:59.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.994+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x7f0c3c03b070 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x7f0c3c03b070 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x7f0c3c03b070 2024-04-27T03:19:59.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f0c2c02b040 con 0x7f0c3c03b070 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b977cf0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b977cf0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x7f0c3c03b070 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f0c2c02b040 con 0x7f0c3c03b070 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:58.998+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b977cf0 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x7f0c3c03b070 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x7f0c3c03b070 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b96e850 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c02b040 con 0x55db9b96e850 2024-04-27T03:19:59.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x7f0c3c03b070 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b977cf0 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x7f0c3c03b070 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b977cf0 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.002+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97a760 con 0x55db9b972cc0 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:59.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b976930 con 0x55db9b96e850 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b9771d0 con 0x55db9b972cc0 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b976570 con 0x55db9b96e850 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 zone default found 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 4 Realm: () 2024-04-27T03:19:59.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 zonegroup default 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97b070 con 0x7f0c3c03b070 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97ce70 con 0x55db9b977cf0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97d240 con 0x55db9b977cf0 2024-04-27T03:19:59.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 started sync module instance, tier type = 2024-04-27T03:19:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:59.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.006+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97c310 con 0x55db9b972cc0 2024-04-27T03:19:59.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94401696614448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97c310 con 0x55db9b972cc0 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97dd20 con 0x55db9b96e850 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94401696618512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97dce0 con 0x55db9b96e850 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97ec50 con 0x55db9b972cc0 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94401696614448] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94401696618512] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94401696620816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97dce0 con 0x55db9b972cc0 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.010+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97f0b0 con 0x7f0c3c03b070 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94401696611936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97f070 con 0x7f0c3c03b070 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97ff90 con 0x55db9b972cc0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94401696611936] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94401696620816] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94401696622432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97ff90 con 0x55db9b972cc0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.014+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b980380 con 0x55db9b977cf0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94401696625824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b980c20 con 0x55db9b977cf0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b981620 con 0x55db9b977cf0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94401696622432] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94401696625824] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94401696630640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b981130 con 0x55db9b977cf0 2024-04-27T03:19:59.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b982030 con 0x7f0c3c03b070 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94401696630640] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'267 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94401696627392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b982030 con 0x7f0c3c03b070 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=0 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=1 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=2 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=3 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=4 2024-04-27T03:19:59.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=5 2024-04-27T03:19:59.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=6 2024-04-27T03:19:59.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94401696627392] v31'268 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 20 add_watcher() i=7 2024-04-27T03:19:59.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:59.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.018+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b985310 con 0x55db9b96e850 2024-04-27T03:19:59.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94401696636144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b9825c0 con 0x55db9b96e850 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94401696636144] v31'269 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0af67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 5 note: GC not initialized 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0af57e2640 20 reqs_thread_entry: start 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 20 Filter name: none 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0a66ffd640 20 reqs_thread_entry: start 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:59.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b9aae00 con 0x55db9b96e850 2024-04-27T03:19:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~503 out=503b,getxattrs out=75b] v0'0 uv8 ondisk = 0) ==== 231+0+626 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 20 rados_obj.operate() r=0 bl.length=503 2024-04-27T03:19:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:59.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.022+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~460 in=460b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b9b17d0 con 0x55db9b96e850 2024-04-27T03:19:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~460] v31'9 uv9 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c43599980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c43599980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c43599980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94401696830016 in=648b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55db9b9acee0 con 0x55db9b972cc0 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94401696622432 notify 133143986236 ret 0) ==== 678+0+0 (crc 0 0 0) 0x55db9b966680 con 0x55db9b972cc0 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0b1f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986236 cookie 94401696622432 notifier 4884 bl.length()=636 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0b1f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0b1f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0b1f7fe640 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f0b04003eb0 con 0x55db9b972cc0 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94401696830016 out=8b] v0'0 uv267 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94401696830016 notify 133143986236 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55db9b9639a0 con 0x55db9b972cc0 2024-04-27T03:19:59.038 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0af67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.026+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94401696636144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b9b0730 con 0x55db9b96e850 2024-04-27T03:19:59.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.030+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94401696636144] v31'270 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.030+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94401696614448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b981cf0 con 0x55db9b972cc0 2024-04-27T03:19:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.030+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94401696614448] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.030+0000 7f0c43599980 20 remove_watcher() i=0 2024-04-27T03:19:59.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.030+0000 7f0c43599980 2 removed watcher, disabling cache 2024-04-27T03:19:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.030+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94401696618512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b981cf0 con 0x55db9b96e850 2024-04-27T03:19:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94401696618512] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x55db9b96e850 2024-04-27T03:19:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c43599980 20 remove_watcher() i=1 2024-04-27T03:19:59.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94401696620816] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b981cf0 con 0x55db9b972cc0 2024-04-27T03:19:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94401696620816] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c43599980 20 remove_watcher() i=2 2024-04-27T03:19:59.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94401696611936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b981cf0 con 0x7f0c3c03b070 2024-04-27T03:19:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94401696611936] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c43599980 20 remove_watcher() i=3 2024-04-27T03:19:59.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.034+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94401696622432] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b981cf0 con 0x55db9b972cc0 2024-04-27T03:19:59.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c418b5640 1 -- 172.21.15.110:0/1949103662 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94401696622432] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c3c053070 con 0x55db9b972cc0 2024-04-27T03:19:59.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c43599980 20 remove_watcher() i=4 2024-04-27T03:19:59.048 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94401696625824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97e760 con 0x55db9b977cf0 2024-04-27T03:19:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94401696625824] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c43599980 20 remove_watcher() i=5 2024-04-27T03:19:59.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94401696630640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b97e760 con 0x55db9b977cf0 2024-04-27T03:19:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c410b4640 1 -- 172.21.15.110:0/1949103662 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94401696630640] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c340c5020 con 0x55db9b977cf0 2024-04-27T03:19:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c43599980 20 remove_watcher() i=6 2024-04-27T03:19:59.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.038+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94401696627392] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55db9b980770 con 0x7f0c3c03b070 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94401696627392] v31'269 uv267 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f0c2c03e040 con 0x7f0c3c03b070 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 20 remove_watcher() i=7 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b972cc0 msgr2=0x55db9b9750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b972cc0 0x55db9b9750b0 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b96e850 msgr2=0x55db9b970cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b96e850 0x55db9b970cc0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c3c03b070 msgr2=0x7f0c3c03b870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f0c3c03b070 0x7f0c3c03b870 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db9b977cf0 msgr2=0x55db9b97a0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55db9b977cf0 0x55db9b97a0e0 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b10034650 msgr2=0x7f0b10036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b10034650 0x7f0b10036b10 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0c3401bbe0 tx=0x7f0c340d8000 comp rx=0 tx=0).stop 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c3bfff640 1 -- 172.21.15.110:0/1949103662 reap_dead start 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 msgr2=0x55db9b9631d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b9631d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c01e910 tx=0x7f0c3c04a630 comp rx=0 tx=0).stop 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 shutdown_connections 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/1949103662 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b95bcb0 0x55db9b9631d0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 >> 172.21.15.110:0/1949103662 conn(0x55db9b958040 msgr2=0x55db9b95d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 shutdown_connections 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/1949103662 wait complete. 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b180346a0 msgr2=0x7f0b18036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b180346a0 0x7f0b18036b60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x55db9b93ef80 tx=0x7f0c340bff90 comp rx=0 tx=0).stop 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b94eb50 msgr2=0x55db9b94ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b94eb50 0x55db9b94ef70 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c017680 tx=0x7f0c3c017a80 comp rx=0 tx=0).stop 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 shutdown_connections 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b180346a0 0x7f0b18036b60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/666304321 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b94eb50 0x55db9b94ef70 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 >> 172.21.15.110:0/666304321 conn(0x55db9b943960 msgr2=0x55db9b944470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 shutdown_connections 2024-04-27T03:19:59.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/666304321 wait complete. 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b93bd70 msgr2=0x55db9b93e210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b93bd70 0x55db9b93e210 crc :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b843340 msgr2=0x55db9b93ad80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b843340 0x55db9b93ad80 crc :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b280346a0 msgr2=0x7f0b28036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b280346a0 0x7f0b28036b60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c001e50 tx=0x7f0c2c0013b0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 msgr2=0x55db9b933d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b933d30 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0c3401b380 tx=0x7f0c340180e0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 shutdown_connections 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55db9b93bd70 0x55db9b93e210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55db9b843340 0x55db9b93ad80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f0b280346a0 0x7f0b28036b60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 --2- 172.21.15.110:0/2463236437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55db9b842a20 0x55db9b933d30 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 >> 172.21.15.110:0/2463236437 conn(0x55db9b83b1d0 msgr2=0x55db9b927450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 shutdown_connections 2024-04-27T03:19:59.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.042+0000 7f0c43599980 1 -- 172.21.15.110:0/2463236437 wait complete. 2024-04-27T03:19:59.061 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.062 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.062 ERROR:root:0 2024-04-27T03:19:59.062 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'ada', '--caps', 'user-info-without-keys=write'] 2024-04-27T03:19:59.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'ada', '--caps', 'user-info-without-keys=write'] 2024-04-27T03:19:59.062 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid ada --caps user-info-without-keys=write 2024-04-27T03:19:59.142 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:59.143 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc503e73980 1 Processor -- start 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc503e73980 1 -- start start 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc503e73980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a914f4a4a0 0x55a914f4a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc503e73980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55a914e070d0 con 0x55a914f4adc0 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc503e73980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55a914dfc430 con 0x55a914f4a4a0 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc50198e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a914f4a4a0 0x55a914f4a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc50198e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55a914f4a4a0 0x55a914f4a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52024/0 (socket says 172.21.15.110:52024) 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc50118d640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55a914f4adc0 0x55a914f4b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44628/0 (socket says 172.21.15.110:44628) 2024-04-27T03:19:59.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc50198e640 1 -- 172.21.15.110:0/875573112 learned_addr learned my addr 172.21.15.110:0/875573112 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc50198e640 1 -- 172.21.15.110:0/875573112 >> v1:172.21.15.77:6789/0 conn(0x55a914f4adc0 legacy=0x55a914f4b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.134+0000 7fc50198e640 1 -- 172.21.15.110:0/875573112 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a914df5510 con 0x55a914f4a4a0 2024-04-27T03:19:59.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc50198e640 1 --2- 172.21.15.110:0/875573112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a914f4a880 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00a4e0 tx=0x7fc4fc01b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19213ce4c5c524a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc50098c640 1 -- 172.21.15.110:0/875573112 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4fc0aa540 con 0x55a914f4a4a0 2024-04-27T03:19:59.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc50098c640 1 -- 172.21.15.110:0/875573112 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc0aa6e0 con 0x55a914f4a4a0 2024-04-27T03:19:59.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc50098c640 1 -- 172.21.15.110:0/875573112 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4fc0aaa40 con 0x55a914f4a4a0 2024-04-27T03:19:59.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc503e73980 1 -- 172.21.15.110:0/875573112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 msgr2=0x55a914f4a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc503e73980 1 --2- 172.21.15.110:0/875573112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a914f4a880 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc00a4e0 tx=0x7fc4fc01b090 comp rx=0 tx=0).stop 2024-04-27T03:19:59.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc503e73980 1 -- 172.21.15.110:0/875573112 shutdown_connections 2024-04-27T03:19:59.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc503e73980 1 --2- 172.21.15.110:0/875573112 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a914f4a880 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc503e73980 1 -- 172.21.15.110:0/875573112 >> 172.21.15.110:0/875573112 conn(0x55a914ebdba0 msgr2=0x55a914ebdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc503e73980 1 -- 172.21.15.110:0/875573112 shutdown_connections 2024-04-27T03:19:59.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.138+0000 7fc503e73980 1 -- 172.21.15.110:0/875573112 wait complete. 2024-04-27T03:19:59.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc503e73980 1 Processor -- start 2024-04-27T03:19:59.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc503e73980 1 -- start start 2024-04-27T03:19:59.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc503e73980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a915035da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc503e73980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a914e4b2d0 con 0x55a914f4a4a0 2024-04-27T03:19:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc50198e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a915035da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc50198e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a915035da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52034/0 (socket says 172.21.15.110:52034) 2024-04-27T03:19:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc50198e640 1 -- 172.21.15.110:0/1808212791 learned_addr learned my addr 172.21.15.110:0/1808212791 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc50198e640 1 -- 172.21.15.110:0/1808212791 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a914dd1ae0 con 0x55a914f4a4a0 2024-04-27T03:19:59.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc50198e640 1 --2- 172.21.15.110:0/1808212791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a915035da0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc017ca0 tx=0x7fc4fc0033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf6010e41a9263d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc3efe07640 1 -- 172.21.15.110:0/1808212791 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4fc0b9030 con 0x55a914f4a4a0 2024-04-27T03:19:59.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc3efe07640 1 -- 172.21.15.110:0/1808212791 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc0aaac0 con 0x55a914f4a4a0 2024-04-27T03:19:59.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.150+0000 7fc3efe07640 1 -- 172.21.15.110:0/1808212791 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4fc0aada0 con 0x55a914f4a4a0 2024-04-27T03:19:59.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/1808212791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 msgr2=0x55a915035da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 --2- 172.21.15.110:0/1808212791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a915035da0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc017ca0 tx=0x7fc4fc0033b0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/1808212791 shutdown_connections 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 --2- 172.21.15.110:0/1808212791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a915035da0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/1808212791 >> 172.21.15.110:0/1808212791 conn(0x55a914ebdba0 msgr2=0x55a915031930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/1808212791 shutdown_connections 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/1808212791 wait complete. 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 Processor -- start 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- start start 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a91503cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a914f54500 con 0x55a914f4a4a0 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc50198e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a91503cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc50198e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a91503cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52040/0 (socket says 172.21.15.110:52040) 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc50198e640 1 -- 172.21.15.110:0/2114588627 learned_addr learned my addr 172.21.15.110:0/2114588627 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc50198e640 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a91503d260 con 0x55a914f4a4a0 2024-04-27T03:19:59.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc50198e640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a91503cd20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc01b0d0 tx=0x7fc4fc017e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc3ee604640 1 -- 172.21.15.110:0/2114588627 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4fc0b9030 con 0x55a914f4a4a0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a91503d4f0 con 0x55a914f4a4a0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc3ee604640 1 -- 172.21.15.110:0/2114588627 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc0c3070 con 0x55a914f4a4a0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc3ee604640 1 -- 172.21.15.110:0/2114588627 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4fc0bf750 con 0x55a914f4a4a0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9150400d0 con 0x55a914f4a4a0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc3ee604640 1 -- 172.21.15.110:0/2114588627 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc4fc0c5030 con 0x55a914f4a4a0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc3ee604640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3e8034650 0x7fc3e8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc3ee604640 1 -- 172.21.15.110:0/2114588627 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc4fc0ba070 con 0x55a914f4a4a0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc50118d640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3e8034650 0x7fc3e8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a914f4adc0 0x55a915043bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915044130 con 0x55a914f4adc0 2024-04-27T03:19:59.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.154+0000 7fc50218f640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a914f4adc0 0x55a915043bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50118d640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3e8034650 0x7fc3e8036b10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8000f70 tx=0x7fc4f8001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50218f640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a914f4adc0 0x55a915043bb0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50218f640 1 -- 172.21.15.110:0/2114588627 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc4f40083f0 con 0x55a914f4adc0 2024-04-27T03:19:59.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a915044f00 0x55a9150472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915047930 con 0x55a915044f00 2024-04-27T03:19:59.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50198e640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a915044f00 0x55a9150472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50198e640 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a915044f00 0x55a9150472f0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50198e640 1 -- 172.21.15.110:0/2114588627 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915044f00 2024-04-27T03:19:59.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9150491e0 con 0x55a914f4adc0 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50218f640 1 -- 172.21.15.110:0/2114588627 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fc4f40083f0 con 0x55a914f4adc0 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915048410 con 0x55a915044f00 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50198e640 1 -- 172.21.15.110:0/2114588627 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915044f00 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915048800 con 0x55a914f4adc0 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc50218f640 1 -- 172.21.15.110:0/2114588627 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fc4f40083f0 con 0x55a914f4adc0 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc503e73980 1 Processor -- start 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.158+0000 7fc503e73980 1 -- start start 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915057ae0 0x55a915057f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a914e4f310 con 0x55a915057ae0 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915057ae0 0x55a915057f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915057ae0 0x55a915057f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52056/0 (socket says 172.21.15.110:52056) 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 -- 172.21.15.110:0/506405485 learned_addr learned my addr 172.21.15.110:0/506405485 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 -- 172.21.15.110:0/506405485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a915058440 con 0x55a915057ae0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915057ae0 0x55a915057f00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400aab0 tx=0x7fc4f400af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc3e77fe640 1 -- 172.21.15.110:0/506405485 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4007d30 con 0x55a915057ae0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc3e77fe640 1 -- 172.21.15.110:0/506405485 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f401d070 con 0x55a915057ae0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc3e77fe640 1 -- 172.21.15.110:0/506405485 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4018440 con 0x55a915057ae0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a915058ee0 con 0x55a915057ae0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9150593b0 con 0x55a915057ae0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc3e77fe640 1 -- 172.21.15.110:0/506405485 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc4f4013070 con 0x55a915057ae0 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc3e77fe640 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3dc034650 0x7fc3dc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc3e77fe640 1 -- 172.21.15.110:0/506405485 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc4f404b660 con 0x55a915057ae0 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50198e640 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3dc034650 0x7fc3dc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 Processor -- start 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 -- start start 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50198e640 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3dc034650 0x7fc3dc036b10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55a915047f30 tx=0x7fc4fc0e3800 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a915065100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc503e73980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a915036430 con 0x55a915064ce0 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a915065100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a915065100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52072/0 (socket says 172.21.15.110:52072) 2024-04-27T03:19:59.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.162+0000 7fc50218f640 1 -- 172.21.15.110:0/877178392 learned_addr learned my addr 172.21.15.110:0/877178392 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc50218f640 1 -- 172.21.15.110:0/877178392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9150656d0 con 0x55a915064ce0 2024-04-27T03:19:59.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc50218f640 1 --2- 172.21.15.110:0/877178392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a915065100 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4049930 tx=0x7fc4f4018950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe78251709e3218 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc3e57fa640 1 -- 172.21.15.110:0/877178392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4047070 con 0x55a915064ce0 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc3e57fa640 1 -- 172.21.15.110:0/877178392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f4018bc0 con 0x55a915064ce0 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc3e57fa640 1 -- 172.21.15.110:0/877178392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4020360 con 0x55a915064ce0 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- 172.21.15.110:0/877178392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 msgr2=0x55a915065100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 --2- 172.21.15.110:0/877178392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a915065100 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4049930 tx=0x7fc4f4018950 comp rx=0 tx=0).stop 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- 172.21.15.110:0/877178392 shutdown_connections 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 --2- 172.21.15.110:0/877178392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a915065100 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- 172.21.15.110:0/877178392 >> 172.21.15.110:0/877178392 conn(0x55a915061020 msgr2=0x55a91505f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- 172.21.15.110:0/877178392 shutdown_connections 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- 172.21.15.110:0/877178392 wait complete. 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 Processor -- start 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- start start 2024-04-27T03:19:59.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a91506c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55a915065e30 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc50218f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a91506c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc50218f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a91506c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52076/0 (socket says 172.21.15.110:52076) 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 learned_addr learned my addr 172.21.15.110:0/1474917481 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a91506c700 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc50218f640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a91506c1c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40168b0 tx=0x7fc4f40498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc3d37fe640 1 -- 172.21.15.110:0/1474917481 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4031070 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a91506c990 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc3d37fe640 1 -- 172.21.15.110:0/1474917481 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f404b2e0 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc3d37fe640 1 -- 172.21.15.110:0/1474917481 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4f4049a70 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.166+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a91506f670 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc3d37fe640 1 -- 172.21.15.110:0/1474917481 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc4f4034070 con 0x55a915064ce0 2024-04-27T03:19:59.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc3d37fe640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3cc034650 0x7fc3cc036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc3d37fe640 1 -- 172.21.15.110:0/1474917481 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc4f400c0b0 con 0x55a915064ce0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50198e640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3cc034650 0x7fc3cc036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a915077830 0x55a915079cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a91507a440 con 0x55a915077830 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50118d640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a915077830 0x55a915079cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50198e640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3cc034650 0x7fc3cc036b10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0c6fd0 tx=0x7fc4fc0db000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50118d640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a915077830 0x55a915079cf0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc4f800f620 con 0x55a915077830 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 realm 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915076d00 con 0x55a915077830 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc4f800f620 con 0x55a915077830 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a91507af10 con 0x55a915077830 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc4f800f620 con 0x55a915077830 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a91507bcf0 0x55a91507e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a91507e770 con 0x55a91507bcf0 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50218f640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a91507bcf0 0x55a91507e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50218f640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a91507bcf0 0x55a91507e0e0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.170+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a91507efc0 con 0x55a915077830 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fc4f800f620 con 0x55a915077830 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc503e73980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc503e73980 20 searching for the correct realm 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a915080d30 0x55a915083120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915080d30 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50198e640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a915080d30 0x55a915083120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50198e640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a915080d30 0x55a915083120 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50218f640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc4f4036070 0x7fc4f402e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x7fc4f4036070 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc4f4036070 0x7fc4f402e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc4f4036070 0x7fc4f402e8d0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f800f620 con 0x7fc4f4036070 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f800f620 con 0x55a915077830 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915080d30 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.174+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x7fc4f4036070 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x7fc4f4036070 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x7fc4f4036070 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fc4f802b040 con 0x7fc4f4036070 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915080d30 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915080d30 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.178+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x7fc4f4036070 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fc4f802b040 con 0x7fc4f4036070 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915080d30 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x7fc4f4036070 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x7fc4f4036070 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915077830 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f802b040 con 0x55a915077830 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x7fc4f4036070 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915080d30 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x7fc4f4036070 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a915080d30 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.182+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55a915083730 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:59.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a91507f970 con 0x55a915077830 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915080210 con 0x55a91507bcf0 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a91507f5b0 con 0x55a915077830 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 zone default found 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 4 Realm: () 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:59.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 zonegroup default 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915084060 con 0x7fc4f4036070 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915085e60 con 0x55a915080d30 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a915086200 con 0x55a915080d30 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 started sync module instance, tier type = 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.186+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a915085300 con 0x55a91507bcf0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94184690702112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a915084f60 con 0x55a91507bcf0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150870c0 con 0x55a915077830 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94184690707472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150874e0 con 0x55a915077830 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a915087f90 con 0x55a91507bcf0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94184690702112] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.190+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94184690707472] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94184690705904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a915087030 con 0x55a91507bcf0 2024-04-27T03:19:59.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a915088830 con 0x7fc4f4036070 2024-04-27T03:19:59.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94184690713472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a915087030 con 0x7fc4f4036070 2024-04-27T03:19:59.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150892f0 con 0x55a91507bcf0 2024-04-27T03:19:59.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94184690713472] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94184690705904] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94184690716096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150892b0 con 0x55a91507bcf0 2024-04-27T03:19:59.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.194+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150896e0 con 0x55a915080d30 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94184690711152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a915089f80 con 0x55a915080d30 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a91508a940 con 0x55a915080d30 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94184690716096] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94184690711152] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94184690721936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a91508ace0 con 0x55a915080d30 2024-04-27T03:19:59.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a91508b570 con 0x7fc4f4036070 2024-04-27T03:19:59.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'270 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94184690724064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a91508b140 con 0x7fc4f4036070 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 20 add_watcher() i=0 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 20 add_watcher() i=1 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 20 add_watcher() i=2 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 20 add_watcher() i=3 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 20 add_watcher() i=4 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 20 add_watcher() i=5 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94184690721936] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc503e73980 20 add_watcher() i=6 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.198+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94184690724064] v31'271 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 20 add_watcher() i=7 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9150887a0 con 0x55a915077830 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94184690714528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a91508bfc0 con 0x55a915077830 2024-04-27T03:19:59.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94184690714528] v31'271 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc3b37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 5 note: GC not initialized 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc3b27fc640 20 reqs_thread_entry: start 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 20 Filter name: none 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc323fff640 20 reqs_thread_entry: start 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9150b22b0 con 0x55a915077830 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~460 out=460b,getxattrs out=75b] v0'0 uv9 ondisk = 0) ==== 231+0+583 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 20 rados_obj.operate() r=0 bl.length=460 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:59.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.202+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~490 in=490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150b88e0 con 0x55a915077830 2024-04-27T03:19:59.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~490] v31'10 uv10 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc503e73980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc503e73980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc503e73980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:59.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94184690910032 in=678b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55a9150b3fd0 con 0x55a91507bcf0 2024-04-27T03:19:59.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94184690716096 notify 133143986237 ret 0) ==== 708+0+0 (crc 0 0 0) 0x55a91506f670 con 0x55a91507bcf0 2024-04-27T03:19:59.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc3e57fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986237 cookie 94184690716096 notifier 4891 bl.length()=666 2024-04-27T03:19:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc3e57fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc3e57fa640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc3e57fa640 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fc3c4003eb0 con 0x55a91507bcf0 2024-04-27T03:19:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94184690910032 out=8b] v0'0 uv270 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.206+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94184690910032 notify 133143986237 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55a91506c990 con 0x55a91507bcf0 2024-04-27T03:19:59.219 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"user-info-without-keys","perm":"write"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:59.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.210+0000 7fc3b37fe640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.210+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94184690714528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x55a915077830 2024-04-27T03:19:59.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.210+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94184690714528] v31'272 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.210+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94184690702112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x55a91507bcf0 2024-04-27T03:19:59.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94184690702112] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc503e73980 20 remove_watcher() i=0 2024-04-27T03:19:59.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc503e73980 2 removed watcher, disabling cache 2024-04-27T03:19:59.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94184690707472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x55a915077830 2024-04-27T03:19:59.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94184690707472] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x55a915077830 2024-04-27T03:19:59.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc503e73980 20 remove_watcher() i=1 2024-04-27T03:19:59.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94184690705904] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x55a91507bcf0 2024-04-27T03:19:59.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94184690705904] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc503e73980 20 remove_watcher() i=2 2024-04-27T03:19:59.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.214+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94184690713472] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x7fc4f4036070 2024-04-27T03:19:59.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.218+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94184690713472] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.218+0000 7fc503e73980 20 remove_watcher() i=3 2024-04-27T03:19:59.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.218+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94184690716096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x55a91507bcf0 2024-04-27T03:19:59.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.218+0000 7fc50218f640 1 -- 172.21.15.110:0/1474917481 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94184690716096] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f4033030 con 0x55a91507bcf0 2024-04-27T03:19:59.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.218+0000 7fc503e73980 20 remove_watcher() i=4 2024-04-27T03:19:59.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.218+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94184690711152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x55a915080d30 2024-04-27T03:19:59.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94184690711152] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 20 remove_watcher() i=5 2024-04-27T03:19:59.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94184690721936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x55a915080d30 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc50198e640 1 -- 172.21.15.110:0/1474917481 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94184690721936] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4fc0c6ce0 con 0x55a915080d30 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 20 remove_watcher() i=6 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94184690724064] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55a9150affb0 con 0x7fc4f4036070 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94184690724064] v31'272 uv270 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fc4f803e040 con 0x7fc4f4036070 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 20 remove_watcher() i=7 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a91507bcf0 msgr2=0x55a91507e0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a91507bcf0 0x55a91507e0e0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a915077830 msgr2=0x55a915079cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a915077830 0x55a915079cf0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc4f4036070 msgr2=0x7fc4f402e8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fc4f4036070 0x7fc4f402e8d0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a915080d30 msgr2=0x55a915083120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55a915080d30 0x55a915083120 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3cc034650 msgr2=0x7fc3cc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3cc034650 0x7fc3cc036b10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0c6fd0 tx=0x7fc4fc0db000 comp rx=0 tx=0).stop 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 msgr2=0x55a91506c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc50118d640 1 -- 172.21.15.110:0/1474917481 reap_dead start 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a91506c1c0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40168b0 tx=0x7fc4f40498c0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 shutdown_connections 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/1474917481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915064ce0 0x55a91506c1c0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 >> 172.21.15.110:0/1474917481 conn(0x55a915061020 msgr2=0x55a91505e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 shutdown_connections 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/1474917481 wait complete. 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3dc034650 msgr2=0x7fc3dc036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3dc034650 0x7fc3dc036b10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55a915047f30 tx=0x7fc4fc0e3800 comp rx=0 tx=0).stop 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915057ae0 msgr2=0x55a915057f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915057ae0 0x55a915057f00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400aab0 tx=0x7fc4f400af70 comp rx=0 tx=0).stop 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 shutdown_connections 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3dc034650 0x7fc3dc036b10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/506405485 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a915057ae0 0x55a915057f00 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 >> 172.21.15.110:0/506405485 conn(0x55a91504c8f0 msgr2=0x55a91504d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 shutdown_connections 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/506405485 wait complete. 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a915044f00 msgr2=0x55a9150472f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a915044f00 0x55a9150472f0 crc :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a914f4adc0 msgr2=0x55a915043bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a914f4adc0 0x55a915043bb0 crc :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3e8034650 msgr2=0x7fc3e8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3e8034650 0x7fc3e8036b10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8000f70 tx=0x7fc4f8001380 comp rx=0 tx=0).stop 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 msgr2=0x55a91503cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a91503cd20 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc01b0d0 tx=0x7fc4fc017e30 comp rx=0 tx=0).stop 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 shutdown_connections 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55a915044f00 0x55a9150472f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55a914f4adc0 0x55a915043bb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3e8034650 0x7fc3e8036b10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 --2- 172.21.15.110:0/2114588627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55a914f4a4a0 0x55a91503cd20 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 >> 172.21.15.110:0/2114588627 conn(0x55a914ebdba0 msgr2=0x55a915030440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 shutdown_connections 2024-04-27T03:19:59.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.222+0000 7fc503e73980 1 -- 172.21.15.110:0/2114588627 wait complete. 2024-04-27T03:19:59.243 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': 'write'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.244 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'user-info-without-keys', 'perm': 'write'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.244 ERROR:root:0 2024-04-27T03:19:59.244 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.249 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'AccessDenied', 'Message': '', 'RequestId': 'tx000005af78213f31d39af-00662c6edf-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:19:59.249 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'rm', '--uid', 'ada', '--caps', 'user-info-without-keys=write'] 2024-04-27T03:19:59.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'rm', '--uid', 'ada', '--caps', 'user-info-without-keys=write'] 2024-04-27T03:19:59.250 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps rm --uid ada --caps user-info-without-keys=write 2024-04-27T03:19:59.322 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:59.323 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:59.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc07198980 1 Processor -- start 2024-04-27T03:19:59.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc07198980 1 -- start start 2024-04-27T03:19:59.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc07198980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d06e0784c0 0x55d06e0788a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc07198980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55d06df350d0 con 0x55d06e078de0 2024-04-27T03:19:59.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc07198980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55d06df2a430 con 0x55d06e0784c0 2024-04-27T03:19:59.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc04cb3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d06e0784c0 0x55d06e0788a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc04cb3640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55d06e0784c0 0x55d06e0788a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52088/0 (socket says 172.21.15.110:52088) 2024-04-27T03:19:59.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc044b2640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55d06e078de0 0x55d06e0797a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44634/0 (socket says 172.21.15.110:44634) 2024-04-27T03:19:59.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc04cb3640 1 -- 172.21.15.110:0/3994590609 learned_addr learned my addr 172.21.15.110:0/3994590609 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc04cb3640 1 -- 172.21.15.110:0/3994590609 >> v1:172.21.15.77:6789/0 conn(0x55d06e078de0 legacy=0x55d06e0797a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.326 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.314+0000 7fdc04cb3640 1 -- 172.21.15.110:0/3994590609 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06df23510 con 0x55d06e0784c0 2024-04-27T03:19:59.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/3994590609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e0788a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400a4e0 tx=0x7fdbf401b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afd54e2efad1d24 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc03cb1640 1 -- 172.21.15.110:0/3994590609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbf40aa540 con 0x55d06e0784c0 2024-04-27T03:19:59.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc03cb1640 1 -- 172.21.15.110:0/3994590609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf40aa6e0 con 0x55d06e0784c0 2024-04-27T03:19:59.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc03cb1640 1 -- 172.21.15.110:0/3994590609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbf40aaa40 con 0x55d06e0784c0 2024-04-27T03:19:59.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc07198980 1 -- 172.21.15.110:0/3994590609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 msgr2=0x55d06e0788a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc07198980 1 --2- 172.21.15.110:0/3994590609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e0788a0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400a4e0 tx=0x7fdbf401b090 comp rx=0 tx=0).stop 2024-04-27T03:19:59.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc07198980 1 -- 172.21.15.110:0/3994590609 shutdown_connections 2024-04-27T03:19:59.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc07198980 1 --2- 172.21.15.110:0/3994590609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e0788a0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc07198980 1 -- 172.21.15.110:0/3994590609 >> 172.21.15.110:0/3994590609 conn(0x55d06dfebba0 msgr2=0x55d06dfebfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc07198980 1 -- 172.21.15.110:0/3994590609 shutdown_connections 2024-04-27T03:19:59.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.318+0000 7fdc07198980 1 -- 172.21.15.110:0/3994590609 wait complete. 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc07198980 1 Processor -- start 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc07198980 1 -- start start 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc07198980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e163d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc07198980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d06df792d0 con 0x55d06e0784c0 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc04cb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e163d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc04cb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e163d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52098/0 (socket says 172.21.15.110:52098) 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc04cb3640 1 -- 172.21.15.110:0/3901159961 learned_addr learned my addr 172.21.15.110:0/3901159961 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc04cb3640 1 -- 172.21.15.110:0/3901159961 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06deffae0 con 0x55d06e0784c0 2024-04-27T03:19:59.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/3901159961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e163d80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4017ca0 tx=0x7fdbf40033b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=658d546d2d0fdeeb server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdaf2e0d640 1 -- 172.21.15.110:0/3901159961 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbf40b9030 con 0x55d06e0784c0 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdaf2e0d640 1 -- 172.21.15.110:0/3901159961 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf40aaac0 con 0x55d06e0784c0 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.330+0000 7fdaf2e0d640 1 -- 172.21.15.110:0/3901159961 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbf40aada0 con 0x55d06e0784c0 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- 172.21.15.110:0/3901159961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 msgr2=0x55d06e163d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 --2- 172.21.15.110:0/3901159961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e163d80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4017ca0 tx=0x7fdbf40033b0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- 172.21.15.110:0/3901159961 shutdown_connections 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 --2- 172.21.15.110:0/3901159961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e163d80 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- 172.21.15.110:0/3901159961 >> 172.21.15.110:0/3901159961 conn(0x55d06dfebba0 msgr2=0x55d06e15f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- 172.21.15.110:0/3901159961 shutdown_connections 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- 172.21.15.110:0/3901159961 wait complete. 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 Processor -- start 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- start start 2024-04-27T03:19:59.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e16ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d06e082510 con 0x55d06e0784c0 2024-04-27T03:19:59.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc04cb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e16ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc04cb3640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e16ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52100/0 (socket says 172.21.15.110:52100) 2024-04-27T03:19:59.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc04cb3640 1 -- 172.21.15.110:0/378135596 learned_addr learned my addr 172.21.15.110:0/378135596 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc04cb3640 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06e16b240 con 0x55d06e0784c0 2024-04-27T03:19:59.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e16ad00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fdbf401b0d0 tx=0x7fdbf4017e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdaf160a640 1 -- 172.21.15.110:0/378135596 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbf40b9030 con 0x55d06e0784c0 2024-04-27T03:19:59.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdaf160a640 1 -- 172.21.15.110:0/378135596 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf40c3070 con 0x55d06e0784c0 2024-04-27T03:19:59.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d06e16b4d0 con 0x55d06e0784c0 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdaf160a640 1 -- 172.21.15.110:0/378135596 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbf40bf750 con 0x55d06e0784c0 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.334+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d06e16e0b0 con 0x55d06e0784c0 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdaf160a640 1 -- 172.21.15.110:0/378135596 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fdbf40c5030 con 0x55d06e0784c0 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdaf160a640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdaec034650 0x7fdaec036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdaf160a640 1 -- 172.21.15.110:0/378135596 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fdbf40ba070 con 0x55d06e0784c0 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc044b2640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdaec034650 0x7fdaec036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e078de0 0x55d06e171b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e172110 con 0x55d06e078de0 2024-04-27T03:19:59.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc054b4640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e078de0 0x55d06e171b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc044b2640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdaec034650 0x7fdaec036b10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdbec000f70 tx=0x7fdbec001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc054b4640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e078de0 0x55d06e171b90 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc054b4640 1 -- 172.21.15.110:0/378135596 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e078de0 2024-04-27T03:19:59.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e172f00 0x55d06e1752f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e175930 con 0x55d06e172f00 2024-04-27T03:19:59.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e172f00 0x55d06e1752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e172f00 0x55d06e1752f0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc04cb3640 1 -- 172.21.15.110:0/378135596 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e172f00 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1771e0 con 0x55d06e078de0 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc054b4640 1 -- 172.21.15.110:0/378135596 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e078de0 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e176410 con 0x55d06e172f00 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc04cb3640 1 -- 172.21.15.110:0/378135596 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e172f00 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.338+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e176800 con 0x55d06e078de0 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc054b4640 1 -- 172.21.15.110:0/378135596 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e078de0 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc07198980 1 Processor -- start 2024-04-27T03:19:59.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc07198980 1 -- start start 2024-04-27T03:19:59.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc07198980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e185b00 0x55d06e185f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc07198980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d06df7d310 con 0x55d06e185b00 2024-04-27T03:19:59.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc054b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e185b00 0x55d06e185f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc054b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e185b00 0x55d06e185f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52116/0 (socket says 172.21.15.110:52116) 2024-04-27T03:19:59.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc054b4640 1 -- 172.21.15.110:0/2776562569 learned_addr learned my addr 172.21.15.110:0/2776562569 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc054b4640 1 -- 172.21.15.110:0/2776562569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06e186460 con 0x55d06e185b00 2024-04-27T03:19:59.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc054b4640 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e185b00 0x55d06e185f20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00aab0 tx=0x7fdbfc00af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdaea7fc640 1 -- 172.21.15.110:0/2776562569 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc007d30 con 0x55d06e185b00 2024-04-27T03:19:59.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d06e186f00 con 0x55d06e185b00 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdaea7fc640 1 -- 172.21.15.110:0/2776562569 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbfc01d070 con 0x55d06e185b00 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d06e1873d0 con 0x55d06e185b00 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdaea7fc640 1 -- 172.21.15.110:0/2776562569 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc022460 con 0x55d06e185b00 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdaea7fc640 1 -- 172.21.15.110:0/2776562569 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fdbfc013070 con 0x55d06e185b00 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdaea7fc640 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdadc0346a0 0x7fdadc036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.342+0000 7fdaea7fc640 1 -- 172.21.15.110:0/2776562569 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fdbfc04b6c0 con 0x55d06e185b00 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdadc0346a0 0x7fdadc036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 Processor -- start 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- start start 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdadc0346a0 0x7fdadc036b60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55d06e175f30 tx=0x7fdbf40e3810 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e1930b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d06e164410 con 0x55d06e192c90 2024-04-27T03:19:59.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e1930b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e1930b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52120/0 (socket says 172.21.15.110:52120) 2024-04-27T03:19:59.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 -- 172.21.15.110:0/4158863813 learned_addr learned my addr 172.21.15.110:0/4158863813 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 -- 172.21.15.110:0/4158863813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06e193680 con 0x55d06e192c90 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 --2- 172.21.15.110:0/4158863813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e1930b0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0204a0 tx=0x7fdbfc049ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32446a4c8a2c9679 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdadbfff640 1 -- 172.21.15.110:0/4158863813 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc03b2a0 con 0x55d06e192c90 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdadbfff640 1 -- 172.21.15.110:0/4158863813 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbfc021c90 con 0x55d06e192c90 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdadbfff640 1 -- 172.21.15.110:0/4158863813 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc018440 con 0x55d06e192c90 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- 172.21.15.110:0/4158863813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 msgr2=0x55d06e1930b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 --2- 172.21.15.110:0/4158863813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e1930b0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0204a0 tx=0x7fdbfc049ad0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- 172.21.15.110:0/4158863813 shutdown_connections 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 --2- 172.21.15.110:0/4158863813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e1930b0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- 172.21.15.110:0/4158863813 >> 172.21.15.110:0/4158863813 conn(0x55d06e18efd0 msgr2=0x55d06e18d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- 172.21.15.110:0/4158863813 shutdown_connections 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- 172.21.15.110:0/4158863813 wait complete. 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 Processor -- start 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- start start 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc07198980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55d06e193e00 con 0x55d06e192c90 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52128/0 (socket says 172.21.15.110:52128) 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 learned_addr learned my addr 172.21.15.110:0/210221818 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d06e19a730 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.346+0000 7fdc054b4640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e19a1f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0487f0 tx=0x7fdbfc049860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdada7fc640 1 -- 172.21.15.110:0/210221818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc018440 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdada7fc640 1 -- 172.21.15.110:0/210221818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbfc0213c0 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d06e19a9c0 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdada7fc640 1 -- 172.21.15.110:0/210221818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdbfc0185e0 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d06e19add0 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdada7fc640 1 -- 172.21.15.110:0/210221818 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fdbfc033060 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdada7fc640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdad00346a0 0x7fdad0036b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdada7fc640 1 -- 172.21.15.110:0/210221818 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fdbfc041070 con 0x55d06e192c90 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdad00346a0 0x7fdad0036b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e1a58d0 0x55d06e1a7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1a84e0 con 0x55d06e1a58d0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc044b2640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e1a58d0 0x55d06e1a7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdad00346a0 0x7fdad0036b60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40bf010 tx=0x7fdbf40db000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc044b2640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e1a58d0 0x55d06e1a7d90 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fdbec00f620 con 0x55d06e1a58d0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 realm 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1a4d60 con 0x55d06e1a58d0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fdbec00f620 con 0x55d06e1a58d0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:59.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1a8fb0 con 0x55d06e1a58d0 2024-04-27T03:19:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fdbec00f620 con 0x55d06e1a58d0 2024-04-27T03:19:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e1a9d90 0x55d06e1ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ac810 con 0x55d06e1a9d90 2024-04-27T03:19:59.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc054b4640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e1a9d90 0x55d06e1ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.350+0000 7fdc054b4640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e1a9d90 0x55d06e1ac180 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc07198980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ad060 con 0x55d06e1a58d0 2024-04-27T03:19:59.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7fdbec00f620 con 0x55d06e1a58d0 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc07198980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc07198980 20 searching for the correct realm 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d06e1aedd0 0x55d06e1b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1aedd0 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d06e1aedd0 0x55d06e1b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc04cb3640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d06e1aedd0 0x55d06e1b11c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc054b4640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fdbfc035070 0x7fdbfc019810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x7fdbfc035070 2024-04-27T03:19:59.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fdbfc035070 0x7fdbfc019810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fdbfc035070 0x7fdbfc019810 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec00f620 con 0x7fdbfc035070 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec00f620 con 0x55d06e1a58d0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1aedd0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.354+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x7fdbfc035070 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x7fdbfc035070 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x7fdbfc035070 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7fdbec02b040 con 0x7fdbfc035070 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1aedd0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.358+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1aedd0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x7fdbfc035070 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7fdbec02b040 con 0x7fdbfc035070 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1aedd0 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x7fdbfc035070 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x7fdbfc035070 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a58d0 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec02b040 con 0x55d06e1a58d0 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x7fdbfc035070 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1aedd0 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x7fdbfc035070 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.362+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1aedd0 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b17d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:59.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ad9b0 con 0x55d06e1a58d0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ae250 con 0x55d06e1a9d90 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ad5f0 con 0x55d06e1a58d0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 zone default found 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 4 Realm: () 2024-04-27T03:19:59.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 zonegroup default 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b2100 con 0x7fdbfc035070 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b3ed0 con 0x55d06e1aedd0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.366+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b4290 con 0x55d06e1aedd0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 20 started sync module instance, tier type = 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b3370 con 0x55d06e1a9d90 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94353688835936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b2fd0 con 0x55d06e1a9d90 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b51c0 con 0x55d06e1a58d0 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94353688841488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b55e0 con 0x55d06e1a58d0 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.370+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b6090 con 0x55d06e1a9d90 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94353688841488] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94353688835936] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94353688839920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b5130 con 0x55d06e1a9d90 2024-04-27T03:19:59.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b6930 con 0x7fdbfc035070 2024-04-27T03:19:59.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94353688847488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b5130 con 0x7fdbfc035070 2024-04-27T03:19:59.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b73d0 con 0x55d06e1a9d90 2024-04-27T03:19:59.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.374+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94353688847488] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94353688839920] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94353688850112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b7390 con 0x55d06e1a9d90 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b77c0 con 0x55d06e1aedd0 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94353688845168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b8060 con 0x55d06e1aedd0 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b8a20 con 0x55d06e1aedd0 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94353688850112] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94353688845168] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94353688855920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b8dc0 con 0x55d06e1aedd0 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b9610 con 0x7fdbfc035070 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.378+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'273 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94353688858592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b9aa0 con 0x7fdbfc035070 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=0 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=1 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=2 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=3 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=4 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=5 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94353688855920] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=6 2024-04-27T03:19:59.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94353688858592] v31'274 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 add_watcher() i=7 2024-04-27T03:19:59.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:59.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b68a0 con 0x55d06e1a58d0 2024-04-27T03:19:59.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94353688848544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1b9f80 con 0x55d06e1a58d0 2024-04-27T03:19:59.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94353688848544] v31'273 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdabb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.382+0000 7fdc07198980 5 note: GC not initialized 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdaba7e4640 20 reqs_thread_entry: start 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 20 Filter name: none 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdab37fe640 20 reqs_thread_entry: start 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1e02f0 con 0x55d06e1a58d0 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~490 out=490b,getxattrs out=75b] v0'0 uv10 ondisk = 0) ==== 231+0+613 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 20 rados_obj.operate() r=0 bl.length=490 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:59.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~460 in=460b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1e6c40 con 0x55d06e1a58d0 2024-04-27T03:19:59.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~460] v31'11 uv11 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:59.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94353689044656 in=648b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1e2330 con 0x55d06e1a9d90 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94353688850112 notify 133143986238 ret 0) ==== 678+0+0 (crc 0 0 0) 0x55d06e19add0 con 0x55d06e1a9d90 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdadbfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986238 cookie 94353688850112 notifier 4898 bl.length()=636 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdadbfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdadbfff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdadbfff640 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7fdac8003eb0 con 0x55d06e1a9d90 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94353689044656 out=8b] v0'0 uv273 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.386+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94353689044656 notify 133143986238 ret 0) ==== 42+0+48 (crc 0 0 0) 0x55d06e19a9c0 con 0x55d06e1a9d90 2024-04-27T03:19:59.399 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:59.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.390+0000 7fdabb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.390+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94353688848544] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x55d06e1a58d0 2024-04-27T03:19:59.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.390+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94353688848544] v31'274 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.390+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94353688835936] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x55d06e1a9d90 2024-04-27T03:19:59.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94353688835936] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc07198980 20 remove_watcher() i=0 2024-04-27T03:19:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc07198980 2 removed watcher, disabling cache 2024-04-27T03:19:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94353688841488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x55d06e1a58d0 2024-04-27T03:19:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94353688841488] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x55d06e1a58d0 2024-04-27T03:19:59.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc07198980 20 remove_watcher() i=1 2024-04-27T03:19:59.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94353688839920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x55d06e1a9d90 2024-04-27T03:19:59.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94353688839920] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc07198980 20 remove_watcher() i=2 2024-04-27T03:19:59.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.394+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94353688847488] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x7fdbfc035070 2024-04-27T03:19:59.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94353688847488] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc07198980 20 remove_watcher() i=3 2024-04-27T03:19:59.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94353688850112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x55d06e1a9d90 2024-04-27T03:19:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc054b4640 1 -- 172.21.15.110:0/210221818 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94353688850112] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbfc0083f0 con 0x55d06e1a9d90 2024-04-27T03:19:59.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc07198980 20 remove_watcher() i=4 2024-04-27T03:19:59.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94353688845168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x55d06e1aedd0 2024-04-27T03:19:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94353688845168] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc07198980 20 remove_watcher() i=5 2024-04-27T03:19:59.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.398+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94353688855920] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x55d06e1aedd0 2024-04-27T03:19:59.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc04cb3640 1 -- 172.21.15.110:0/210221818 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94353688855920] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbf40be7b0 con 0x55d06e1aedd0 2024-04-27T03:19:59.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 20 remove_watcher() i=6 2024-04-27T03:19:59.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94353688858592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55d06e1ddff0 con 0x7fdbfc035070 2024-04-27T03:19:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94353688858592] v31'275 uv273 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7fdbec03e040 con 0x7fdbfc035070 2024-04-27T03:19:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 20 remove_watcher() i=7 2024-04-27T03:19:59.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e1a9d90 msgr2=0x55d06e1ac180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e1a9d90 0x55d06e1ac180 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e1a58d0 msgr2=0x55d06e1a7d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e1a58d0 0x55d06e1a7d90 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fdbfc035070 msgr2=0x7fdbfc019810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7fdbfc035070 0x7fdbfc019810 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d06e1aedd0 msgr2=0x55d06e1b11c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55d06e1aedd0 0x55d06e1b11c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdad00346a0 msgr2=0x7fdad0036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdad00346a0 0x7fdad0036b60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40bf010 tx=0x7fdbf40db000 comp rx=0 tx=0).stop 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc044b2640 1 -- 172.21.15.110:0/210221818 reap_dead start 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 msgr2=0x55d06e19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e19a1f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0487f0 tx=0x7fdbfc049860 comp rx=0 tx=0).stop 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 shutdown_connections 2024-04-27T03:19:59.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/210221818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e192c90 0x55d06e19a1f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 >> 172.21.15.110:0/210221818 conn(0x55d06e18efd0 msgr2=0x55d06e18c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 shutdown_connections 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/210221818 wait complete. 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdadc0346a0 msgr2=0x7fdadc036b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdadc0346a0 0x7fdadc036b60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55d06e175f30 tx=0x7fdbf40e3810 comp rx=0 tx=0).stop 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e185b00 msgr2=0x55d06e185f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e185b00 0x55d06e185f20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc00aab0 tx=0x7fdbfc00af70 comp rx=0 tx=0).stop 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 shutdown_connections 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdadc0346a0 0x7fdadc036b60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/2776562569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e185b00 0x55d06e185f20 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 >> 172.21.15.110:0/2776562569 conn(0x55d06e17a910 msgr2=0x55d06e17b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 shutdown_connections 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/2776562569 wait complete. 2024-04-27T03:19:59.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e172f00 msgr2=0x55d06e1752f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e172f00 0x55d06e1752f0 crc :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e078de0 msgr2=0x55d06e171b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e078de0 0x55d06e171b90 crc :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdaec034650 msgr2=0x7fdaec036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdaec034650 0x7fdaec036b10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdbec000f70 tx=0x7fdbec001380 comp rx=0 tx=0).stop 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 msgr2=0x55d06e16ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e16ad00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fdbf401b0d0 tx=0x7fdbf4017e30 comp rx=0 tx=0).stop 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 shutdown_connections 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55d06e172f00 0x55d06e1752f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55d06e078de0 0x55d06e171b90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fdaec034650 0x7fdaec036b10 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdbec000f70 tx=0x7fdbec001380 comp rx=0 tx=0).stop 2024-04-27T03:19:59.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 --2- 172.21.15.110:0/378135596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55d06e0784c0 0x55d06e16ad00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 >> 172.21.15.110:0/378135596 conn(0x55d06dfebba0 msgr2=0x55d06e15e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 shutdown_connections 2024-04-27T03:19:59.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.402+0000 7fdc07198980 1 -- 172.21.15.110:0/378135596 wait complete. 2024-04-27T03:19:59.423 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.423 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.423 ERROR:root:0 2024-04-27T03:19:59.423 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:59.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:59.424 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid ada --caps 'users=read, write' 2024-04-27T03:19:59.502 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:59.502 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:59.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb3ab7a980 1 Processor -- start 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb3ab7a980 1 -- start start 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb3ab7a980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5611af6d29d0 0x5611af6d4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb3ab7a980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5611af58b0d0 con 0x5611af6ce4c0 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb3ab7a980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5611af580430 con 0x5611af6d29d0 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37e94640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5611af6d29d0 0x5611af6d4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37e94640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5611af6d29d0 0x5611af6d4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52130/0 (socket says 172.21.15.110:52130) 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37e94640 1 -- 172.21.15.110:0/3298284965 learned_addr learned my addr 172.21.15.110:0/3298284965 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb38695640 1 --1- 172.21.15.110:0/3298284965 >> v1:172.21.15.77:6789/0 conn(0x5611af6ce4c0 0x5611af6ce8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:44638/0 (socket says 172.21.15.110:44638) 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37e94640 1 -- 172.21.15.110:0/3298284965 >> v1:172.21.15.77:6789/0 conn(0x5611af6ce4c0 legacy=0x5611af6ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37e94640 1 -- 172.21.15.110:0/3298284965 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611af579510 con 0x5611af6d29d0 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37e94640 1 --2- 172.21.15.110:0/3298284965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6d29d0 0x5611af6d4dc0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800a4e0 tx=0x7ffb2801b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f88376e51df8b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37693640 1 -- 172.21.15.110:0/3298284965 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb280aa540 con 0x5611af6d29d0 2024-04-27T03:19:59.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37693640 1 -- 172.21.15.110:0/3298284965 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb280aa6e0 con 0x5611af6d29d0 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.494+0000 7ffb37693640 1 -- 172.21.15.110:0/3298284965 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb280aaa40 con 0x5611af6d29d0 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.498+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3298284965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6d29d0 msgr2=0x5611af6d4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.498+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3298284965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6d29d0 0x5611af6d4dc0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800a4e0 tx=0x7ffb2801b090 comp rx=0 tx=0).stop 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.498+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3298284965 shutdown_connections 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.498+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3298284965 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6d29d0 0x5611af6d4dc0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.498+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3298284965 >> 172.21.15.110:0/3298284965 conn(0x5611af641ba0 msgr2=0x5611af641fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.498+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3298284965 shutdown_connections 2024-04-27T03:19:59.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.498+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3298284965 wait complete. 2024-04-27T03:19:59.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb3ab7a980 1 Processor -- start 2024-04-27T03:19:59.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb3ab7a980 1 -- start start 2024-04-27T03:19:59.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb3ab7a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7b9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb3ab7a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5611af5cf2d0 con 0x5611af6ce4c0 2024-04-27T03:19:59.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb38695640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7b9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb38695640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7b9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52136/0 (socket says 172.21.15.110:52136) 2024-04-27T03:19:59.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb38695640 1 -- 172.21.15.110:0/865528104 learned_addr learned my addr 172.21.15.110:0/865528104 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb38695640 1 -- 172.21.15.110:0/865528104 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611af555ae0 con 0x5611af6ce4c0 2024-04-27T03:19:59.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffb38695640 1 --2- 172.21.15.110:0/865528104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7b9610 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000b240 tx=0x7ffb3000b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5f31f3f61601e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffa2660c640 1 -- 172.21.15.110:0/865528104 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb3000e700 con 0x5611af6ce4c0 2024-04-27T03:19:59.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffa2660c640 1 -- 172.21.15.110:0/865528104 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb3000e8a0 con 0x5611af6ce4c0 2024-04-27T03:19:59.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.510+0000 7ffa2660c640 1 -- 172.21.15.110:0/865528104 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb3000eb80 con 0x5611af6ce4c0 2024-04-27T03:19:59.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/865528104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 msgr2=0x5611af7b9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/865528104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7b9610 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000b240 tx=0x7ffb3000b700 comp rx=0 tx=0).stop 2024-04-27T03:19:59.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/865528104 shutdown_connections 2024-04-27T03:19:59.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/865528104 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7b9610 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/865528104 >> 172.21.15.110:0/865528104 conn(0x5611af641ba0 msgr2=0x5611af7b51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/865528104 shutdown_connections 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/865528104 wait complete. 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 Processor -- start 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- start start 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7c0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5611af6d7df0 con 0x5611af6ce4c0 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb38695640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7c0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb38695640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7c0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52146/0 (socket says 172.21.15.110:52146) 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb38695640 1 -- 172.21.15.110:0/3345127540 learned_addr learned my addr 172.21.15.110:0/3345127540 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb38695640 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611af7c0b90 con 0x5611af6ce4c0 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb38695640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7c0650 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ffb30000fe0 tx=0x7ffb30000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffa24e09640 1 -- 172.21.15.110:0/3345127540 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb3000f9c0 con 0x5611af6ce4c0 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5611af7c0e20 con 0x5611af6ce4c0 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffa24e09640 1 -- 172.21.15.110:0/3345127540 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb3000fb80 con 0x5611af6ce4c0 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611af7c3aa0 con 0x5611af6ce4c0 2024-04-27T03:19:59.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffa24e09640 1 -- 172.21.15.110:0/3345127540 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb3002c060 con 0x5611af6ce4c0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffa24e09640 1 -- 172.21.15.110:0/3345127540 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ffb30017070 con 0x5611af6ce4c0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffa24e09640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa20034650 0x7ffa20036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffa24e09640 1 -- 172.21.15.110:0/3345127540 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7ffb30015020 con 0x5611af6ce4c0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb37e94640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa20034650 0x7ffa20036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af6d29d0 0x5611af7c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7c7b50 con 0x5611af6d29d0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.514+0000 7ffb38e96640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af6d29d0 0x5611af7c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb37e94640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa20034650 0x7ffa20036b10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ffb280ab000 tx=0x7ffb280a5000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38e96640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af6d29d0 0x5611af7c7580 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38e96640 1 -- 172.21.15.110:0/3345127540 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ffb200083f0 con 0x5611af6d29d0 2024-04-27T03:19:59.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7c8920 0x5611af7cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7cb350 con 0x5611af7c8920 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38695640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7c8920 0x5611af7cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38695640 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7c8920 0x5611af7cad10 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38695640 1 -- 172.21.15.110:0/3345127540 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af7c8920 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7ccc00 con 0x5611af6d29d0 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38e96640 1 -- 172.21.15.110:0/3345127540 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7ffb200083f0 con 0x5611af6d29d0 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7cbe30 con 0x5611af7c8920 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38695640 1 -- 172.21.15.110:0/3345127540 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af7c8920 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7cc220 con 0x5611af6d29d0 2024-04-27T03:19:59.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb38e96640 1 -- 172.21.15.110:0/3345127540 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7ffb200083f0 con 0x5611af6d29d0 2024-04-27T03:19:59.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb3ab7a980 1 Processor -- start 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.518+0000 7ffb3ab7a980 1 -- start start 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7db540 0x5611af7db960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5611af5d3310 con 0x5611af7db540 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7db540 0x5611af7db960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7db540 0x5611af7db960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52148/0 (socket says 172.21.15.110:52148) 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 -- 172.21.15.110:0/2387028940 learned_addr learned my addr 172.21.15.110:0/2387028940 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 -- 172.21.15.110:0/2387028940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611af7dbea0 con 0x5611af7db540 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7db540 0x5611af7db960 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000aab0 tx=0x7ffb2000af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffa1dffb640 1 -- 172.21.15.110:0/2387028940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb20007d30 con 0x5611af7db540 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffa1dffb640 1 -- 172.21.15.110:0/2387028940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb2001d070 con 0x5611af7db540 2024-04-27T03:19:59.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5611af7dc940 con 0x5611af7db540 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffa1dffb640 1 -- 172.21.15.110:0/2387028940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb20018440 con 0x5611af7db540 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611af7dce10 con 0x5611af7db540 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffa1dffb640 1 -- 172.21.15.110:0/2387028940 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ffb20013070 con 0x5611af7db540 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffa1dffb640 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa10034650 0x7ffa10036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffa1dffb640 1 -- 172.21.15.110:0/2387028940 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7ffb2004b720 con 0x5611af7db540 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38695640 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa10034650 0x7ffa10036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 Processor -- start 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 -- start start 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38695640 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa10034650 0x7ffa10036b10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x5611af7cb950 tx=0x7ffb30001660 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7e8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb3ab7a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5611af7b9d60 con 0x5611af7e8710 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7e8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7e8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52158/0 (socket says 172.21.15.110:52158) 2024-04-27T03:19:59.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.522+0000 7ffb38e96640 1 -- 172.21.15.110:0/17891508 learned_addr learned my addr 172.21.15.110:0/17891508 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb38e96640 1 -- 172.21.15.110:0/17891508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611af7e9100 con 0x5611af7e8710 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb38e96640 1 --2- 172.21.15.110:0/17891508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7e8b30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ffb20049930 tx=0x7ffb20049b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea60710919d8812 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffa177fe640 1 -- 172.21.15.110:0/17891508 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb20047070 con 0x5611af7e8710 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffa177fe640 1 -- 172.21.15.110:0/17891508 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb20022460 con 0x5611af7e8710 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffa177fe640 1 -- 172.21.15.110:0/17891508 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb20018bc0 con 0x5611af7e8710 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/17891508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 msgr2=0x5611af7e8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/17891508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7e8b30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ffb20049930 tx=0x7ffb20049b30 comp rx=0 tx=0).stop 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/17891508 shutdown_connections 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/17891508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7e8b30 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/17891508 >> 172.21.15.110:0/17891508 conn(0x5611af7e4a50 msgr2=0x5611af7e2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/17891508 shutdown_connections 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/17891508 wait complete. 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 Processor -- start 2024-04-27T03:19:59.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- start start 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7efbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5611af7e9860 con 0x5611af7e8710 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb38e96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7efbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb38e96640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7efbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52174/0 (socket says 172.21.15.110:52174) 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 learned_addr learned my addr 172.21.15.110:0/4104463367 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5611af7f0130 con 0x5611af7e8710 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb38e96640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7efbf0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ffb20049930 tx=0x7ffb200498c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffa15ffb640 1 -- 172.21.15.110:0/4104463367 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb20031070 con 0x5611af7e8710 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5611af7f03c0 con 0x5611af7e8710 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffa15ffb640 1 -- 172.21.15.110:0/4104463367 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb20020900 con 0x5611af7e8710 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffa15ffb640 1 -- 172.21.15.110:0/4104463367 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb20017b70 con 0x5611af7e8710 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.526+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5611af7f30a0 con 0x5611af7e8710 2024-04-27T03:19:59.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffa15ffb640 1 -- 172.21.15.110:0/4104463367 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ffb20034070 con 0x5611af7e8710 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffa15ffb640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa08034650 0x7ffa08036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffa15ffb640 1 -- 172.21.15.110:0/4104463367 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7ffb20037dd0 con 0x5611af7e8710 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb38695640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa08034650 0x7ffa08036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af7fb2a0 0x5611af7fd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7fdeb0 con 0x5611af7fb2a0 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb37e94640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af7fb2a0 0x5611af7fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb38695640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa08034650 0x7ffa08036b10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ffb30029000 tx=0x7ffb3003e000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb37e94640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af7fb2a0 0x5611af7fd760 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 realm 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7fe280 con 0x5611af7fb2a0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af7feb50 con 0x5611af7fb2a0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7ff570 0x5611af8019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af802110 con 0x5611af7ff570 2024-04-27T03:19:59.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb38e96640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7ff570 0x5611af8019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb38e96640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7ff570 0x5611af8019a0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af802960 con 0x5611af7fb2a0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 20 searching for the correct realm 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5611af8046d0 0x5611af806ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.530+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af8046d0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38695640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5611af8046d0 0x5611af806ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38695640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5611af8046d0 0x5611af806ac0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38e96640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ffb20036070 0x7ffb2003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x7ffb20036070 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ffb20036070 0x7ffb2003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ffb20036070 0x7ffb2003b620 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af8046d0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.534+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x7ffb20036070 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x7ffb20036070 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af8046d0 2024-04-27T03:19:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af8046d0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x7ffb20036070 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.538+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af8046d0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x7ffb20036070 2024-04-27T03:19:59.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7fb2a0 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x7ffb20036070 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af8046d0 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x7ffb20036070 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af8046d0 2024-04-27T03:19:59.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.542+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5611af807140 con 0x5611af7ff570 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af8032b0 con 0x5611af7fb2a0 2024-04-27T03:19:59.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af802ef0 con 0x5611af7ff570 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af803c10 con 0x5611af7fb2a0 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 zone default found 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 4 Realm: () 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:59.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 zonegroup default 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af807a50 con 0x7ffb20036070 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af809cd0 con 0x5611af8046d0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af80a070 con 0x5611af8046d0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 started sync module instance, tier type = 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.546+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af809110 con 0x5611af7ff570 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94633958872368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80a8c0 con 0x5611af7ff570 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80b110 con 0x5611af7fb2a0 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94633958877376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80b190 con 0x5611af7fb2a0 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80bc90 con 0x5611af7ff570 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94633958872368] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.550+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94633958877376] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94633958881152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80ace0 con 0x5611af7ff570 2024-04-27T03:19:59.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80c530 con 0x7ffb20036070 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94633958883456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80ace0 con 0x7ffb20036070 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80cfb0 con 0x5611af7ff570 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94633958881152] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94633958883456] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94633958884512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80cac0 con 0x5611af7ff570 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.554+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80d850 con 0x5611af8046d0 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94633958888352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80cac0 con 0x5611af8046d0 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80e6c0 con 0x5611af8046d0 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94633958884512] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94633958888352] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94633958876176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80e6c0 con 0x5611af8046d0 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80ef10 con 0x7ffb20036070 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94633958876176] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'276 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94633958892528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80edc0 con 0x7ffb20036070 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=0 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=1 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=2 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=3 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=4 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=5 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=6 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94633958892528] v31'277 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 20 add_watcher() i=7 2024-04-27T03:19:59.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:59.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af812300 con 0x5611af7fb2a0 2024-04-27T03:19:59.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.558+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94633958893648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af80f7c0 con 0x5611af7fb2a0 2024-04-27T03:19:59.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94633958893648] v31'275 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ff9eefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:59.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:59.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 5 note: GC not initialized 2024-04-27T03:19:59.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ff9edfe3640 20 reqs_thread_entry: start 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 20 Filter name: none 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ff9e6ffd640 20 reqs_thread_entry: start 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af835c00 con 0x5611af7fb2a0 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~460 out=460b,getxattrs out=75b] v0'0 uv11 ondisk = 0) ==== 231+0+583 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 20 rados_obj.operate() r=0 bl.length=460 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.562+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~473 in=473b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af83c170 con 0x5611af7fb2a0 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~473] v31'12 uv12 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb3ab7a980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb3ab7a980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb3ab7a980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94633959078880 in=661b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5611af837880 con 0x5611af7ff570 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94633958884512 notify 133143986239 ret 0) ==== 691+0+0 (crc 0 0 0) 0x5611af7f30a0 con 0x5611af7ff570 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffa177fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986239 cookie 94633958884512 notifier 4905 bl.length()=649 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94633959078880 out=8b] v0'0 uv276 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffa177fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffa177fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffa177fe640 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7ff9fc003eb0 con 0x5611af7ff570 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== watch-notify(notify_complete (2) cookie 94633959078880 notify 133143986239 ret 0) ==== 42+0+48 (crc 0 0 0) 0x7ffb20063020 con 0x5611af7ff570 2024-04-27T03:19:59.595 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"users","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ff9eefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.566+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94633958893648] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af83b4c0 con 0x5611af7fb2a0 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94633958893648] v31'276 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94633958872368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af83b4c0 con 0x5611af7ff570 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94633958872368] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb3ab7a980 20 remove_watcher() i=0 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb3ab7a980 2 removed watcher, disabling cache 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94633958877376] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af83b4c0 con 0x5611af7fb2a0 2024-04-27T03:19:59.600 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94633958877376] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x5611af7fb2a0 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb3ab7a980 20 remove_watcher() i=1 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.570+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94633958881152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af83b4c0 con 0x5611af7ff570 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.574+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94633958881152] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.574+0000 7ffb3ab7a980 20 remove_watcher() i=2 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.574+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94633958883456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af83b4c0 con 0x7ffb20036070 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.574+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94633958883456] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.574+0000 7ffb3ab7a980 20 remove_watcher() i=3 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.574+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94633958884512] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af833900 con 0x5611af7ff570 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb38e96640 1 -- 172.21.15.110:0/4104463367 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94633958884512] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb20033030 con 0x5611af7ff570 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 20 remove_watcher() i=4 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94633958888352] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af833900 con 0x5611af8046d0 2024-04-27T03:19:59.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94633958888352] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 20 remove_watcher() i=5 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94633958876176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af833900 con 0x5611af8046d0 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb38695640 1 -- 172.21.15.110:0/4104463367 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94633958876176] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb3002e070 con 0x5611af8046d0 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 20 remove_watcher() i=6 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94633958892528] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5611af833900 con 0x7ffb20036070 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94633958892528] v31'278 uv276 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb280aabd0 con 0x7ffb20036070 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 20 remove_watcher() i=7 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7ff570 msgr2=0x5611af8019a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.578+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7ff570 0x5611af8019a0 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af7fb2a0 msgr2=0x5611af7fd760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af7fb2a0 0x5611af7fd760 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ffb20036070 msgr2=0x7ffb2003b620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7ffb20036070 0x7ffb2003b620 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5611af8046d0 msgr2=0x5611af806ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5611af8046d0 0x5611af806ac0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa08034650 msgr2=0x7ffa08036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa08034650 0x7ffa08036b10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ffb30029000 tx=0x7ffb3003e000 comp rx=0 tx=0).stop 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb37e94640 1 -- 172.21.15.110:0/4104463367 reap_dead start 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 msgr2=0x5611af7efbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7efbf0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7ffb20049930 tx=0x7ffb200498c0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 shutdown_connections 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/4104463367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7e8710 0x5611af7efbf0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 >> 172.21.15.110:0/4104463367 conn(0x5611af7e4a50 msgr2=0x5611af7e20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 shutdown_connections 2024-04-27T03:19:59.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/4104463367 wait complete. 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa10034650 msgr2=0x7ffa10036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa10034650 0x7ffa10036b10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x5611af7cb950 tx=0x7ffb30001660 comp rx=0 tx=0).stop 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7db540 msgr2=0x5611af7db960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7db540 0x5611af7db960 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000aab0 tx=0x7ffb2000af70 comp rx=0 tx=0).stop 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 shutdown_connections 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa10034650 0x7ffa10036b10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/2387028940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af7db540 0x5611af7db960 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 >> 172.21.15.110:0/2387028940 conn(0x5611af7d0350 msgr2=0x5611af7d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 shutdown_connections 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/2387028940 wait complete. 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7c8920 msgr2=0x5611af7cad10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7c8920 0x5611af7cad10 crc :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af6d29d0 msgr2=0x5611af7c7580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af6d29d0 0x5611af7c7580 crc :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa20034650 msgr2=0x7ffa20036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa20034650 0x7ffa20036b10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ffb280ab000 tx=0x7ffb280a5000 comp rx=0 tx=0).stop 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 msgr2=0x5611af7c0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7c0650 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ffb30000fe0 tx=0x7ffb30000ee0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 shutdown_connections 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5611af7c8920 0x5611af7cad10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5611af6d29d0 0x5611af7c7580 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ffa20034650 0x7ffa20036b10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 --2- 172.21.15.110:0/3345127540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5611af6ce4c0 0x5611af7c0650 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 >> 172.21.15.110:0/3345127540 conn(0x5611af641ba0 msgr2=0x5611af7b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 shutdown_connections 2024-04-27T03:19:59.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.582+0000 7ffb3ab7a980 1 -- 172.21.15.110:0/3345127540 wait complete. 2024-04-27T03:19:59.606 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.606 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.606 ERROR:root:0 2024-04-27T03:19:59.606 INFO:tasks.util.rgw:rgwadmin: client.0 : ['caps', 'add', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:59.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'caps', 'add', '--uid', 'ada', '--caps', 'users=read, write'] 2024-04-27T03:19:59.607 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph caps add --uid ada --caps 'users=read, write' 2024-04-27T03:19:59.674 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:19:59.675 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:19:59.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e5b5b980 1 Processor -- start 2024-04-27T03:19:59.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e5b5b980 1 -- start start 2024-04-27T03:19:59.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e5b5b980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5640aed114c0 0x5640aed118a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e5b5b980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x5640aebce0d0 con 0x5640aed11de0 2024-04-27T03:19:59.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e5b5b980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x5640aebc3430 con 0x5640aed114c0 2024-04-27T03:19:59.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e3676640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5640aed114c0 0x5640aed118a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e3676640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x5640aed114c0 0x5640aed118a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52190/0 (socket says 172.21.15.110:52190) 2024-04-27T03:19:59.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e3676640 1 -- 172.21.15.110:0/200515131 learned_addr learned my addr 172.21.15.110:0/200515131 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e3676640 1 -- 172.21.15.110:0/200515131 >> v1:172.21.15.77:6789/0 conn(0x5640aed11de0 legacy=0x5640aed127a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.666+0000 7f89e3676640 1 -- 172.21.15.110:0/200515131 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640aebbc510 con 0x5640aed114c0 2024-04-27T03:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e3676640 1 --2- 172.21.15.110:0/200515131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aed118a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f89d400a5c0 tx=0x7f89d401b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d02c36ccd06504 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e2674640 1 -- 172.21.15.110:0/200515131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89d40a6070 con 0x5640aed114c0 2024-04-27T03:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e2674640 1 -- 172.21.15.110:0/200515131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d40af070 con 0x5640aed114c0 2024-04-27T03:19:59.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e2674640 1 -- 172.21.15.110:0/200515131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89d40aa5c0 con 0x5640aed114c0 2024-04-27T03:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e5b5b980 1 -- 172.21.15.110:0/200515131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 msgr2=0x5640aed118a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/200515131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aed118a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f89d400a5c0 tx=0x7f89d401b340 comp rx=0 tx=0).stop 2024-04-27T03:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e5b5b980 1 -- 172.21.15.110:0/200515131 shutdown_connections 2024-04-27T03:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/200515131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aed118a0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e5b5b980 1 -- 172.21.15.110:0/200515131 >> 172.21.15.110:0/200515131 conn(0x5640aec84ba0 msgr2=0x5640aec84fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e5b5b980 1 -- 172.21.15.110:0/200515131 shutdown_connections 2024-04-27T03:19:59.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.670+0000 7f89e5b5b980 1 -- 172.21.15.110:0/200515131 wait complete. 2024-04-27T03:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e5b5b980 1 Processor -- start 2024-04-27T03:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e5b5b980 1 -- start start 2024-04-27T03:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e5b5b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aedfcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e5b5b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640aec122d0 con 0x5640aed114c0 2024-04-27T03:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e3676640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aedfcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e3676640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aedfcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52194/0 (socket says 172.21.15.110:52194) 2024-04-27T03:19:59.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e3676640 1 -- 172.21.15.110:0/3025007688 learned_addr learned my addr 172.21.15.110:0/3025007688 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e3676640 1 -- 172.21.15.110:0/3025007688 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640aeb98ae0 con 0x5640aed114c0 2024-04-27T03:19:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.682+0000 7f89e3676640 1 --2- 172.21.15.110:0/3025007688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aedfcd40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f89d40199f0 tx=0x7f89d4003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb27484133573499 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f88d160a640 1 -- 172.21.15.110:0/3025007688 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89d40aaa00 con 0x5640aed114c0 2024-04-27T03:19:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f88d160a640 1 -- 172.21.15.110:0/3025007688 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d40aaba0 con 0x5640aed114c0 2024-04-27T03:19:59.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f88d160a640 1 -- 172.21.15.110:0/3025007688 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89d40bd3f0 con 0x5640aed114c0 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3025007688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 msgr2=0x5640aedfcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3025007688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aedfcd40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f89d40199f0 tx=0x7f89d4003030 comp rx=0 tx=0).stop 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3025007688 shutdown_connections 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3025007688 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aedfcd40 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3025007688 >> 172.21.15.110:0/3025007688 conn(0x5640aec84ba0 msgr2=0x5640aedf88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3025007688 shutdown_connections 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3025007688 wait complete. 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 Processor -- start 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- start start 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aee03cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640aed1b510 con 0x5640aed114c0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e3676640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aee03cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e3676640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aee03cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52200/0 (socket says 172.21.15.110:52200) 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e3676640 1 -- 172.21.15.110:0/1527443400 learned_addr learned my addr 172.21.15.110:0/1527443400 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e3676640 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640aee04200 con 0x5640aed114c0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e3676640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aee03cc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f89d401b170 tx=0x7f89d40180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f88cfe07640 1 -- 172.21.15.110:0/1527443400 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89d40bd7c0 con 0x5640aed114c0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f88cfe07640 1 -- 172.21.15.110:0/1527443400 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d40bd960 con 0x5640aed114c0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5640aee04490 con 0x5640aed114c0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f88cfe07640 1 -- 172.21.15.110:0/1527443400 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89d40bdc40 con 0x5640aed114c0 2024-04-27T03:19:59.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.686+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640aee07070 con 0x5640aed114c0 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f88cfe07640 1 -- 172.21.15.110:0/1527443400 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f89d40ce020 con 0x5640aed114c0 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f88cfe07640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88c8034650 0x7f88c8036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f88cfe07640 1 -- 172.21.15.110:0/1527443400 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f89d40cc020 con 0x5640aed114c0 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e2e75640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88c8034650 0x7f88c8036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aed11de0 0x5640aee0ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee0b150 con 0x5640aed11de0 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3e77640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aed11de0 0x5640aee0ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e2e75640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88c8034650 0x7f88c8036b10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000f70 tx=0x7f89d8001420 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3e77640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aed11de0 0x5640aee0ab50 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3e77640 1 -- 172.21.15.110:0/1527443400 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f89dc00a4d0 con 0x5640aed11de0 2024-04-27T03:19:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee0bf20 0x5640aee0e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee0e970 con 0x5640aee0bf20 2024-04-27T03:19:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3676640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee0bf20 0x5640aee0e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3676640 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee0bf20 0x5640aee0e310 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3676640 1 -- 172.21.15.110:0/1527443400 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee0bf20 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee10220 con 0x5640aed11de0 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3e77640 1 -- 172.21.15.110:0/1527443400 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f89dc00a4d0 con 0x5640aed11de0 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee0f450 con 0x5640aee0bf20 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3676640 1 -- 172.21.15.110:0/1527443400 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee0bf20 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee0f840 con 0x5640aed11de0 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.690+0000 7f89e3e77640 1 -- 172.21.15.110:0/1527443400 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f89dc00a4d0 con 0x5640aed11de0 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e5b5b980 1 Processor -- start 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e5b5b980 1 -- start start 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e5b5b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee1eb90 0x5640aee1efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e5b5b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640aec16310 con 0x5640aee1eb90 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e3e77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee1eb90 0x5640aee1efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e3e77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee1eb90 0x5640aee1efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52214/0 (socket says 172.21.15.110:52214) 2024-04-27T03:19:59.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e3e77640 1 -- 172.21.15.110:0/268137072 learned_addr learned my addr 172.21.15.110:0/268137072 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e3e77640 1 -- 172.21.15.110:0/268137072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640aee1f4f0 con 0x5640aee1eb90 2024-04-27T03:19:59.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e3e77640 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee1eb90 0x5640aee1efb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00dfc0 tx=0x7f89dc0178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f88cd602640 1 -- 172.21.15.110:0/268137072 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc0157d0 con 0x5640aee1eb90 2024-04-27T03:19:59.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5640aee1ff90 con 0x5640aee1eb90 2024-04-27T03:19:59.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f88cd602640 1 -- 172.21.15.110:0/268137072 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89dc015970 con 0x5640aee1eb90 2024-04-27T03:19:59.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640aee20460 con 0x5640aee1eb90 2024-04-27T03:19:59.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.694+0000 7f88cd602640 1 -- 172.21.15.110:0/268137072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc015c50 con 0x5640aee1eb90 2024-04-27T03:19:59.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f88cd602640 1 -- 172.21.15.110:0/268137072 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f89dc020440 con 0x5640aee1eb90 2024-04-27T03:19:59.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f88cd602640 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88bc034640 0x7f88bc036b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f88cd602640 1 -- 172.21.15.110:0/268137072 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f89dc00e040 con 0x5640aee1eb90 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3676640 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88bc034640 0x7f88bc036b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 Processor -- start 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- start start 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee2c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640aedfd3d0 con 0x5640aee2bd90 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3676640 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88bc034640 0x7f88bc036b00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x5640aee0b9c0 tx=0x7f89d40e3820 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee2c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee2c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52218/0 (socket says 172.21.15.110:52218) 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 -- 172.21.15.110:0/2940622984 learned_addr learned my addr 172.21.15.110:0/2940622984 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 -- 172.21.15.110:0/2940622984 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640aee2c780 con 0x5640aee2bd90 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 --2- 172.21.15.110:0/2940622984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee2c1b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f89dc04aaa0 tx=0x7f89dc013870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5dec4d23151760 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f88c6ffd640 1 -- 172.21.15.110:0/2940622984 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc041070 con 0x5640aee2bd90 2024-04-27T03:19:59.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f88c6ffd640 1 -- 172.21.15.110:0/2940622984 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89dc020b50 con 0x5640aee2bd90 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f88c6ffd640 1 -- 172.21.15.110:0/2940622984 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc011cc0 con 0x5640aee2bd90 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- 172.21.15.110:0/2940622984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 msgr2=0x5640aee2c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/2940622984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee2c1b0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f89dc04aaa0 tx=0x7f89dc013870 comp rx=0 tx=0).stop 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- 172.21.15.110:0/2940622984 shutdown_connections 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/2940622984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee2c1b0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- 172.21.15.110:0/2940622984 >> 172.21.15.110:0/2940622984 conn(0x5640aee280d0 msgr2=0x5640aee26130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- 172.21.15.110:0/2940622984 shutdown_connections 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- 172.21.15.110:0/2940622984 wait complete. 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 Processor -- start 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- start start 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee331c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e5b5b980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x5640aee2cdf0 con 0x5640aee2bd90 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee331c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:52220/0 (socket says 172.21.15.110:52220) 2024-04-27T03:19:59.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.698+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 learned_addr learned my addr 172.21.15.110:0/3652795658 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640aee33700 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e3e77640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee331c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f89dc046040 tx=0x7f89dc04aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f88c57fa640 1 -- 172.21.15.110:0/3652795658 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc038070 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5640aee33990 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f88c57fa640 1 -- 172.21.15.110:0/3652795658 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89dc011730 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f88c57fa640 1 -- 172.21.15.110:0/3652795658 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89dc041040 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640aee36670 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f88c57fa640 1 -- 172.21.15.110:0/3652795658 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f89dc0118e0 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f88c57fa640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88ac034650 0x7f88ac036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f88c57fa640 1 -- 172.21.15.110:0/3652795658 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f89dc04b030 con 0x5640aee2bd90 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e3676640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88ac034650 0x7f88ac036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aee3e870 0x5640aee40d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee41480 con 0x5640aee3e870 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e2e75640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aee3e870 0x5640aee40d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e3676640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88ac034650 0x7f88ac036b10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x5640aee081b0 tx=0x7f89d40ed000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e2e75640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aee3e870 0x5640aee40d30 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f89d800d040 con 0x5640aee3e870 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 realm 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee3dcc0 con 0x5640aee3e870 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f89d800d040 con 0x5640aee3e870 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee41f80 con 0x5640aee3e870 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f89d800d040 con 0x5640aee3e870 2024-04-27T03:19:59.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee42d60 0x5640aee45150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee45820 con 0x5640aee42d60 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.702+0000 7f89e3e77640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee42d60 0x5640aee45150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3e77640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee42d60 0x5640aee45150 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e5b5b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee46070 con 0x5640aee3e870 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f89d800d040 con 0x5640aee3e870 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e5b5b980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e5b5b980 20 searching for the correct realm 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5640aee47de0 0x5640aee4a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee47de0 2024-04-27T03:19:59.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3676640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5640aee47de0 0x5640aee4a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3676640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5640aee47de0 0x5640aee4a1d0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3e77640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f89dc0133b0 0x7f89dc013a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:19:59.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x7f89dc0133b0 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e2e75640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f89dc0133b0 0x7f89dc013a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e2e75640 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f89dc0133b0 0x7f89dc013a70 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d800d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.706+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d800d040 con 0x5640aee3e870 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee47de0 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x7f89dc0133b0 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x7f89dc0133b0 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x7f89dc0133b0 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f89d802a040 con 0x7f89dc0133b0 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee47de0 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.710+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee47de0 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x7f89dc0133b0 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f89d802a040 con 0x7f89dc0133b0 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee47de0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x7f89dc0133b0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x7f89dc0133b0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee3e870 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d802a040 con 0x5640aee3e870 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x7f89dc0133b0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee47de0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x7f89dc0133b0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.714+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee47de0 2024-04-27T03:19:59.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4a850 con 0x5640aee42d60 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee469c0 con 0x5640aee3e870 2024-04-27T03:19:59.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee465d0 con 0x5640aee42d60 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee472b0 con 0x5640aee3e870 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 zone default found 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 4 Realm: () 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:19:59.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 zonegroup default 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4acf0 con 0x7f89dc0133b0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 165+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4cb10 con 0x5640aee47de0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.718+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4ced0 con 0x5640aee47de0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 20 started sync module instance, tier type = 2024-04-27T03:19:59.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:19:59.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4bfb0 con 0x5640aee42d60 2024-04-27T03:19:59.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94835812123824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4bfb0 con 0x5640aee42d60 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4da00 con 0x5640aee3e870 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94835812127952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4d9c0 con 0x5640aee3e870 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4e900 con 0x5640aee42d60 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.722+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94835812123824] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94835812127952] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94835812130288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4e900 con 0x5640aee42d60 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4ed50 con 0x7f89dc0133b0 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94835812121344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4f1c0 con 0x7f89dc0133b0 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4fc30 con 0x5640aee42d60 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94835812121344] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.726+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94835812130288] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94835812136704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4ed10 con 0x5640aee42d60 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee504d0 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94835812139040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee4ed10 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee51340 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94835812136704] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94835812139040] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94835812141168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee50e90 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee51850 con 0x7f89dc0133b0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94835812141168] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'279 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94835812131856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee51850 con 0x7f89dc0133b0 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 20 add_watcher() i=0 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 20 add_watcher() i=1 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 20 add_watcher() i=2 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 20 add_watcher() i=3 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 20 add_watcher() i=4 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 20 add_watcher() i=5 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.730+0000 7f89e5b5b980 20 add_watcher() i=6 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94835812131856] v31'280 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 20 add_watcher() i=7 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 2 all 8 watchers are set, enabling cache 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee54b50 con 0x5640aee3e870 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94835812144400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee51de0 con 0x5640aee3e870 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94835812144400] v31'277 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f8899fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 5 note: GC not initialized 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f8898fe1640 20 reqs_thread_entry: start 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 20 init_complete bucket index max shards: 11 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 20 Filter name: none 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f88127fc640 20 reqs_thread_entry: start 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 10 cache get: name=default.rgw.meta+users.uid+ada : miss 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 20 rados->read ofs=0 len=0 2024-04-27T03:19:59.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.734+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:62 8.7 8:e1d59a8c:users.uid::ada:head [call version.read in=11b,read 0~0,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee78450 con 0x5640aee3e870 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(62 ada [call out=48b,read 0~473 out=473b,getxattrs out=75b] v0'0 uv12 ondisk = 0) ==== 231+0+596 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 20 rados_obj.operate() r=0 bl.length=473 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x13 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 10 adding default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:63 8.7 8:e1d59a8c:users.uid::ada:head [delete,create,call version.check_conds in=122b,call version.set in=58b,writefull 0~473 in=473b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7ed70 con 0x5640aee3e870 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(63 ada [delete,create,call,call,writefull 0~473] v31'13 uv13 ondisk = 0) ==== 315+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 10 cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 10 moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ada, ofs0, ns] 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:64 7.d 7:b4812045:::notify.4:head [notify cookie 94835812329440 in=661b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7a460 con 0x5640aee42d60 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== watch-notify(notify (1) cookie 94835812136704 notify 133143986240 ret 0) ==== 691+0+0 (crc 0 0 0) 0x5640aee36670 con 0x5640aee42d60 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f88c6ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 133143986240 cookie 94835812136704 notifier 4912 bl.length()=649 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f88c6ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ada info.flags=0x17 2024-04-27T03:19:59.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f88c6ffd640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ada to cache LRU end 2024-04-27T03:19:59.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(64 notify.4 [notify cookie 94835812329440 out=8b] v0'0 uv279 ondisk = 0) ==== 152+0+8 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f88c6ffd640 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:65 7.d 7:b4812045:::notify.4:head [notify-ack in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x7f88a8003eb0 con 0x5640aee42d60 2024-04-27T03:19:59.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== watch-notify(notify_complete (2) cookie 94835812329440 notify 133143986240 ret 0) ==== 42+0+48 (crc 0 0 0) 0x5640aee33990 con 0x5640aee42d60 2024-04-27T03:19:59.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(65 notify.4 [notify-ack] v0'0 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.753 INFO:teuthology.orchestra.run.smithi110.stdout:{"user_id":"ada","display_name":"Ms. Admin User","email":"","suspended":0,"max_buckets":0,"subusers":[],"keys":[{"user":"ada","access_key":"MH1WC2XQ1S8UISFDZC8W","secret_key":"dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG","active":true,"create_date":"2024-04-27T03:19:57.242378Z"}],"swift_keys":[],"caps":[{"type":"buckets","perm":"*"},{"type":"info","perm":"read"},{"type":"ratelimit","perm":"*"},{"type":"usage","perm":"*"},{"type":"users","perm":"*"},{"type":"zone","perm":"*"}],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[],"account_id":"","path":"/","create_date":"2024-04-27T03:19:57.242366Z","tags":[],"group_ids":[]} 2024-04-27T03:19:59.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f8899fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:19:59.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.738+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:66 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94835812144400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7e080 con 0x5640aee3e870 2024-04-27T03:19:59.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.742+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(66 data_loggenerations_metadata [watch unwatch cookie 94835812144400] v31'278 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.742+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:67 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94835812123824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7e080 con 0x5640aee42d60 2024-04-27T03:19:59.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.742+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(67 notify.0 [watch unwatch cookie 94835812123824] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.742+0000 7f89e5b5b980 20 remove_watcher() i=0 2024-04-27T03:19:59.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.742+0000 7f89e5b5b980 2 removed watcher, disabling cache 2024-04-27T03:19:59.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.742+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:68 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94835812127952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7e080 con 0x5640aee3e870 2024-04-27T03:19:59.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(68 notify.1 [watch unwatch cookie 94835812127952] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x5640aee3e870 2024-04-27T03:19:59.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e5b5b980 20 remove_watcher() i=1 2024-04-27T03:19:59.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:69 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94835812130288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7e080 con 0x5640aee42d60 2024-04-27T03:19:59.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(69 notify.2 [watch unwatch cookie 94835812130288] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e5b5b980 20 remove_watcher() i=2 2024-04-27T03:19:59.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:70 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94835812121344] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7e080 con 0x7f89dc0133b0 2024-04-27T03:19:59.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(70 notify.3 [watch unwatch cookie 94835812121344] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e5b5b980 20 remove_watcher() i=3 2024-04-27T03:19:59.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.746+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94835812136704] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee7e080 con 0x5640aee42d60 2024-04-27T03:19:59.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e3e77640 1 -- 172.21.15.110:0/3652795658 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(71 notify.4 [watch unwatch cookie 94835812136704] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89dc044040 con 0x5640aee42d60 2024-04-27T03:19:59.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e5b5b980 20 remove_watcher() i=4 2024-04-27T03:19:59.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:72 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94835812139040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee76150 con 0x5640aee47de0 2024-04-27T03:19:59.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(72 notify.5 [watch unwatch cookie 94835812139040] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e5b5b980 20 remove_watcher() i=5 2024-04-27T03:19:59.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94835812141168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee76150 con 0x5640aee47de0 2024-04-27T03:19:59.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e3676640 1 -- 172.21.15.110:0/3652795658 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(73 notify.6 [watch unwatch cookie 94835812141168] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d40c6cd0 con 0x5640aee47de0 2024-04-27T03:19:59.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e5b5b980 20 remove_watcher() i=6 2024-04-27T03:19:59.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.750+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:74 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94835812131856] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x5640aee76150 con 0x7f89dc0133b0 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(74 notify.7 [watch unwatch cookie 94835812131856] v31'281 uv279 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f89d803d040 con 0x7f89dc0133b0 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 20 remove_watcher() i=7 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee42d60 msgr2=0x5640aee45150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee42d60 0x5640aee45150 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aee3e870 msgr2=0x5640aee40d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aee3e870 0x5640aee40d30 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f89dc0133b0 msgr2=0x7f89dc013a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f89dc0133b0 0x7f89dc013a70 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5640aee47de0 msgr2=0x5640aee4a1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x5640aee47de0 0x5640aee4a1d0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88ac034650 msgr2=0x7f88ac036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88ac034650 0x7f88ac036b10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x5640aee081b0 tx=0x7f89d40ed000 comp rx=0 tx=0).stop 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e2e75640 1 -- 172.21.15.110:0/3652795658 reap_dead start 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 msgr2=0x5640aee331c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee331c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f89dc046040 tx=0x7f89dc04aa30 comp rx=0 tx=0).stop 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 shutdown_connections 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/3652795658 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee2bd90 0x5640aee331c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 >> 172.21.15.110:0/3652795658 conn(0x5640aee280d0 msgr2=0x5640aee25730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 shutdown_connections 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/3652795658 wait complete. 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88bc034640 msgr2=0x7f88bc036b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88bc034640 0x7f88bc036b00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x5640aee0b9c0 tx=0x7f89d40e3820 comp rx=0 tx=0).stop 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee1eb90 msgr2=0x5640aee1efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee1eb90 0x5640aee1efb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00dfc0 tx=0x7f89dc0178e0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 shutdown_connections 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88bc034640 0x7f88bc036b00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/268137072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aee1eb90 0x5640aee1efb0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 >> 172.21.15.110:0/268137072 conn(0x5640aee139a0 msgr2=0x5640aee144b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 shutdown_connections 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/268137072 wait complete. 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee0bf20 msgr2=0x5640aee0e310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee0bf20 0x5640aee0e310 crc :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aed11de0 msgr2=0x5640aee0ab50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aed11de0 0x5640aee0ab50 crc :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88c8034650 msgr2=0x7f88c8036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88c8034650 0x7f88c8036b10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000f70 tx=0x7f89d8001420 comp rx=0 tx=0).stop 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 msgr2=0x5640aee03cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aee03cc0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f89d401b170 tx=0x7f89d40180e0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 shutdown_connections 2024-04-27T03:19:59.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x5640aee0bf20 0x5640aee0e310 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x5640aed11de0 0x5640aee0ab50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f88c8034650 0x7f88c8036b10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 --2- 172.21.15.110:0/1527443400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x5640aed114c0 0x5640aee03cc0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:19:59.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 >> 172.21.15.110:0/1527443400 conn(0x5640aec84ba0 msgr2=0x5640aedf73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:19:59.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 shutdown_connections 2024-04-27T03:19:59.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:19:59.754+0000 7f89e5b5b980 1 -- 172.21.15.110:0/1527443400 wait complete. 2024-04-27T03:19:59.774 DEBUG:tasks.util.rgw: json result: {'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.774 ERROR:root:{'user_id': 'ada', 'display_name': 'Ms. Admin User', 'email': '', 'suspended': 0, 'max_buckets': 0, 'subusers': [], 'keys': [{'user': 'ada', 'access_key': 'MH1WC2XQ1S8UISFDZC8W', 'secret_key': 'dQyrTPA0s248YeN5bBv4ukvKU0kh54LWWywkrpoG', 'active': True, 'create_date': '2024-04-27T03:19:57.242378Z'}], 'swift_keys': [], 'caps': [{'type': 'buckets', 'perm': '*'}, {'type': 'info', 'perm': 'read'}, {'type': 'ratelimit', 'perm': '*'}, {'type': 'usage', 'perm': '*'}, {'type': 'users', 'perm': '*'}, {'type': 'zone', 'perm': '*'}], 'op_mask': 'read, write, delete', 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': [], 'account_id': '', 'path': '/', 'create_date': '2024-04-27T03:19:57.242366Z', 'tags': [], 'group_ids': []} 2024-04-27T03:19:59.775 ERROR:root:0 2024-04-27T03:19:59.775 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': True} 2024-04-27T03:19:59.782 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.783 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.786 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.787 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': 'false'} 2024-04-27T03:19:59.794 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.795 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.799 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.799 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['key', 'create'] {'uid': 'foo', 'access-key': 'p5YnriCv1nAtykxBrupQ', 'secret-key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh'} 2024-04-27T03:19:59.809 INFO:tasks.radosgw_admin_rest: json result: [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True}] 2024-04-27T03:19:59.810 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.814 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}, {'user': 'foo', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.814 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['key', 'rm'] {'uid': 'foo', 'access-key': 'p5YnriCv1nAtykxBrupQ'} 2024-04-27T03:19:59.822 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:19:59.822 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.826 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.826 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'create'] {'subuser': 'foo:foo1', 'secret-key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'key-type': 'swift'} 2024-04-27T03:19:59.835 INFO:tasks.radosgw_admin_rest: json result: [{'id': 'foo:foo1', 'permissions': ''}] 2024-04-27T03:19:59.836 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.839 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'active': True}], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.839 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'create'] {'subuser': 'foo:foo2', 'secret-key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'key-type': 'swift'} 2024-04-27T03:19:59.849 INFO:tasks.radosgw_admin_rest: json result: [{'id': 'foo:foo1', 'permissions': ''}, {'id': 'foo:foo2', 'permissions': ''}] 2024-04-27T03:19:59.849 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.853 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': ''}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [{'user': 'foo:foo1', 'secret_key': 'gpS2G9RREMrnbqlp29PP2D36kgPR1tm72n5fPYfL', 'active': True}, {'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'active': True}], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.853 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['key', 'rm'] {'subuser': 'foo:foo1', 'key-type': 'swift'} 2024-04-27T03:19:59.862 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:19:59.862 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.866 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo1', 'permissions': ''}, {'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'active': True}], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.867 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'rm'] {'subuser': 'foo:foo1'} 2024-04-27T03:19:59.873 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:19:59.873 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.877 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [{'id': 'foo:foo2', 'permissions': ''}], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [{'user': 'foo:foo2', 'secret_key': 'ri2VJQcKSYATOY6uaDUX7pxgkW+W1YmC6OCxPHwy', 'active': True}], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.877 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['subuser', 'rm'] {'subuser': 'foo:foo2', 'key-type': 'swift', '{purge-keys': True} 2024-04-27T03:19:59.887 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:19:59.887 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.890 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:19:59.891 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.896 INFO:tasks.radosgw_admin_rest: json result: [] 2024-04-27T03:19:59.897 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo', 'stats': True} 2024-04-27T03:19:59.901 INFO:tasks.radosgw_admin_rest: json result: [] 2024-04-27T03:19:59.917 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo'} 2024-04-27T03:19:59.922 INFO:tasks.radosgw_admin_rest: json result: ['myfoo'] 2024-04-27T03:19:59.922 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'myfoo', 'stats': True} 2024-04-27T03:19:59.927 INFO:tasks.radosgw_admin_rest: json result: {'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:19:59.904640Z', 'creation_time': '2024-04-27T03:19:59.900451Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2024-04-27T03:19:59.927 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'uid': 'foo', 'stats': True} 2024-04-27T03:19:59.933 INFO:tasks.radosgw_admin_rest: json result: [{'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:19:59.904640Z', 'creation_time': '2024-04-27T03:19:59.900451Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}] 2024-04-27T03:19:59.939 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'myfoo', 'stats': True} 2024-04-27T03:19:59.945 INFO:tasks.radosgw_admin_rest: json result: {'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#2,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:19:59.904640Z', 'creation_time': '2024-04-27T03:19:59.900451Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 3, 'size_actual': 4096, 'size_utilized': 3, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2024-04-27T03:19:59.945 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'doesnotexist'} 2024-04-27T03:19:59.950 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchBucket', 'Message': '', 'RequestId': 'tx000000725a1445266b2a2-00662c6edf-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:19:59.956 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'unlink'] {'uid': 'foo', 'bucket': 'myfoo'} 2024-04-27T03:19:59.964 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:19:59.964 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'create'] {'uid': 'fud', 'display-name': 'Fud', 'access-key': 'p5YnriCv1nAtykxBrupQ', 'secret-key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'max-buckets': '1'} 2024-04-27T03:19:59.977 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'fud', 'display_name': 'Fud', 'email': '', 'suspended': 0, 'max_buckets': 1, 'subusers': [], 'keys': [{'user': 'fud', 'access_key': 'p5YnriCv1nAtykxBrupQ', 'secret_key': 'Q8Tk6Q/27hfbFSYdSkPtUqhqx1GgzvpXa4WARozh', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:20:00.032 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'link'] {'uid': 'fud', 'bucket': 'myfoo', 'bucket-id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11'} 2024-04-27T03:20:00.051 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:20:00.054 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'link'] {'uid': 'foo', 'bucket': 'myfoo', 'bucket-id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11'} 2024-04-27T03:20:00.072 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:20:00.073 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'fud'} 2024-04-27T03:20:00.089 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:20:00.104 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['object', 'rm'] {'bucket': 'myfoo', 'object': 'four'} 2024-04-27T03:20:00.118 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:20:00.119 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'info'] {'bucket': 'myfoo', 'stats': True} 2024-04-27T03:20:00.125 INFO:tasks.radosgw_admin_rest: json result: {'bucket': 'myfoo', 'tenant': '', 'versioning': 'off', 'zonegroup': '3fb181a9-2f1b-4a6e-85a1-ca8305b0483c', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'marker': '27a51273-3ccc-49e2-8682-1122abc3d898.4220.11', 'index_type': 'Normal', 'index_generation': 0, 'num_shards': 11, 'object_lock_enabled': False, 'mfa_enabled': False, 'owner': 'foo', 'ver': '0#1,1#1,2#1,3#1,4#1,5#3,6#3,7#1,8#3,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2024-04-27T03:20:00.056110Z', 'creation_time': '2024-04-27T03:19:59.900451Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}}, 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}} 2024-04-27T03:21:00.230 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'categories': 'delete_obj'} 2024-04-27T03:21:00.260 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'anonymous', 'categories': []}]}, {'user': 'bar', 'buckets': [{'bucket': 'myfoo6', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'bar', 'categories': []}]}, {'user': 'bud', 'buckets': [{'bucket': 'myfoo6', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'bud', 'categories': []}]}, {'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}]}, {'bucket': 'useless-bucket', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'fud', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'fud', 'categories': []}, {'bucket': 'myfoo7', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'fud', 'categories': []}]}], 'summary': [{'user': 'anonymous', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'bar', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'bud', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}, {'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}}, {'user': 'fud', 'categories': [], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 0, 'successful_ops': 0}}]} 2024-04-27T03:21:00.261 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] None 2024-04-27T03:21:00.286 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'anonymous', 'buckets': [{'bucket': '', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 204, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'bar', 'buckets': [{'bucket': 'myfoo6', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'bar', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 8, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'bud', 'buckets': [{'bucket': 'myfoo6', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'bud', 'categories': [{'category': 'get_obj', 'bytes_sent': 16, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}]}]}, {'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 10, 'ops': 3, 'successful_ops': 3}]}, {'bucket': 'useless-bucket', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 14, 'ops': 1, 'successful_ops': 1}]}]}, {'user': 'fud', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'fud', 'categories': [{'category': 'put_obj', 'bytes_sent': 234, 'bytes_received': 0, 'ops': 1, 'successful_ops': 0}]}, {'bucket': 'myfoo7', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'fud', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'get_obj', 'bytes_sent': 8, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 8, 'ops': 1, 'successful_ops': 1}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'anonymous', 'categories': [{'category': 'list_buckets', 'bytes_sent': 204, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 204, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}}, {'user': 'bar', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 8, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 0, 'bytes_received': 8, 'ops': 2, 'successful_ops': 2}}, {'user': 'bud', 'categories': [{'category': 'get_obj', 'bytes_sent': 16, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}], 'total': {'bytes_sent': 16, 'bytes_received': 0, 'ops': 4, 'successful_ops': 4}}, {'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 24, 'ops': 10, 'successful_ops': 10}}, {'user': 'fud', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'get_obj', 'bytes_sent': 8, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 234, 'bytes_received': 8, 'ops': 2, 'successful_ops': 1}, {'category': 'stat_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 242, 'bytes_received': 8, 'ops': 5, 'successful_ops': 4}}]} 2024-04-27T03:21:00.286 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo'} 2024-04-27T03:21:00.292 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 10, 'ops': 3, 'successful_ops': 3}]}, {'bucket': 'useless-bucket', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 14, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}, {'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}, {'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}, {'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 24, 'ops': 10, 'successful_ops': 10}}]} 2024-04-27T03:21:00.292 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'create_bucket'} 2024-04-27T03:21:00.297 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}, {'bucket': 'useless-bucket', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'create_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}}]} 2024-04-27T03:21:00.297 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'put_obj'} 2024-04-27T03:21:00.302 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 10, 'ops': 3, 'successful_ops': 3}]}, {'bucket': 'useless-bucket', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 14, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'put_obj', 'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}], 'total': {'bytes_sent': 0, 'bytes_received': 24, 'ops': 4, 'successful_ops': 4}}]} 2024-04-27T03:21:00.302 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'delete_obj'} 2024-04-27T03:21:00.306 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 2, 'successful_ops': 2}]}, {'bucket': 'useless-bucket', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_obj', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 3, 'successful_ops': 3}}]} 2024-04-27T03:21:00.307 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo', 'categories': 'delete_bucket'} 2024-04-27T03:21:00.311 INFO:tasks.radosgw_admin_rest: json result: {'entries': [{'user': 'foo', 'buckets': [{'bucket': 'myfoo', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': []}, {'bucket': 'useless-bucket', 'time': '2024-04-27T03:00:00.000000Z', 'epoch': 1714186800, 'owner': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}]}]}], 'summary': [{'user': 'foo', 'categories': [{'category': 'delete_bucket', 'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}], 'total': {'bytes_sent': 0, 'bytes_received': 0, 'ops': 1, 'successful_ops': 1}}]} 2024-04-27T03:21:00.312 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'trim'] {'uid': 'foo'} 2024-04-27T03:21:00.321 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:21:00.321 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['usage', 'show'] {'uid': 'foo'} 2024-04-27T03:21:00.326 INFO:tasks.radosgw_admin_rest: json result: {'entries': [], 'summary': []} 2024-04-27T03:21:00.326 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': True} 2024-04-27T03:21:00.341 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 1, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:21:00.346 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'modify'] {'uid': 'foo', 'suspended': 'false'} 2024-04-27T03:21:00.360 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'foo', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 4, 'subusers': [], 'keys': [{'user': 'foo', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:21:00.733 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'foo'} 2024-04-27T03:21:00.740 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'BucketAlreadyExists', 'Message': '', 'RequestId': 'tx000006ed60b0291833027-00662c6f1c-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:00.921 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['policy', 'show'] {'bucket': 'myfoo', 'object': b'bb3aec0fdcdbc2974890f805c585d432'} 2024-04-27T03:21:00.927 INFO:tasks.radosgw_admin_rest: json result: {'acl': {'acl_user_map': [{'user': 'foo', 'acl': 15}], 'acl_group_map': [], 'grant_map': [{'id': 'foo', 'grant': {'type': {'type': 0}, 'id': 'foo', 'name': 'Foo', 'permission': {'flags': 15}}}]}, 'owner': {'id': 'foo', 'display_name': 'Foo'}} 2024-04-27T03:21:00.939 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['policy', 'show'] {'bucket': 'myfoo', 'object': b'bb3aec0fdcdbc2974890f805c585d432'} 2024-04-27T03:21:00.944 INFO:tasks.radosgw_admin_rest: json result: {'acl': {'acl_user_map': [{'user': 'foo', 'acl': 15}], 'acl_group_map': [{'group': 1, 'acl': 1}], 'grant_map': [{'id': '', 'grant': {'type': {'type': 2}, 'group': 1, 'permission': {'flags': 1}}}, {'id': 'foo', 'grant': {'type': {'type': 0}, 'id': 'foo', 'name': 'Foo', 'permission': {'flags': 15}}}]}, 'owner': {'id': 'foo', 'display_name': 'Foo'}} 2024-04-27T03:21:01.086 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['bucket', 'rm'] {'bucket': 'myfoo', 'purge-objects': True} 2024-04-27T03:21:01.178 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:21:01.178 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['caps', 'add'] {'uid': 'foo', 'user-caps': 'usage=read'} 2024-04-27T03:21:01.188 INFO:tasks.radosgw_admin_rest: json result: [{'type': 'usage', 'perm': 'read'}] 2024-04-27T03:21:01.188 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['caps', 'rm'] {'uid': 'foo', 'user-caps': 'usage=read'} 2024-04-27T03:21:01.197 INFO:tasks.radosgw_admin_rest: json result: [] 2024-04-27T03:21:01.222 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'foo'} 2024-04-27T03:21:01.228 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'BucketAlreadyExists', 'Message': '', 'RequestId': 'tx00000220a4ec39d497539-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.273 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'rm'] {'uid': 'foo', 'purge-data': True} 2024-04-27T03:21:01.347 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:21:01.348 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'info'] {'uid': 'foo'} 2024-04-27T03:21:01.353 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchUser', 'Message': '', 'RequestId': 'tx0000033d574adb6697cc3-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.354 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['info', ''] None 2024-04-27T03:21:01.358 INFO:tasks.radosgw_admin_rest: json result: {'info': {'storage_backends': [{'name': 'rados', 'cluster_id': '3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3'}]}} 2024-04-27T03:21:01.359 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['user', 'create'] {'uid': 'ratelimit_user', 'display-name': 'Foo', 'email': 'foo@foo.com', 'access-key': '9te6NH5mcdcq0Tc5i8i1', 'secret-key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'max-buckets': '1000'} 2024-04-27T03:21:01.379 INFO:tasks.radosgw_admin_rest: json result: {'tenant': '', 'user_id': 'ratelimit_user', 'display_name': 'Foo', 'email': 'foo@foo.com', 'suspended': 0, 'max_buckets': 1000, 'subusers': [], 'keys': [{'user': 'ratelimit_user', 'access_key': '9te6NH5mcdcq0Tc5i8i1', 'secret_key': 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', 'active': True}], 'swift_keys': [], 'caps': [], 'op_mask': 'read, write, delete', 'system': False, 'admin': False, 'default_placement': '', 'default_storage_class': '', 'placement_tags': [], 'bucket_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'user_quota': {'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}, 'temp_url_keys': [], 'type': 'rgw', 'mfa_ids': []} 2024-04-27T03:21:01.379 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_user'} 2024-04-27T03:21:01.384 INFO:tasks.radosgw_admin_rest: json result: {'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}} 2024-04-27T03:21:01.384 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_userstring'} 2024-04-27T03:21:01.389 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'Message': '', 'RequestId': 'tx00000c20f52c36fcb0ce8-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.389 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user'} 2024-04-27T03:21:01.393 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'Message': '', 'RequestId': 'tx00000ddf980a2d6f60c4b-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.417 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucket'} 2024-04-27T03:21:01.421 INFO:tasks.radosgw_admin_rest: json result: {'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}} 2024-04-27T03:21:01.422 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucketstring'} 2024-04-27T03:21:01.427 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'Message': '', 'RequestId': 'tx000000068963e231d69fd-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.427 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket'} 2024-04-27T03:21:01.431 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'Message': '', 'RequestId': 'tx00000bee19a4e1f2f972d-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.432 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'global': 'true'} 2024-04-27T03:21:01.437 INFO:tasks.radosgw_admin_rest: json result: {'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}, 'anonymous_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 0, 'enabled': False}} 2024-04-27T03:21:01.437 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_userstring', 'enabled': 'true'} 2024-04-27T03:21:01.441 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'Message': '', 'RequestId': 'tx000007d0786557dec5db4-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.442 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'user'} 2024-04-27T03:21:01.446 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'Message': '', 'RequestId': 'tx0000028333961552be591-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.446 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucketstring', 'enabled': 'true'} 2024-04-27T03:21:01.450 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'NoSuchKey', 'Message': '', 'RequestId': 'tx0000035f9570db1a00eb5-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.451 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'enabled': 'true'} 2024-04-27T03:21:01.454 INFO:tasks.radosgw_admin_rest: json result: {'Code': 'InvalidArgument', 'Message': '', 'RequestId': 'tx00000f102ee762ae31975-00662c6f1d-4220-default', 'HostId': '4220-default-default'} 2024-04-27T03:21:01.455 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_user', 'enabled': 'true', 'max-read-bytes': '2'} 2024-04-27T03:21:01.472 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:21:01.473 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'user', 'uid': 'ratelimit_user'} 2024-04-27T03:21:01.477 INFO:tasks.radosgw_admin_rest: json result: {'user_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 2, 'max_write_bytes': 0, 'enabled': True}} 2024-04-27T03:21:01.477 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucket', 'enabled': 'true', 'max-write-bytes': '2'} 2024-04-27T03:21:01.487 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:21:01.487 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'info'] {'ratelimit-scope': 'bucket', 'bucket': 'ratelimitbucket'} 2024-04-27T03:21:01.491 INFO:tasks.radosgw_admin_rest: json result: {'bucket_ratelimit': {'max_read_ops': 0, 'max_write_ops': 0, 'max_read_bytes': 0, 'max_write_bytes': 2, 'enabled': True}} 2024-04-27T03:21:01.491 INFO:tasks.radosgw_admin_rest:radosgw-admin-rest: ['ratelimit', 'modify'] {'ratelimit-scope': 'bucket', 'global': 'true', 'enabled': 'true'} 2024-04-27T03:21:01.500 INFO:tasks.radosgw_admin_rest: empty result 2024-04-27T03:21:01.505 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-04-27T03:21:01.515 DEBUG:tasks.rgw.client.0:waiting for process to exit 2024-04-27T03:21:01.515 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T03:21:01.517 INFO:tasks.rgw.client.0.smithi110.stdout:2024-04-27T03:21:01.505+0000 7f14749cd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground (PID: 27713) UID: 0 2024-04-27T03:21:01.517 INFO:tasks.rgw.client.0.smithi110.stdout:2024-04-27T03:21:01.505+0000 7f1478215a40 -1 shutting down 2024-04-27T03:21:05.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2024-04-27T03:21:07.617 INFO:tasks.rgw.client.0:Stopped 2024-04-27T03:21:07.618 DEBUG:teuthology.orchestra.run.smithi110:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2024-04-27T03:21:07.624 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/vault-root-token 2024-04-27T03:21:07.679 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /home/ubuntu/cephtest/url_file 2024-04-27T03:21:07.735 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'process', '--include-all'] 2024-04-27T03:21:07.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'process', '--include-all'] 2024-04-27T03:21:07.735 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process --include-all 2024-04-27T03:21:07.814 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setuser ceph since I am not root 2024-04-27T03:21:07.814 INFO:teuthology.orchestra.run.smithi110.stderr:ignoring --setgroup ceph since I am not root 2024-04-27T03:21:07.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4309a69980 1 Processor -- start 2024-04-27T03:21:07.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4309a69980 1 -- start start 2024-04-27T03:21:07.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4309a69980 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f79435f150 0x55f79435f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4309a69980 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 26 bytes epoch 0) -- 0x55f79421f0d0 con 0x55f79435fa70 2024-04-27T03:21:07.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4309a69980 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x55f794214430 con 0x55f79435f150 2024-04-27T03:21:07.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4307584640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f79435f150 0x55f79435f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4307584640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x55f79435f150 0x55f79435f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:43970/0 (socket says 172.21.15.110:43970) 2024-04-27T03:21:07.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4306d83640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x55f79435fa70 0x55f794360480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.110:46378/0 (socket says 172.21.15.110:46378) 2024-04-27T03:21:07.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4307584640 1 -- 172.21.15.110:0/2941571672 learned_addr learned my addr 172.21.15.110:0/2941571672 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:21:07.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4307584640 1 -- 172.21.15.110:0/2941571672 >> v1:172.21.15.77:6789/0 conn(0x55f79435fa70 legacy=0x55f794360480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:07.816 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.805+0000 7f4307584640 1 -- 172.21.15.110:0/2941571672 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f79420d510 con 0x55f79435f150 2024-04-27T03:21:07.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4307584640 1 --2- 172.21.15.110:0/2941571672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79435f530 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f42f000a5c0 tx=0x7f42f001b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3b13b84e2ddbb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4306582640 1 -- 172.21.15.110:0/2941571672 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f00a6070 con 0x55f79435f150 2024-04-27T03:21:07.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4306582640 1 -- 172.21.15.110:0/2941571672 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f00af070 con 0x55f79435f150 2024-04-27T03:21:07.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4306582640 1 -- 172.21.15.110:0/2941571672 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f00aa5c0 con 0x55f79435f150 2024-04-27T03:21:07.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4309a69980 1 -- 172.21.15.110:0/2941571672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 msgr2=0x55f79435f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:07.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4309a69980 1 --2- 172.21.15.110:0/2941571672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79435f530 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f42f000a5c0 tx=0x7f42f001b340 comp rx=0 tx=0).stop 2024-04-27T03:21:07.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4309a69980 1 -- 172.21.15.110:0/2941571672 shutdown_connections 2024-04-27T03:21:07.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4309a69980 1 --2- 172.21.15.110:0/2941571672 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79435f530 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:07.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4309a69980 1 -- 172.21.15.110:0/2941571672 >> 172.21.15.110:0/2941571672 conn(0x55f79435aa20 msgr2=0x55f79435ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:07.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4309a69980 1 -- 172.21.15.110:0/2941571672 shutdown_connections 2024-04-27T03:21:07.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.809+0000 7f4309a69980 1 -- 172.21.15.110:0/2941571672 wait complete. 2024-04-27T03:21:07.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4309a69980 1 Processor -- start 2024-04-27T03:21:07.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4309a69980 1 -- start start 2024-04-27T03:21:07.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4309a69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79444aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4309a69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f7942632d0 con 0x55f79435f150 2024-04-27T03:21:07.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4307584640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79444aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4307584640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79444aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:43974/0 (socket says 172.21.15.110:43974) 2024-04-27T03:21:07.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4307584640 1 -- 172.21.15.110:0/865152647 learned_addr learned my addr 172.21.15.110:0/865152647 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:21:07.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4307584640 1 -- 172.21.15.110:0/865152647 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7941e9ae0 con 0x55f79435f150 2024-04-27T03:21:07.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f4307584640 1 --2- 172.21.15.110:0/865152647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79444aa10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f42f00199f0 tx=0x7f42f00a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4f236a8c03737b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f41f560a640 1 -- 172.21.15.110:0/865152647 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f00c3070 con 0x55f79435f150 2024-04-27T03:21:07.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f41f560a640 1 -- 172.21.15.110:0/865152647 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f00b4b20 con 0x55f79435f150 2024-04-27T03:21:07.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.825+0000 7f41f560a640 1 -- 172.21.15.110:0/865152647 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f00b4e00 con 0x55f79435f150 2024-04-27T03:21:07.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- 172.21.15.110:0/865152647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 msgr2=0x55f79444aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:07.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 --2- 172.21.15.110:0/865152647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79444aa10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f42f00199f0 tx=0x7f42f00a6000 comp rx=0 tx=0).stop 2024-04-27T03:21:07.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- 172.21.15.110:0/865152647 shutdown_connections 2024-04-27T03:21:07.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 --2- 172.21.15.110:0/865152647 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f79444aa10 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:07.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- 172.21.15.110:0/865152647 >> 172.21.15.110:0/865152647 conn(0x55f79435aa20 msgr2=0x55f7944465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:07.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- 172.21.15.110:0/865152647 shutdown_connections 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- 172.21.15.110:0/865152647 wait complete. 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 Processor -- start 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- start start 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f7944508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f7943691f0 con 0x55f79435f150 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4307584640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f7944508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4307584640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f7944508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:43976/0 (socket says 172.21.15.110:43976) 2024-04-27T03:21:07.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4307584640 1 -- 172.21.15.110:0/3782688643 learned_addr learned my addr 172.21.15.110:0/3782688643 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:21:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4307584640 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f794450de0 con 0x55f79435f150 2024-04-27T03:21:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4307584640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f7944508a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f42f001b170 tx=0x7f42f00180e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f41f3e07640 1 -- 172.21.15.110:0/3782688643 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f00c3040 con 0x55f79435f150 2024-04-27T03:21:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f41f3e07640 1 -- 172.21.15.110:0/3782688643 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f00b4cd0 con 0x55f79435f150 2024-04-27T03:21:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f794451070 con 0x55f79435f150 2024-04-27T03:21:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f41f3e07640 1 -- 172.21.15.110:0/3782688643 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f42f00a3f00 con 0x55f79435f150 2024-04-27T03:21:07.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f794451500 con 0x55f79435f150 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f41f3e07640 1 -- 172.21.15.110:0/3782688643 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f42f00d0060 con 0x55f79435f150 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.829+0000 7f41f3e07640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41ec034650 0x7f41ec036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f41f3e07640 1 -- 172.21.15.110:0/3782688643 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f42f00bb070 con 0x55f79435f150 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4306d83640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41ec034650 0x7f41ec036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79435fa70 0x55f794457720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794457d20 con 0x55f79435fa70 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4307d85640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79435fa70 0x55f794457720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4306d83640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41ec034650 0x7f41ec036b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f42f8000f70 tx=0x7f42f8001380 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4307d85640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79435fa70 0x55f794457720 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4307d85640 1 -- 172.21.15.110:0/3782688643 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f430000a4d0 con 0x55f79435fa70 2024-04-27T03:21:07.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f794458af0 0x55f79445aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:2 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f79445b510 con 0x55f794458af0 2024-04-27T03:21:07.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4307584640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f794458af0 0x55f79445aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4307584640 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f794458af0 0x55f79445aee0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4307584640 1 -- 172.21.15.110:0/3782688643 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(2 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f42f00d3020 con 0x55f794458af0 2024-04-27T03:21:07.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f79445cdc0 con 0x55f79435fa70 2024-04-27T03:21:07.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4307d85640 1 -- 172.21.15.110:0/3782688643 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(3 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [call out=48b,read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 232+0+1041 (crc 0 0 0) 0x7f430000a4d0 con 0x55f79435fa70 2024-04-27T03:21:07.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.833+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f79445bff0 con 0x55f794458af0 2024-04-27T03:21:07.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4307584640 1 -- 172.21.15.110:0/3782688643 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(4 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f42f00d3020 con 0x55f794458af0 2024-04-27T03:21:07.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f79445c3e0 con 0x55f79435fa70 2024-04-27T03:21:07.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4307d85640 1 -- 172.21.15.110:0/3782688643 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(5 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [call out=48b,read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 237+0+484 (crc 0 0 0) 0x7f430000a4d0 con 0x55f79435fa70 2024-04-27T03:21:07.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4309a69980 1 Processor -- start 2024-04-27T03:21:07.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4309a69980 1 -- start start 2024-04-27T03:21:07.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4309a69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79446b6c0 0x55f79446bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4309a69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f794267310 con 0x55f79446b6c0 2024-04-27T03:21:07.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4307d85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79446b6c0 0x55f79446bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4307d85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79446b6c0 0x55f79446bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:43990/0 (socket says 172.21.15.110:43990) 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4307d85640 1 -- 172.21.15.110:0/3844520047 learned_addr learned my addr 172.21.15.110:0/3844520047 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4307d85640 1 -- 172.21.15.110:0/3844520047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f79446c020 con 0x55f79446b6c0 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4307d85640 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79446b6c0 0x55f79446bae0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f430000dfc0 tx=0x7f43000178e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f41f1602640 1 -- 172.21.15.110:0/3844520047 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43000157d0 con 0x55f79446b6c0 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f79446cac0 con 0x55f79446b6c0 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f41f1602640 1 -- 172.21.15.110:0/3844520047 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4300015970 con 0x55f79446b6c0 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f41f1602640 1 -- 172.21.15.110:0/3844520047 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4300015c70 con 0x55f79446b6c0 2024-04-27T03:21:07.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f79446cf90 con 0x55f79446b6c0 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.837+0000 7f41f1602640 1 -- 172.21.15.110:0/3844520047 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4300029440 con 0x55f79446b6c0 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f41f1602640 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41e0034650 0x7f41e0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f41f1602640 1 -- 172.21.15.110:0/3844520047 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f430000e040 con 0x55f79446b6c0 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307584640 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41e0034650 0x7f41e0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 Processor -- start 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- start start 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307584640 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41e0034650 0x7f41e0036b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55f79445bb10 tx=0x7f42f00bef70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f794478c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f79444b100 con 0x55f794478870 2024-04-27T03:21:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f794478c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f794478c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:44006/0 (socket says 172.21.15.110:44006) 2024-04-27T03:21:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 -- 172.21.15.110:0/3814527963 learned_addr learned my addr 172.21.15.110:0/3814527963 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:21:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 -- 172.21.15.110:0/3814527963 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f794479260 con 0x55f794478870 2024-04-27T03:21:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 --2- 172.21.15.110:0/3814527963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f794478c90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4300015f40 tx=0x7f430004aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb66bc4d9e2afc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f41eaffd640 1 -- 172.21.15.110:0/3814527963 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43000283a0 con 0x55f794478870 2024-04-27T03:21:07.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f41eaffd640 1 -- 172.21.15.110:0/3814527963 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4300028540 con 0x55f794478870 2024-04-27T03:21:07.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f41eaffd640 1 -- 172.21.15.110:0/3814527963 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f430001eb90 con 0x55f794478870 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- 172.21.15.110:0/3814527963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 msgr2=0x55f794478c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 --2- 172.21.15.110:0/3814527963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f794478c90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4300015f40 tx=0x7f430004aa00 comp rx=0 tx=0).stop 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- 172.21.15.110:0/3814527963 shutdown_connections 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 --2- 172.21.15.110:0/3814527963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f794478c90 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- 172.21.15.110:0/3814527963 >> 172.21.15.110:0/3814527963 conn(0x55f794474c00 msgr2=0x55f794472c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- 172.21.15.110:0/3814527963 shutdown_connections 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- 172.21.15.110:0/3814527963 wait complete. 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 Processor -- start 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- start start 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f79447fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4309a69980 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x55f7944799d0 con 0x55f794478870 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f79447fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f79447fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.110:44010/0 (socket says 172.21.15.110:44010) 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.841+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 learned_addr learned my addr 172.21.15.110:0/939126641 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7944802e0 con 0x55f794478870 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4307d85640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f79447fda0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4300015f40 tx=0x7f430004a850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f41e97fa640 1 -- 172.21.15.110:0/939126641 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4300036070 con 0x55f794478870 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f41e97fa640 1 -- 172.21.15.110:0/939126641 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f430004c110 con 0x55f794478870 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f794480570 con 0x55f794478870 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f41e97fa640 1 -- 172.21.15.110:0/939126641 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4300041040 con 0x55f794478870 2024-04-27T03:21:07.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f794483250 con 0x55f794478870 2024-04-27T03:21:07.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f41e97fa640 1 -- 172.21.15.110:0/939126641 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4300039070 con 0x55f794478870 2024-04-27T03:21:07.858 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f41e97fa640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41d0034650 0x7f41d0036b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f41e97fa640 1 -- 172.21.15.110:0/939126641 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f4300032d70 con 0x55f794478870 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4307584640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41d0034650 0x7f41d0036b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79448b3a0 0x55f79448d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:1 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f79448dfb0 con 0x55f79448b3a0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4306d83640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79448b3a0 0x55f79448d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4307584640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41d0034650 0x7f41d0036b10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f42f00a9960 tx=0x7f42f00dd000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4306d83640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79448b3a0 0x55f79448d860 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== osd_op_reply(1 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 20 realm 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:2 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f79448a7f0 con 0x55f79448b3a0 2024-04-27T03:21:07.859 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== osd_op_reply(2 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.845+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:3 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f79448eab0 con 0x55f79448b3a0 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 3 ==== osd_op_reply(3 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f79448f890 0x55f794491c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:4 5.f 5:f43b6ece:::zone_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794492300 con 0x55f79448f890 2024-04-27T03:21:07.860 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4307d85640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f79448f890 0x55f794491c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4307d85640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f79448f890 0x55f794491c80 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== osd_op_reply(4 zone_names.default [read 0~46 out=46b] v0'0 uv3 ondisk = 0) ==== 162+0+46 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:21:07.862 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:5 5.1e 5:7c17a155:::zone_info.27a51273-3ccc-49e2-8682-1122abc3d898:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794492b50 con 0x55f79448b3a0 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 4 ==== osd_op_reply(5 zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 [read 0~993 out=993b] v0'0 uv4 ondisk = 0) ==== 190+0+993 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 20 rados_obj.operate() r=0 bl.length=993 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 20 searching for the correct realm 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f7944948c0 0x55f794496cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:6 5.0 5:00000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f7944948c0 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4307584640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f7944948c0 0x55f794496cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4307584640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f7944948c0 0x55f794496cb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== osd_op_reply(6 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.849+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:7 5.10 5:08000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448f890 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== osd_op_reply(7 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307d85640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4300014030 0x7f4300007950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:8 5.8 5:10000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x7f4300014030 2024-04-27T03:21:07.863 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4300014030 0x7f4300007950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:07.864 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4300014030 0x7f4300007950 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== osd_op_reply(8 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:9 5.18 5:18000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 5 ==== osd_op_reply(9 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:10 5.4 5:20000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 6 ==== osd_op_reply(10 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:11 5.14 5:28000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f7944948c0 2024-04-27T03:21:07.865 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== osd_op_reply(11 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:12 5.c 5:30000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 7 ==== osd_op_reply(12 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:13 5.1c 5:38000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448f890 2024-04-27T03:21:07.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 3 ==== osd_op_reply(13 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.866 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:14 5.2 5:40000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448f890 2024-04-27T03:21:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 4 ==== osd_op_reply(14 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:15 5.12 5:48000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 8 ==== osd_op_reply(15 [pgnls start_epoch 31 out=112b] v20'1 uv1 ondisk = 1) ==== 144+0+112 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.853+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:16 5.a 5:50000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x7f4300014030 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== osd_op_reply(16 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:17 5.1a 5:58000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 9 ==== osd_op_reply(17 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:18 5.6 5:60000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x7f4300014030 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 3 ==== osd_op_reply(18 [pgnls start_epoch 31 out=79b] v20'1 uv1 ondisk = 1) ==== 144+0+79 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:19 5.16 5:68000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448f890 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 5 ==== osd_op_reply(19 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:20 5.e 5:70000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f7944948c0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 3 ==== osd_op_reply(20 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:21 5.1e 5:78000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 10 ==== osd_op_reply(21 [pgnls start_epoch 31 out=107b] v31'4 uv4 ondisk = 1) ==== 144+0+107 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:22 5.1 5:80000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448f890 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 6 ==== osd_op_reply(22 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:23 5.11 5:88000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f7944948c0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 4 ==== osd_op_reply(23 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:24 5.9 5:90000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 11 ==== osd_op_reply(24 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:25 5.19 5:98000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x7f4300014030 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 4 ==== osd_op_reply(25 [pgnls start_epoch 31 out=74b] v31'2 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.870 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.857+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:26 5.5 5:a0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 12 ==== osd_op_reply(26 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:27 5.15 5:a8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f7944948c0 2024-04-27T03:21:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 5 ==== osd_op_reply(27 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:28 5.d 5:b0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x7f4300014030 2024-04-27T03:21:07.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 5 ==== osd_op_reply(28 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:29 5.1d 5:b8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x55f79448b3a0 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 13 ==== osd_op_reply(29 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:30 5.3 5:c0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x55f794497330 con 0x7f4300014030 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 6 ==== osd_op_reply(30 [pgnls start_epoch 31 out=82b] v31'1 uv1 ondisk = 1) ==== 144+0+82 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:31 5.13 5:c8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f42f803e060 con 0x55f7944948c0 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 6 ==== osd_op_reply(31 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:32 5.b 5:d0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f42f803e060 con 0x7f4300014030 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 7 ==== osd_op_reply(32 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:33 5.1b 5:d8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f42f803e060 con 0x55f79448f890 2024-04-27T03:21:07.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 7 ==== osd_op_reply(33 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:34 5.7 5:e0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f42f803e060 con 0x55f79448f890 2024-04-27T03:21:07.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 8 ==== osd_op_reply(34 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:35 5.17 5:e8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f42f803e060 con 0x55f7944948c0 2024-04-27T03:21:07.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 7 ==== osd_op_reply(35 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.861+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:36 5.f 5:f0000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f42f803e060 con 0x55f79448f890 2024-04-27T03:21:07.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 9 ==== osd_op_reply(36 [pgnls start_epoch 31 out=115b] v31'3 uv3 ondisk = 1) ==== 144+0+115 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:37 5.1f 5:f8000000::::head [pgnls start_epoch 31 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e31) -- 0x7f42f803e060 con 0x55f79448f890 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 10 ==== osd_op_reply(37 [pgnls start_epoch 31 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 RGWRados::pool_iterate: got zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 RGWRados::pool_iterate: got default.zonegroup. 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 RGWRados::pool_iterate: got zone_info.27a51273-3ccc-49e2-8682-1122abc3d898 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 RGWRados::pool_iterate: got default.zone. 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 RGWRados::pool_iterate: got period_config.default 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 RGWRados::pool_iterate: got zone_names.default 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-04-27T03:21:07.878 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:38 5.12 5:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794493500 con 0x55f79448b3a0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 14 ==== osd_op_reply(38 default.realm [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 157+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:39 5.f 5:f4c53578:::zonegroups_names.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794493da0 con 0x55f79448f890 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 11 ==== osd_op_reply(39 zonegroups_names.default [read 0~46 out=46b] v0'0 uv2 ondisk = 0) ==== 168+0+46 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados_obj.operate() r=0 bl.length=46 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:40 5.12 5:4d6f95c2:::zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794493140 con 0x55f79448b3a0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 15 ==== osd_op_reply(40 zonegroup_info.3fb181a9-2f1b-4a6e-85a1-ca8305b0483c [read 0~436 out=436b] v0'0 uv1 ondisk = 0) ==== 195+0+436 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados_obj.operate() r=0 bl.length=436 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 zone default found 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 4 Realm: () 2024-04-27T03:21:07.879 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 4 ZoneGroup: default (3fb181a9-2f1b-4a6e-85a1-ca8305b0483c) 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 4 Zone: default (27a51273-3ccc-49e2-8682-1122abc3d898) 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 10 cannot find current period zonegroup using local zonegroup configuration 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 zonegroup default 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:41 5.3 5:c52100b6:::period_config.default:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794497c40 con 0x7f4300014030 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 8 ==== osd_op_reply(41 period_config.default [read 0~187 out=187b] v0'0 uv1 ondisk = 0) ==== 165+0+187 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados_obj.operate() r=0 bl.length=187 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:42 6.2 6:4347d321:::bucket.sync-source-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794499a10 con 0x55f7944948c0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 8 ==== osd_op_reply(42 bucket.sync-source-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados->read ofs=0 len=0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:43 6.b 6:d467b91b:::bucket.sync-target-hints.:head [call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f794499ed0 con 0x55f7944948c0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 9 ==== osd_op_reply(43 bucket.sync-target-hints. [call,read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 211+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.880 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 rados_obj.operate() r=-2 bl.length=0 2024-04-27T03:21:07.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 started sync module instance, tier type = 2024-04-27T03:21:07.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 20 started zone id=27a51273-3ccc-49e2-8682-1122abc3d898 (name=default) with tier type = 2024-04-27T03:21:07.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.865+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:44 7.1f 7:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944988e0 con 0x55f79448f890 2024-04-27T03:21:07.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.869+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 12 ==== osd_op_reply(44 notify.0 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.869+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:45 7.1f 7:f95f44c2:::notify.0:head [watch watch cookie 94521833131520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944988e0 con 0x55f79448f890 2024-04-27T03:21:07.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.869+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:46 7.0 7:05bf5b68:::notify.1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449a950 con 0x55f79448b3a0 2024-04-27T03:21:07.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.869+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 16 ==== osd_op_reply(46 notify.1 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.869+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:47 7.0 7:05bf5b68:::notify.1:head [watch watch cookie 94521833137312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449a8c0 con 0x55f79448b3a0 2024-04-27T03:21:07.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.869+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:48 7.15 7:a93a5511:::notify.2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449b940 con 0x55f79448f890 2024-04-27T03:21:07.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.869+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 13 ==== osd_op_reply(45 notify.0 [watch watch cookie 94521833131520] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 17 ==== osd_op_reply(47 notify.1 [watch watch cookie 94521833137312] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 14 ==== osd_op_reply(48 notify.2 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:49 7.15 7:a93a5511:::notify.2:head [watch watch cookie 94521833139520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449b900 con 0x55f79448f890 2024-04-27T03:21:07.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:50 7.e 7:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449bd70 con 0x7f4300014030 2024-04-27T03:21:07.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 9 ==== osd_op_reply(50 notify.3 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:51 7.e 7:7759931f:::notify.3:head [watch watch cookie 94521833129008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449bd30 con 0x7f4300014030 2024-04-27T03:21:07.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:52 7.d 7:b4812045:::notify.4:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449cc30 con 0x55f79448f890 2024-04-27T03:21:07.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 15 ==== osd_op_reply(49 notify.2 [watch watch cookie 94521833139520] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.873+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 10 ==== osd_op_reply(51 notify.3 [watch watch cookie 94521833129008] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 16 ==== osd_op_reply(52 notify.4 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:53 7.d 7:b4812045:::notify.4:head [watch watch cookie 94521833144672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449c780 con 0x55f79448f890 2024-04-27T03:21:07.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:54 7.3 7:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449d4d0 con 0x55f7944948c0 2024-04-27T03:21:07.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 10 ==== osd_op_reply(54 notify.5 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:55 7.3 7:c609908c:::notify.5:head [watch watch cookie 94521833148448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449c780 con 0x55f7944948c0 2024-04-27T03:21:07.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:56 7.14 7:2b04a3e9:::notify.6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449e340 con 0x55f7944948c0 2024-04-27T03:21:07.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 17 ==== osd_op_reply(53 notify.4 [watch watch cookie 94521833144672] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 11 ==== osd_op_reply(55 notify.5 [watch watch cookie 94521833148448] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 12 ==== osd_op_reply(56 notify.6 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:57 7.14 7:2b04a3e9:::notify.6:head [watch watch cookie 94521833141264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449d980 con 0x55f7944948c0 2024-04-27T03:21:07.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.877+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:58 7.9 7:93e5b521:::notify.7:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449ea40 con 0x7f4300014030 2024-04-27T03:21:07.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 11 ==== osd_op_reply(58 notify.7 [create] v31'283 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 13 ==== osd_op_reply(57 notify.6 [watch watch cookie 94521833141264] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:59 7.9 7:93e5b521:::notify.7:head [watch watch cookie 94521833153808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449ea00 con 0x7f4300014030 2024-04-27T03:21:07.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=0 2024-04-27T03:21:07.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=1 2024-04-27T03:21:07.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=2 2024-04-27T03:21:07.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=3 2024-04-27T03:21:07.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=4 2024-04-27T03:21:07.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=5 2024-04-27T03:21:07.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=6 2024-04-27T03:21:07.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 12 ==== osd_op_reply(59 notify.7 [watch watch cookie 94521833153808] v31'284 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 20 add_watcher() i=7 2024-04-27T03:21:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 2 all 8 watchers are set, enabling cache 2024-04-27T03:21:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:60 6.4 6:22d26bf9:::data_loggenerations_metadata:head [call version.check_conds in=74b,call version.read in=11b,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a1ec0 con 0x55f79448b3a0 2024-04-27T03:21:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 18 ==== osd_op_reply(60 data_loggenerations_metadata [call,call out=48b,read 0~28 out=28b] v0'0 uv1 ondisk = 0) ==== 256+0+76 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.881+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:61 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94521833152144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449f3e0 con 0x55f79448b3a0 2024-04-27T03:21:07.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.885+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 19 ==== osd_op_reply(61 data_loggenerations_metadata [watch watch cookie 94521833152144] v31'280 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.885+0000 7f41bdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:21:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.885+0000 7f4309a69980 20 rgw_check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-04-27T03:21:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.885+0000 7f4309a69980 20 rgw_check_secure_mon_conn(): mode 1 is insecure 2024-04-27T03:21:07.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.885+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:62 6.19 6:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944aa800 con 0x55f79448f890 2024-04-27T03:21:07.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.889+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 18 ==== osd_op_reply(62 gc.0 [create,call,call,call] v31'61 uv51 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.889+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:63 6.18 6:1e85421f:gc::gc.1:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944aa8f0 con 0x7f4300014030 2024-04-27T03:21:07.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.889+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 13 ==== osd_op_reply(63 gc.1 [create,call,call,call] v31'39 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.889+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:64 6.b 6:d242335b:gc::gc.2:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ab020 con 0x55f7944948c0 2024-04-27T03:21:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.889+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 14 ==== osd_op_reply(64 gc.2 [create,call,call,call] v31'38 uv28 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.889+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:65 6.4 6:2166d8e1:gc::gc.3:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ab0b0 con 0x55f79448b3a0 2024-04-27T03:21:07.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.893+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 20 ==== osd_op_reply(65 gc.3 [create,call,call,call] v31'281 uv168 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.893+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:66 6.3 6:c60da066:gc::gc.4:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ab620 con 0x55f79448f890 2024-04-27T03:21:07.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.897+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 19 ==== osd_op_reply(66 gc.4 [create,call,call,call] v31'93 uv59 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.897+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:67 6.9 6:954be8d2:gc::gc.5:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944aba40 con 0x55f7944948c0 2024-04-27T03:21:07.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.897+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 15 ==== osd_op_reply(67 gc.5 [create,call,call,call] v31'67 uv46 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.897+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:68 6.e 6:748722ff:gc::gc.6:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944aba40 con 0x55f79448f890 2024-04-27T03:21:07.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.901+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 20 ==== osd_op_reply(68 gc.6 [create,call,call,call] v31'41 uv31 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.901+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:69 6.1d 6:beaa4be4:gc::gc.7:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944aba40 con 0x55f79448b3a0 2024-04-27T03:21:07.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.901+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 21 ==== osd_op_reply(69 gc.7 [create,call,call,call] v31'63 uv41 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.901+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:70 6.12 6:4f78e9a2:gc::gc.8:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abad0 con 0x55f79448f890 2024-04-27T03:21:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.905+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 21 ==== osd_op_reply(70 gc.8 [create,call,call,call] v31'32 uv21 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.905+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:71 6.1c 6:3e426232:gc::gc.9:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448f890 2024-04-27T03:21:07.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.909+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 22 ==== osd_op_reply(71 gc.9 [create,call,call,call] v31'37 uv27 ondisk = -125 ((125) Operation canceled)) ==== 274+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.909+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:72 6.3 6:c31f41d7:gc::gc.10:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448f890 2024-04-27T03:21:07.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.913+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 23 ==== osd_op_reply(72 gc.10 [create,call,call,call] v31'94 uv91 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.913+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:73 6.13 6:c8333aab:gc::gc.11:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f7944948c0 2024-04-27T03:21:07.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.913+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 16 ==== osd_op_reply(73 gc.11 [create,call,call,call] v31'41 uv24 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.913+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:74 6.15 6:ab569f81:gc::gc.12:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448f890 2024-04-27T03:21:07.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.917+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 24 ==== osd_op_reply(74 gc.12 [create,call,call,call] v31'37 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.917+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:75 6.0 6:0667bb13:gc::gc.13:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb90 con 0x55f79448b3a0 2024-04-27T03:21:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.917+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 22 ==== osd_op_reply(75 gc.13 [create,call,call,call] v31'74 uv54 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.917+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:76 6.4 6:227f63b5:gc::gc.14:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abc20 con 0x55f79448b3a0 2024-04-27T03:21:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.921+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 23 ==== osd_op_reply(76 gc.14 [create,call,call,call] v31'282 uv170 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.921+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:77 6.7 6:e07a6548:gc::gc.15:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abc20 con 0x55f79448f890 2024-04-27T03:21:07.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.921+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 25 ==== osd_op_reply(77 gc.15 [create,call,call,call] v31'83 uv53 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.921+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:78 6.3 6:c623b173:gc::gc.16:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abc20 con 0x55f79448f890 2024-04-27T03:21:07.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.925+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 26 ==== osd_op_reply(78 gc.16 [create,call,call,call] v31'95 uv63 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.925+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:79 6.16 6:6849fd06:gc::gc.17:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f7944948c0 2024-04-27T03:21:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.929+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 17 ==== osd_op_reply(79 gc.17 [create,call,call,call] v31'44 uv34 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.929+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:80 6.d 6:b019ea83:gc::gc.18:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448f890 2024-04-27T03:21:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.929+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 27 ==== osd_op_reply(80 gc.18 [create,call,call,call] v31'39 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.929+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:81 6.4 6:24c8e1ae:gc::gc.19:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448b3a0 2024-04-27T03:21:07.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.933+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 24 ==== osd_op_reply(81 gc.19 [create,call,call,call] v31'283 uv172 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.933+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:82 6.7 6:e622f76a:gc::gc.20:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abba0 con 0x55f79448f890 2024-04-27T03:21:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.937+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 28 ==== osd_op_reply(82 gc.20 [create,call,call,call] v31'84 uv55 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.937+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:83 6.a 6:5127e642:gc::gc.21:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448f890 2024-04-27T03:21:07.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.937+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 29 ==== osd_op_reply(83 gc.21 [create,call,call,call] v31'48 uv37 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.937+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:84 6.11 6:8eb99c53:gc::gc.22:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f7944948c0 2024-04-27T03:21:07.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.941+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 18 ==== osd_op_reply(84 gc.22 [create,call,call,call] v31'39 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.941+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:85 6.8 6:17d7e8ed:gc::gc.23:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448b3a0 2024-04-27T03:21:07.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.941+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 25 ==== osd_op_reply(85 gc.23 [create,call,call,call] v31'49 uv39 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.941+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:86 6.1b 6:db9d130d:gc::gc.24:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f7944948c0 2024-04-27T03:21:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.945+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 19 ==== osd_op_reply(86 gc.24 [create,call,call,call] v31'33 uv23 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.945+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:87 6.9 6:921ecdb1:gc::gc.25:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f7944948c0 2024-04-27T03:21:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.945+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 20 ==== osd_op_reply(87 gc.25 [create,call,call,call] v31'68 uv48 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.949+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:88 6.1f 6:fde4dd55:gc::gc.26:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f7944948c0 2024-04-27T03:21:07.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.949+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 21 ==== osd_op_reply(88 gc.26 [create,call,call,call] v31'40 uv29 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.949+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:89 6.1a 6:5909e671:gc::gc.27:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abb60 con 0x55f79448f890 2024-04-27T03:21:07.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.953+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 30 ==== osd_op_reply(89 gc.27 [create,call,call,call] v31'45 uv33 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.953+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:90 6.1d 6:b88e3271:gc::gc.28:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abad0 con 0x55f79448b3a0 2024-04-27T03:21:07.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.953+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 26 ==== osd_op_reply(90 gc.28 [create,call,call,call] v31'64 uv43 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.953+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:91 6.7 6:e47c9322:gc::gc.29:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abad0 con 0x55f79448f890 2024-04-27T03:21:07.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.957+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 31 ==== osd_op_reply(91 gc.29 [create,call,call,call] v31'85 uv51 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.957+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:92 6.0 6:013d6c33:gc::gc.30:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abad0 con 0x55f79448b3a0 2024-04-27T03:21:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 27 ==== osd_op_reply(92 gc.30 [create,call,call,call] v31'75 uv52 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:93 6.f 6:f24cb4a8:gc::gc.31:head [create,call version.check_conds in=74b,call rgw_gc.rgw_gc_queue_init in=45b,call version.set in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944abad0 con 0x7f4300014030 2024-04-27T03:21:07.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 14 ==== osd_op_reply(93 gc.31 [create,call,call,call] v31'37 uv27 ondisk = -125 ((125) Operation canceled)) ==== 275+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f41bcfe1640 20 reqs_thread_entry: start 2024-04-27T03:21:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f4309a69980 20 init_complete bucket index max shards: 11 2024-04-27T03:21:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f4309a69980 20 Filter name: none 2024-04-27T03:21:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f41367fc640 20 reqs_thread_entry: start 2024-04-27T03:21:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2024-04-27T03:21:07.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.961+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:94 6.1c 6:3e426232:gc::gc.9:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.965+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 32 ==== osd_op_reply(94 gc.9 [call] v31'38 uv38 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.965+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:95 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.965+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 33 ==== osd_op_reply(95 gc.9 [call out=15b] v0'0 uv38 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.965+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.965+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:96 6.1c 6:3e426232:gc::gc.9:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c7450 con 0x55f79448f890 2024-04-27T03:21:07.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 34 ==== osd_op_reply(96 gc.9 [call out=24b] v0'0 uv38 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:97 6.1c 6:3e426232:gc::gc.9:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 35 ==== osd_op_reply(97 gc.9 [call out=15b] v0'0 uv38 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:98 6.1c 6:3e426232:gc::gc.9:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 36 ==== osd_op_reply(98 gc.9 [call out=15b] v0'0 uv38 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:07.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:99 6.1c 6:3e426232:gc::gc.9:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 37 ==== osd_op_reply(99 gc.9 [call] v31'39 uv39 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2024-04-27T03:21:07.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.969+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:100 6.3 6:c31f41d7:gc::gc.10:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.973+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 38 ==== osd_op_reply(100 gc.10 [call] v31'96 uv96 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.973+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:101 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.973+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 39 ==== osd_op_reply(101 gc.10 [call out=15b] v0'0 uv96 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.973+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:07.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.973+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:102 6.3 6:c31f41d7:gc::gc.10:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c7450 con 0x55f79448f890 2024-04-27T03:21:07.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.973+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 40 ==== osd_op_reply(102 gc.10 [call out=24b] v0'0 uv96 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.973+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:103 6.3 6:c31f41d7:gc::gc.10:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 41 ==== osd_op_reply(103 gc.10 [call out=15b] v0'0 uv96 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:07.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:104 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944c8460 con 0x55f79448f890 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 42 ==== osd_op_reply(104 gc.10 [call out=1715b] v0'0 uv96 ondisk = 0) ==== 149+0+1715 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 20 garbage collection: RGWGC::process iterating over entry tag='27a51273-3ccc-49e2-8682-1122abc3d898.4220.7199520564531325328', time=2024-04-27T03:21:15.725002+0000, chain.objs.size()=7 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_1 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:105 3.0s0 3:01702c4a:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_1:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a9610 con 0x55f79448b3a0 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_2 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:106 3.1es0 3:78543505:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_2:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944cb940 con 0x55f7944948c0 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_3 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:107 3.0s0 3:000bd6a2:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_3:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448b3a0 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_4 2024-04-27T03:21:07.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:108 3.3fs0 3:fdda40c0:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_4:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d0f30 con 0x55f79448f890 2024-04-27T03:21:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_5 2024-04-27T03:21:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:109 3.4s0 3:207acbde:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_5:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d3870 con 0x55f79448b3a0 2024-04-27T03:21:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_6 2024-04-27T03:21:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:110 3.26s0 3:645abdac:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_6:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x7f4300014030 2024-04-27T03:21:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_7 2024-04-27T03:21:07.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.977+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:111 3.21s0 3:851c6164:::27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_7:head [call refcount.put in=84b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d0f30 con 0x7f4300014030 2024-04-27T03:21:07.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.981+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 43 ==== osd_op_reply(108 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_4 [call] v31'2 uv2 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.981+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 28 ==== osd_op_reply(109 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_5 [call] v31'6 uv6 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.981+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 29 ==== osd_op_reply(105 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_1 [call] v31'3 uv3 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.981+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 15 ==== osd_op_reply(110 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_6 [call] v31'4 uv4 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.981+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 22 ==== osd_op_reply(106 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_2 [call] v31'2 uv2 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:07.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.981+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 16 ==== osd_op_reply(111 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_7 [call] v31'2 uv2 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.985+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 30 ==== osd_op_reply(107 27a51273-3ccc-49e2-8682-1122abc3d898.4220.11__shadow_.hfxUKt_FT4zwEJDmToAz9DrA8N5DTvV_3 [call] v31'4 uv4 ondisk = 0) ==== 231+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.985+0000 7f4309a69980 5 garbage collection: RGWGC::process removing entries, marker: 2024-04-27T03:21:07.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.985+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:112 6.3 6:c31f41d7:gc::gc.10:head [call rgw_gc.rgw_gc_queue_remove_entries in=47b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448f890 2024-04-27T03:21:08.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.993+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 44 ==== osd_op_reply(112 gc.10 [call] v31'97 uv97 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.993+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:113 6.3 6:c31f41d7:gc::gc.10:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d0f30 con 0x55f79448f890 2024-04-27T03:21:08.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.993+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 45 ==== osd_op_reply(113 gc.10 [call] v31'98 uv98 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.993+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2024-04-27T03:21:08.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.993+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:114 6.13 6:c8333aab:gc::gc.11:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f7944948c0 2024-04-27T03:21:08.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 23 ==== osd_op_reply(114 gc.11 [call] v31'42 uv42 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:115 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d0f30 con 0x55f7944948c0 2024-04-27T03:21:08.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 24 ==== osd_op_reply(115 gc.11 [call out=15b] v0'0 uv42 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:116 6.13 6:c8333aab:gc::gc.11:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f7944948c0 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 25 ==== osd_op_reply(116 gc.11 [call out=24b] v0'0 uv42 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:117 6.13 6:c8333aab:gc::gc.11:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d0f30 con 0x55f7944948c0 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 26 ==== osd_op_reply(117 gc.11 [call out=15b] v0'0 uv42 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:118 6.13 6:c8333aab:gc::gc.11:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f7944948c0 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 27 ==== osd_op_reply(118 gc.11 [call out=15b] v0'0 uv42 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:119 6.13 6:c8333aab:gc::gc.11:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d0f30 con 0x55f7944948c0 2024-04-27T03:21:08.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 28 ==== osd_op_reply(119 gc.11 [call] v31'43 uv43 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2024-04-27T03:21:08.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:07.997+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:120 6.15 6:ab569f81:gc::gc.12:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448f890 2024-04-27T03:21:08.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.001+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 46 ==== osd_op_reply(120 gc.12 [call] v31'38 uv38 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.001+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:121 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944d0f30 con 0x55f79448f890 2024-04-27T03:21:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.001+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 47 ==== osd_op_reply(121 gc.12 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.001+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.001+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:122 6.15 6:ab569f81:gc::gc.12:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448f890 2024-04-27T03:21:08.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.001+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 48 ==== osd_op_reply(122 gc.12 [call out=24b] v0'0 uv38 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.001+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:123 6.15 6:ab569f81:gc::gc.12:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448f890 2024-04-27T03:21:08.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 49 ==== osd_op_reply(123 gc.12 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:124 6.15 6:ab569f81:gc::gc.12:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448f890 2024-04-27T03:21:08.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 50 ==== osd_op_reply(124 gc.12 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:125 6.15 6:ab569f81:gc::gc.12:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448f890 2024-04-27T03:21:08.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 51 ==== osd_op_reply(125 gc.12 [call] v31'39 uv39 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2024-04-27T03:21:08.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.005+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:126 6.0 6:0667bb13:gc::gc.13:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448b3a0 2024-04-27T03:21:08.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 31 ==== osd_op_reply(126 gc.13 [call] v31'76 uv76 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:127 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 32 ==== osd_op_reply(127 gc.13 [call out=15b] v0'0 uv76 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:128 6.0 6:0667bb13:gc::gc.13:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944ce310 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 33 ==== osd_op_reply(128 gc.13 [call out=24b] v0'0 uv76 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:129 6.0 6:0667bb13:gc::gc.13:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 34 ==== osd_op_reply(129 gc.13 [call out=15b] v0'0 uv76 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:130 6.0 6:0667bb13:gc::gc.13:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 35 ==== osd_op_reply(130 gc.13 [call out=15b] v0'0 uv76 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.009+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:131 6.0 6:0667bb13:gc::gc.13:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 36 ==== osd_op_reply(131 gc.13 [call] v31'77 uv77 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2024-04-27T03:21:08.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:132 6.4 6:227f63b5:gc::gc.14:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 37 ==== osd_op_reply(132 gc.14 [call] v31'284 uv284 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:133 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 38 ==== osd_op_reply(133 gc.14 [call out=15b] v0'0 uv284 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.013+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:134 6.4 6:227f63b5:gc::gc.14:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 39 ==== osd_op_reply(134 gc.14 [call out=24b] v0'0 uv284 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:135 6.4 6:227f63b5:gc::gc.14:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 40 ==== osd_op_reply(135 gc.14 [call out=15b] v0'0 uv284 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:136 6.4 6:227f63b5:gc::gc.14:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 41 ==== osd_op_reply(136 gc.14 [call out=15b] v0'0 uv284 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:137 6.4 6:227f63b5:gc::gc.14:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 42 ==== osd_op_reply(137 gc.14 [call] v31'285 uv285 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2024-04-27T03:21:08.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.017+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:138 6.7 6:e07a6548:gc::gc.15:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.021+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 52 ==== osd_op_reply(138 gc.15 [call] v31'86 uv86 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.021+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:139 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.021+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 53 ==== osd_op_reply(139 gc.15 [call out=15b] v0'0 uv86 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.021+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.021+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:140 6.7 6:e07a6548:gc::gc.15:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.021+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 54 ==== osd_op_reply(140 gc.15 [call out=24b] v0'0 uv86 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.021+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:141 6.7 6:e07a6548:gc::gc.15:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.034 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 55 ==== osd_op_reply(141 gc.15 [call out=15b] v0'0 uv86 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:142 6.7 6:e07a6548:gc::gc.15:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 56 ==== osd_op_reply(142 gc.15 [call out=15b] v0'0 uv86 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:143 6.7 6:e07a6548:gc::gc.15:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 57 ==== osd_op_reply(143 gc.15 [call] v31'87 uv87 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2024-04-27T03:21:08.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.025+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:144 6.3 6:c623b173:gc::gc.16:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.029+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 58 ==== osd_op_reply(144 gc.16 [call] v31'99 uv99 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.040 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.029+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:145 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.029+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 59 ==== osd_op_reply(145 gc.16 [call out=15b] v0'0 uv99 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.029+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.041 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.029+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:146 6.3 6:c623b173:gc::gc.16:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.029+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 60 ==== osd_op_reply(146 gc.16 [call out=24b] v0'0 uv99 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.029+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:147 6.3 6:c623b173:gc::gc.16:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.033+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 61 ==== osd_op_reply(147 gc.16 [call out=15b] v0'0 uv99 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.033+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.033+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:148 6.3 6:c623b173:gc::gc.16:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.033+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 62 ==== osd_op_reply(148 gc.16 [call out=15b] v0'0 uv99 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.033+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.033+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:149 6.3 6:c623b173:gc::gc.16:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 63 ==== osd_op_reply(149 gc.16 [call] v31'100 uv100 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2024-04-27T03:21:08.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:150 6.16 6:6849fd06:gc::gc.17:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 29 ==== osd_op_reply(150 gc.17 [call] v31'45 uv45 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:151 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 30 ==== osd_op_reply(151 gc.17 [call out=15b] v0'0 uv45 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:152 6.16 6:6849fd06:gc::gc.17:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 31 ==== osd_op_reply(152 gc.17 [call out=24b] v0'0 uv45 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.037+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:153 6.16 6:6849fd06:gc::gc.17:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 32 ==== osd_op_reply(153 gc.17 [call out=15b] v0'0 uv45 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:154 6.16 6:6849fd06:gc::gc.17:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 33 ==== osd_op_reply(154 gc.17 [call out=15b] v0'0 uv45 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:155 6.16 6:6849fd06:gc::gc.17:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 34 ==== osd_op_reply(155 gc.17 [call] v31'46 uv46 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2024-04-27T03:21:08.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.041+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:156 6.d 6:b019ea83:gc::gc.18:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 64 ==== osd_op_reply(156 gc.18 [call] v31'40 uv40 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:157 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 65 ==== osd_op_reply(157 gc.18 [call out=15b] v0'0 uv40 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:158 6.d 6:b019ea83:gc::gc.18:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 66 ==== osd_op_reply(158 gc.18 [call out=24b] v0'0 uv40 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:159 6.d 6:b019ea83:gc::gc.18:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 67 ==== osd_op_reply(159 gc.18 [call out=15b] v0'0 uv40 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.045+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:160 6.d 6:b019ea83:gc::gc.18:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.049+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 68 ==== osd_op_reply(160 gc.18 [call out=15b] v0'0 uv40 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.049+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.049+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:161 6.d 6:b019ea83:gc::gc.18:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.049+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 69 ==== osd_op_reply(161 gc.18 [call] v31'41 uv41 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.049+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2024-04-27T03:21:08.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.049+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:162 6.4 6:24c8e1ae:gc::gc.19:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 43 ==== osd_op_reply(162 gc.19 [call] v31'286 uv286 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:163 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 44 ==== osd_op_reply(163 gc.19 [call out=15b] v0'0 uv286 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:164 6.4 6:24c8e1ae:gc::gc.19:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 45 ==== osd_op_reply(164 gc.19 [call out=24b] v0'0 uv286 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:165 6.4 6:24c8e1ae:gc::gc.19:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 46 ==== osd_op_reply(165 gc.19 [call out=15b] v0'0 uv286 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:166 6.4 6:24c8e1ae:gc::gc.19:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 47 ==== osd_op_reply(166 gc.19 [call out=15b] v0'0 uv286 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.053+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:167 6.4 6:24c8e1ae:gc::gc.19:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.057+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 48 ==== osd_op_reply(167 gc.19 [call] v31'287 uv287 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.057+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2024-04-27T03:21:08.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.057+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:168 6.7 6:e622f76a:gc::gc.20:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 70 ==== osd_op_reply(168 gc.20 [call] v31'88 uv88 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:169 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 71 ==== osd_op_reply(169 gc.20 [call out=15b] v0'0 uv88 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:170 6.7 6:e622f76a:gc::gc.20:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 72 ==== osd_op_reply(170 gc.20 [call out=24b] v0'0 uv88 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:171 6.7 6:e622f76a:gc::gc.20:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 73 ==== osd_op_reply(171 gc.20 [call out=15b] v0'0 uv88 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:172 6.7 6:e622f76a:gc::gc.20:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 74 ==== osd_op_reply(172 gc.20 [call out=15b] v0'0 uv88 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.061+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:173 6.7 6:e622f76a:gc::gc.20:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.065+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 75 ==== osd_op_reply(173 gc.20 [call] v31'89 uv89 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.065+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2024-04-27T03:21:08.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.065+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:174 6.a 6:5127e642:gc::gc.21:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 76 ==== osd_op_reply(174 gc.21 [call] v31'49 uv49 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:175 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 77 ==== osd_op_reply(175 gc.21 [call out=15b] v0'0 uv49 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:176 6.a 6:5127e642:gc::gc.21:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 78 ==== osd_op_reply(176 gc.21 [call out=24b] v0'0 uv49 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:177 6.a 6:5127e642:gc::gc.21:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 79 ==== osd_op_reply(177 gc.21 [call out=15b] v0'0 uv49 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.069+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:178 6.a 6:5127e642:gc::gc.21:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.073+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 80 ==== osd_op_reply(178 gc.21 [call out=15b] v0'0 uv49 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.073+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.073+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:179 6.a 6:5127e642:gc::gc.21:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.073+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 81 ==== osd_op_reply(179 gc.21 [call] v31'50 uv50 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.073+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2024-04-27T03:21:08.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.073+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:180 6.11 6:8eb99c53:gc::gc.22:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 35 ==== osd_op_reply(180 gc.22 [call] v31'40 uv40 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:181 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 36 ==== osd_op_reply(181 gc.22 [call out=15b] v0'0 uv40 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:182 6.11 6:8eb99c53:gc::gc.22:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 37 ==== osd_op_reply(182 gc.22 [call out=24b] v0'0 uv40 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:183 6.11 6:8eb99c53:gc::gc.22:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 38 ==== osd_op_reply(183 gc.22 [call out=15b] v0'0 uv40 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:184 6.11 6:8eb99c53:gc::gc.22:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 39 ==== osd_op_reply(184 gc.22 [call out=15b] v0'0 uv40 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.077+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:185 6.11 6:8eb99c53:gc::gc.22:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 40 ==== osd_op_reply(185 gc.22 [call] v31'41 uv41 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2024-04-27T03:21:08.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:186 6.8 6:17d7e8ed:gc::gc.23:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 49 ==== osd_op_reply(186 gc.23 [call] v31'50 uv50 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:187 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 50 ==== osd_op_reply(187 gc.23 [call out=15b] v0'0 uv50 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:188 6.8 6:17d7e8ed:gc::gc.23:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 51 ==== osd_op_reply(188 gc.23 [call out=24b] v0'0 uv50 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:189 6.8 6:17d7e8ed:gc::gc.23:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 52 ==== osd_op_reply(189 gc.23 [call out=15b] v0'0 uv50 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.081+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:190 6.8 6:17d7e8ed:gc::gc.23:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.085+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 53 ==== osd_op_reply(190 gc.23 [call out=15b] v0'0 uv50 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.085+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.085+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:191 6.8 6:17d7e8ed:gc::gc.23:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.085+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 54 ==== osd_op_reply(191 gc.23 [call] v31'51 uv51 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.085+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2024-04-27T03:21:08.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.085+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:192 6.1b 6:db9d130d:gc::gc.24:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 41 ==== osd_op_reply(192 gc.24 [call] v31'34 uv34 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:193 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 42 ==== osd_op_reply(193 gc.24 [call out=15b] v0'0 uv34 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:194 6.1b 6:db9d130d:gc::gc.24:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 43 ==== osd_op_reply(194 gc.24 [call out=24b] v0'0 uv34 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:195 6.1b 6:db9d130d:gc::gc.24:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 44 ==== osd_op_reply(195 gc.24 [call out=15b] v0'0 uv34 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:196 6.1b 6:db9d130d:gc::gc.24:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 45 ==== osd_op_reply(196 gc.24 [call out=15b] v0'0 uv34 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.089+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:197 6.1b 6:db9d130d:gc::gc.24:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.093+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 46 ==== osd_op_reply(197 gc.24 [call] v31'35 uv35 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.093+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2024-04-27T03:21:08.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.093+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:198 6.9 6:921ecdb1:gc::gc.25:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.093+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 47 ==== osd_op_reply(198 gc.25 [call] v31'69 uv69 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:199 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 48 ==== osd_op_reply(199 gc.25 [call out=15b] v0'0 uv69 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:200 6.9 6:921ecdb1:gc::gc.25:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 49 ==== osd_op_reply(200 gc.25 [call out=24b] v0'0 uv69 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:201 6.9 6:921ecdb1:gc::gc.25:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 50 ==== osd_op_reply(201 gc.25 [call out=15b] v0'0 uv69 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:202 6.9 6:921ecdb1:gc::gc.25:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 51 ==== osd_op_reply(202 gc.25 [call out=15b] v0'0 uv69 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.097+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:203 6.9 6:921ecdb1:gc::gc.25:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 52 ==== osd_op_reply(203 gc.25 [call] v31'70 uv70 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2024-04-27T03:21:08.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:204 6.1f 6:fde4dd55:gc::gc.26:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 53 ==== osd_op_reply(204 gc.26 [call] v31'41 uv41 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:205 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 54 ==== osd_op_reply(205 gc.26 [call out=15b] v0'0 uv41 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.101+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:206 6.1f 6:fde4dd55:gc::gc.26:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 55 ==== osd_op_reply(206 gc.26 [call out=24b] v0'0 uv41 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:207 6.1f 6:fde4dd55:gc::gc.26:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 56 ==== osd_op_reply(207 gc.26 [call out=15b] v0'0 uv41 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:208 6.1f 6:fde4dd55:gc::gc.26:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 57 ==== osd_op_reply(208 gc.26 [call out=15b] v0'0 uv41 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:209 6.1f 6:fde4dd55:gc::gc.26:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 58 ==== osd_op_reply(209 gc.26 [call] v31'42 uv42 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2024-04-27T03:21:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.105+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:210 6.1a 6:5909e671:gc::gc.27:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.109+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 82 ==== osd_op_reply(210 gc.27 [call] v31'46 uv46 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.109+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:211 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.109+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 83 ==== osd_op_reply(211 gc.27 [call out=15b] v0'0 uv46 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.109+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.109+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:212 6.1a 6:5909e671:gc::gc.27:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.109+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 84 ==== osd_op_reply(212 gc.27 [call out=24b] v0'0 uv46 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.109+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:213 6.1a 6:5909e671:gc::gc.27:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.113+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 85 ==== osd_op_reply(213 gc.27 [call out=15b] v0'0 uv46 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.113+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.113+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:214 6.1a 6:5909e671:gc::gc.27:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.113+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 86 ==== osd_op_reply(214 gc.27 [call out=15b] v0'0 uv46 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.113+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.113+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:215 6.1a 6:5909e671:gc::gc.27:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.117+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 87 ==== osd_op_reply(215 gc.27 [call] v31'47 uv47 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.117+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2024-04-27T03:21:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.117+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:216 6.1d 6:b88e3271:gc::gc.28:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.117+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 55 ==== osd_op_reply(216 gc.28 [call] v31'65 uv65 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.117+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:217 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.117+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 56 ==== osd_op_reply(217 gc.28 [call out=15b] v0'0 uv65 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:218 6.1d 6:b88e3271:gc::gc.28:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 57 ==== osd_op_reply(218 gc.28 [call out=24b] v0'0 uv65 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:219 6.1d 6:b88e3271:gc::gc.28:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 58 ==== osd_op_reply(219 gc.28 [call out=15b] v0'0 uv65 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:220 6.1d 6:b88e3271:gc::gc.28:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 59 ==== osd_op_reply(220 gc.28 [call out=15b] v0'0 uv65 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.121+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:221 6.1d 6:b88e3271:gc::gc.28:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.125+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 60 ==== osd_op_reply(221 gc.28 [call] v31'66 uv66 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.125+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2024-04-27T03:21:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.125+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:222 6.7 6:e47c9322:gc::gc.29:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 88 ==== osd_op_reply(222 gc.29 [call] v31'90 uv90 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:223 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 89 ==== osd_op_reply(223 gc.29 [call out=15b] v0'0 uv90 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:224 6.7 6:e47c9322:gc::gc.29:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 90 ==== osd_op_reply(224 gc.29 [call out=24b] v0'0 uv90 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:225 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 91 ==== osd_op_reply(225 gc.29 [call out=15b] v0'0 uv90 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:226 6.7 6:e47c9322:gc::gc.29:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 92 ==== osd_op_reply(226 gc.29 [call out=15b] v0'0 uv90 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.129+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:227 6.7 6:e47c9322:gc::gc.29:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.133+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 93 ==== osd_op_reply(227 gc.29 [call] v31'91 uv91 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.133+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2024-04-27T03:21:08.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.133+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:228 6.0 6:013d6c33:gc::gc.30:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 61 ==== osd_op_reply(228 gc.30 [call] v31'78 uv78 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:229 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 62 ==== osd_op_reply(229 gc.30 [call out=15b] v0'0 uv78 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:230 6.0 6:013d6c33:gc::gc.30:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 63 ==== osd_op_reply(230 gc.30 [call out=24b] v0'0 uv78 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:231 6.0 6:013d6c33:gc::gc.30:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 64 ==== osd_op_reply(231 gc.30 [call out=15b] v0'0 uv78 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:232 6.0 6:013d6c33:gc::gc.30:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 65 ==== osd_op_reply(232 gc.30 [call out=15b] v0'0 uv78 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.137+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:233 6.0 6:013d6c33:gc::gc.30:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.141+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 66 ==== osd_op_reply(233 gc.30 [call] v31'79 uv79 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.141+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2024-04-27T03:21:08.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.141+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:234 6.f 6:f24cb4a8:gc::gc.31:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 17 ==== osd_op_reply(234 gc.31 [call] v31'38 uv38 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:235 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 18 ==== osd_op_reply(235 gc.31 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:236 6.f 6:f24cb4a8:gc::gc.31:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 19 ==== osd_op_reply(236 gc.31 [call out=24b] v0'0 uv38 ondisk = 0) ==== 149+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:237 6.f 6:f24cb4a8:gc::gc.31:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 20 ==== osd_op_reply(237 gc.31 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:238 6.f 6:f24cb4a8:gc::gc.31:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 21 ==== osd_op_reply(238 gc.31 [call out=15b] v0'0 uv38 ondisk = 0) ==== 149+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.145+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:239 6.f 6:f24cb4a8:gc::gc.31:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.149+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 22 ==== osd_op_reply(239 gc.31 [call] v31'39 uv39 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.149+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2024-04-27T03:21:08.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.149+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:240 6.19 6:9a6dd8c0:gc::gc.0:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.153+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 94 ==== osd_op_reply(240 gc.0 [call] v31'62 uv62 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.153+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:241 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.153+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 95 ==== osd_op_reply(241 gc.0 [call out=15b] v0'0 uv62 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.153+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.153+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:242 6.19 6:9a6dd8c0:gc::gc.0:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.153+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 96 ==== osd_op_reply(242 gc.0 [call out=24b] v0'0 uv62 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.153+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:243 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 97 ==== osd_op_reply(243 gc.0 [call out=15b] v0'0 uv62 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:244 6.19 6:9a6dd8c0:gc::gc.0:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 98 ==== osd_op_reply(244 gc.0 [call out=15b] v0'0 uv62 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:245 6.19 6:9a6dd8c0:gc::gc.0:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 99 ==== osd_op_reply(245 gc.0 [call] v31'63 uv63 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2024-04-27T03:21:08.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.157+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:246 6.18 6:1e85421f:gc::gc.1:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.161+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 23 ==== osd_op_reply(246 gc.1 [call] v31'40 uv40 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.161+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:247 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.161+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 24 ==== osd_op_reply(247 gc.1 [call out=15b] v0'0 uv40 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.161+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.161+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:248 6.18 6:1e85421f:gc::gc.1:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 25 ==== osd_op_reply(248 gc.1 [call out=24b] v0'0 uv40 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:249 6.18 6:1e85421f:gc::gc.1:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 26 ==== osd_op_reply(249 gc.1 [call out=15b] v0'0 uv40 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:250 6.18 6:1e85421f:gc::gc.1:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 27 ==== osd_op_reply(250 gc.1 [call out=15b] v0'0 uv40 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.165+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:251 6.18 6:1e85421f:gc::gc.1:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.169+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 28 ==== osd_op_reply(251 gc.1 [call] v31'41 uv41 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.169+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2024-04-27T03:21:08.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.169+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:252 6.b 6:d242335b:gc::gc.2:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.169+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 59 ==== osd_op_reply(252 gc.2 [call] v31'39 uv39 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.169+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:253 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 60 ==== osd_op_reply(253 gc.2 [call out=15b] v0'0 uv39 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:254 6.b 6:d242335b:gc::gc.2:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 61 ==== osd_op_reply(254 gc.2 [call out=24b] v0'0 uv39 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:255 6.b 6:d242335b:gc::gc.2:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 62 ==== osd_op_reply(255 gc.2 [call out=15b] v0'0 uv39 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:256 6.b 6:d242335b:gc::gc.2:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 63 ==== osd_op_reply(256 gc.2 [call out=15b] v0'0 uv39 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.173+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:257 6.b 6:d242335b:gc::gc.2:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.177+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 64 ==== osd_op_reply(257 gc.2 [call] v31'40 uv40 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.177+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2024-04-27T03:21:08.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.177+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:258 6.4 6:2166d8e1:gc::gc.3:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 67 ==== osd_op_reply(258 gc.3 [call] v31'288 uv288 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:259 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 68 ==== osd_op_reply(259 gc.3 [call out=15b] v0'0 uv288 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:260 6.4 6:2166d8e1:gc::gc.3:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 69 ==== osd_op_reply(260 gc.3 [call out=24b] v0'0 uv288 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:261 6.4 6:2166d8e1:gc::gc.3:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.181+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 70 ==== osd_op_reply(261 gc.3 [call out=15b] v0'0 uv288 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:262 6.4 6:2166d8e1:gc::gc.3:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 71 ==== osd_op_reply(262 gc.3 [call out=15b] v0'0 uv288 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:263 6.4 6:2166d8e1:gc::gc.3:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 72 ==== osd_op_reply(263 gc.3 [call] v31'289 uv289 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2024-04-27T03:21:08.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.185+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:264 6.3 6:c60da066:gc::gc.4:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.189+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 100 ==== osd_op_reply(264 gc.4 [call] v31'101 uv101 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.189+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:265 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.189+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 101 ==== osd_op_reply(265 gc.4 [call out=15b] v0'0 uv101 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.189+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.189+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:266 6.3 6:c60da066:gc::gc.4:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 102 ==== osd_op_reply(266 gc.4 [call out=24b] v0'0 uv101 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:267 6.3 6:c60da066:gc::gc.4:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 103 ==== osd_op_reply(267 gc.4 [call out=15b] v0'0 uv101 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:268 6.3 6:c60da066:gc::gc.4:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 104 ==== osd_op_reply(268 gc.4 [call out=15b] v0'0 uv101 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.193+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:269 6.3 6:c60da066:gc::gc.4:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 105 ==== osd_op_reply(269 gc.4 [call] v31'102 uv102 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2024-04-27T03:21:08.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:270 6.9 6:954be8d2:gc::gc.5:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 65 ==== osd_op_reply(270 gc.5 [call] v31'71 uv71 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:271 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 66 ==== osd_op_reply(271 gc.5 [call out=15b] v0'0 uv71 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.197+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:272 6.9 6:954be8d2:gc::gc.5:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 67 ==== osd_op_reply(272 gc.5 [call out=24b] v0'0 uv71 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:273 6.9 6:954be8d2:gc::gc.5:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 68 ==== osd_op_reply(273 gc.5 [call out=15b] v0'0 uv71 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:274 6.9 6:954be8d2:gc::gc.5:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 69 ==== osd_op_reply(274 gc.5 [call out=15b] v0'0 uv71 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:275 6.9 6:954be8d2:gc::gc.5:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.201+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 70 ==== osd_op_reply(275 gc.5 [call] v31'72 uv72 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.205+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2024-04-27T03:21:08.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.205+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:276 6.e 6:748722ff:gc::gc.6:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.205+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 106 ==== osd_op_reply(276 gc.6 [call] v31'42 uv42 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.205+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:277 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.205+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 107 ==== osd_op_reply(277 gc.6 [call out=15b] v0'0 uv42 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.205+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.205+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:278 6.e 6:748722ff:gc::gc.6:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 108 ==== osd_op_reply(278 gc.6 [call out=24b] v0'0 uv42 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:279 6.e 6:748722ff:gc::gc.6:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 109 ==== osd_op_reply(279 gc.6 [call out=15b] v0'0 uv42 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:280 6.e 6:748722ff:gc::gc.6:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 110 ==== osd_op_reply(280 gc.6 [call out=15b] v0'0 uv42 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.209+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:281 6.e 6:748722ff:gc::gc.6:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.213+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 111 ==== osd_op_reply(281 gc.6 [call] v31'43 uv43 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.213+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2024-04-27T03:21:08.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.213+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:282 6.1d 6:beaa4be4:gc::gc.7:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 73 ==== osd_op_reply(282 gc.7 [call] v31'67 uv67 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:283 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 74 ==== osd_op_reply(283 gc.7 [call out=15b] v0'0 uv67 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:284 6.1d 6:beaa4be4:gc::gc.7:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 75 ==== osd_op_reply(284 gc.7 [call out=24b] v0'0 uv67 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:285 6.1d 6:beaa4be4:gc::gc.7:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 76 ==== osd_op_reply(285 gc.7 [call out=15b] v0'0 uv67 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:286 6.1d 6:beaa4be4:gc::gc.7:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 77 ==== osd_op_reply(286 gc.7 [call out=15b] v0'0 uv67 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.217+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:287 6.1d 6:beaa4be4:gc::gc.7:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.221+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 78 ==== osd_op_reply(287 gc.7 [call] v31'68 uv68 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.221+0000 7f4309a69980 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2024-04-27T03:21:08.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.221+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:288 6.12 6:4f78e9a2:gc::gc.8:head [call lock.lock in=50b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 112 ==== osd_op_reply(288 gc.8 [call] v31'33 uv33 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:289 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 113 ==== osd_op_reply(289 gc.8 [call out=15b] v0'0 uv33 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:290 6.12 6:4f78e9a2:gc::gc.8:head [call version.read in=11b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 114 ==== osd_op_reply(290 gc.8 [call out=24b] v0'0 uv33 ondisk = 0) ==== 148+0+24 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:291 6.12 6:4f78e9a2:gc::gc.8:head [call rgw.gc_list in=25b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 115 ==== osd_op_reply(291 gc.8 [call out=15b] v0'0 uv33 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2024-04-27T03:21:08.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.225+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:292 6.12 6:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.229+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 116 ==== osd_op_reply(292 gc.8 [call out=15b] v0'0 uv33 ondisk = 0) ==== 148+0+15 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.229+0000 7f4309a69980 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2024-04-27T03:21:08.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.229+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:293 6.12 6:4f78e9a2:gc::gc.8:head [call lock.unlock in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.229+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 117 ==== osd_op_reply(293 gc.8 [call] v31'34 uv34 ondisk = 0) ==== 148+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.233+0000 7f41bdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-04-27T03:21:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.233+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:294 6.4 6:22d26bf9:::data_loggenerations_metadata:head [watch unwatch cookie 94521833152144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.237+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 79 ==== osd_op_reply(294 data_loggenerations_metadata [watch unwatch cookie 94521833152144] v31'290 uv1 ondisk = 0) ==== 172+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.237+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:295 7.1f 7:f95f44c2:::notify.0:head [watch unwatch cookie 94521833131520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.237+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 118 ==== osd_op_reply(295 notify.0 [watch unwatch cookie 94521833131520] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.237+0000 7f4309a69980 20 remove_watcher() i=0 2024-04-27T03:21:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.237+0000 7f4309a69980 2 removed watcher, disabling cache 2024-04-27T03:21:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.237+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- osd_op(unknown.0.0:296 7.0 7:05bf5b68:::notify.1:head [watch unwatch cookie 94521833137312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448b3a0 2024-04-27T03:21:08.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.241+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.1 v2:172.21.15.110:6800/1860462708 80 ==== osd_op_reply(296 notify.1 [watch unwatch cookie 94521833137312] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x55f79448b3a0 2024-04-27T03:21:08.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.241+0000 7f4309a69980 20 remove_watcher() i=1 2024-04-27T03:21:08.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.241+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:297 7.15 7:a93a5511:::notify.2:head [watch unwatch cookie 94521833139520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.245+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 119 ==== osd_op_reply(297 notify.2 [watch unwatch cookie 94521833139520] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.245+0000 7f4309a69980 20 remove_watcher() i=2 2024-04-27T03:21:08.254 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.245+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:298 7.e 7:7759931f:::notify.3:head [watch unwatch cookie 94521833129008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x7f4300014030 2024-04-27T03:21:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.245+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 29 ==== osd_op_reply(298 notify.3 [watch unwatch cookie 94521833129008] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.245+0000 7f4309a69980 20 remove_watcher() i=3 2024-04-27T03:21:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.245+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- osd_op(unknown.0.0:299 7.d 7:b4812045:::notify.4:head [watch unwatch cookie 94521833144672] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f79448f890 2024-04-27T03:21:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.249+0000 7f4307d85640 1 -- 172.21.15.110:0/939126641 <== osd.0 v2:172.21.15.77:6800/3496324319 120 ==== osd_op_reply(299 notify.4 [watch unwatch cookie 94521833144672] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f4300038030 con 0x55f79448f890 2024-04-27T03:21:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.249+0000 7f4309a69980 20 remove_watcher() i=4 2024-04-27T03:21:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.249+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:300 7.3 7:c609908c:::notify.5:head [watch unwatch cookie 94521833148448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f7944a90e0 con 0x55f7944948c0 2024-04-27T03:21:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.249+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 71 ==== osd_op_reply(300 notify.5 [watch unwatch cookie 94521833148448] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.249+0000 7f4309a69980 20 remove_watcher() i=5 2024-04-27T03:21:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.249+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- osd_op(unknown.0.0:301 7.14 7:2b04a3e9:::notify.6:head [watch unwatch cookie 94521833141264] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449a4a0 con 0x55f7944948c0 2024-04-27T03:21:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4307584640 1 -- 172.21.15.110:0/939126641 <== osd.3 v2:172.21.15.110:6816/3916647200 72 ==== osd_op_reply(301 notify.6 [watch unwatch cookie 94521833141264] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f00d3020 con 0x55f7944948c0 2024-04-27T03:21:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 20 remove_watcher() i=6 2024-04-27T03:21:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- osd_op(unknown.0.0:302 7.9 7:93e5b521:::notify.7:head [watch unwatch cookie 94521833153808] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) -- 0x55f79449ad40 con 0x7f4300014030 2024-04-27T03:21:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 <== osd.2 v2:172.21.15.110:6808/3575017494 30 ==== osd_op_reply(302 notify.7 [watch unwatch cookie 94521833153808] v31'285 uv283 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f42f800f620 con 0x7f4300014030 2024-04-27T03:21:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 20 remove_watcher() i=7 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f79448f890 msgr2=0x55f794491c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f79448f890 0x55f794491c80 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79448b3a0 msgr2=0x55f79448d860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79448b3a0 0x55f79448d860 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4300014030 msgr2=0x7f4300007950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f4300014030 0x7f4300007950 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f7944948c0 msgr2=0x55f794496cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.253+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x55f7944948c0 0x55f794496cb0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41d0034650 msgr2=0x7f41d0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41d0034650 0x7f41d0036b10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f42f00a9960 tx=0x7f42f00dd000 comp rx=0 tx=0).stop 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4306d83640 1 -- 172.21.15.110:0/939126641 reap_dead start 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 msgr2=0x55f79447fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f79447fda0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4300015f40 tx=0x7f430004a850 comp rx=0 tx=0).stop 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 shutdown_connections 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/939126641 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f794478870 0x55f79447fda0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 >> 172.21.15.110:0/939126641 conn(0x55f794474c00 msgr2=0x55f79447a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 shutdown_connections 2024-04-27T03:21:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/939126641 wait complete. 2024-04-27T03:21:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41e0034650 msgr2=0x7f41e0036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41e0034650 0x7f41e0036b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55f79445bb10 tx=0x7f42f00bef70 comp rx=0 tx=0).stop 2024-04-27T03:21:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79446b6c0 msgr2=0x55f79446bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79446b6c0 0x55f79446bae0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f430000dfc0 tx=0x7f43000178e0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 shutdown_connections 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41e0034650 0x7f41e0036b10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3844520047 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79446b6c0 0x55f79446bae0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 >> 172.21.15.110:0/3844520047 conn(0x55f7944604d0 msgr2=0x55f794460fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 shutdown_connections 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3844520047 wait complete. 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f794458af0 msgr2=0x55f79445aee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f794458af0 0x55f79445aee0 crc :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79435fa70 msgr2=0x55f794457720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79435fa70 0x55f794457720 crc :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41ec034650 msgr2=0x7f41ec036b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41ec034650 0x7f41ec036b10 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f42f8000f70 tx=0x7f42f8001380 comp rx=0 tx=0).stop 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 msgr2=0x55f7944508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f7944508a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f42f001b170 tx=0x7f42f00180e0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 shutdown_connections 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x55f794458af0 0x55f79445aee0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x55f79435fa70 0x55f794457720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f41ec034650 0x7f41ec036b10 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 --2- 172.21.15.110:0/3782688643 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x55f79435f150 0x55f7944508a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 >> 172.21.15.110:0/3782688643 conn(0x55f79435aa20 msgr2=0x55f7944450b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 shutdown_connections 2024-04-27T03:21:08.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-27T03:21:08.257+0000 7f4309a69980 1 -- 172.21.15.110:0/3782688643 wait complete. 2024-04-27T03:21:08.285 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-27T03:21:08.297 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-27T03:21:08.297 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.414+0000 7f507c94c640 1 Processor -- start 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.414+0000 7f507c94c640 1 -- start start 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f507806d4d0 0x7f507806d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5078071ba0 con 0x7f507806ddf0 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5078098ea0 con 0x7f507806d4d0 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f507806d4d0 0x7f507806d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507659c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f507806d4d0 0x7f507806d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56732/0 (socket says 172.21.15.77:56732) 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f5075d9b640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f507806ddf0 0x7f5078096060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46682/0 (socket says 172.21.15.77:46682) 2024-04-27T03:21:08.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507659c640 1 -- 172.21.15.77:0/4141101754 learned_addr learned my addr 172.21.15.77:0/4141101754 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:08.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507659c640 1 -- 172.21.15.77:0/4141101754 >> v1:172.21.15.77:6789/0 conn(0x7f507806ddf0 legacy=0x7f5078096060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507659c640 1 -- 172.21.15.77:0/4141101754 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507806e970 con 0x7f507806d4d0 2024-04-27T03:21:08.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507659c640 1 --2- 172.21.15.77:0/4141101754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507806d8b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f50680097f0 tx=0x7f506801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36930e4986d27906 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:08.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507559a640 1 -- 172.21.15.77:0/4141101754 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50680b0070 con 0x7f507806d4d0 2024-04-27T03:21:08.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507559a640 1 -- 172.21.15.77:0/4141101754 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50680ab530 con 0x7f507806d4d0 2024-04-27T03:21:08.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507559a640 1 -- 172.21.15.77:0/4141101754 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50680ab830 con 0x7f507806d4d0 2024-04-27T03:21:08.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 -- 172.21.15.77:0/4141101754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 msgr2=0x7f507806d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 --2- 172.21.15.77:0/4141101754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507806d8b0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f50680097f0 tx=0x7f506801c430 comp rx=0 tx=0).stop 2024-04-27T03:21:08.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 -- 172.21.15.77:0/4141101754 shutdown_connections 2024-04-27T03:21:08.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 --2- 172.21.15.77:0/4141101754 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507806d8b0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.431 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 -- 172.21.15.77:0/4141101754 >> 172.21.15.77:0/4141101754 conn(0x7f5078099440 msgr2=0x7f5078099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 -- 172.21.15.77:0/4141101754 shutdown_connections 2024-04-27T03:21:08.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.418+0000 7f507c94c640 1 -- 172.21.15.77:0/4141101754 wait complete. 2024-04-27T03:21:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507c94c640 1 Processor -- start 2024-04-27T03:21:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507c94c640 1 -- start start 2024-04-27T03:21:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507c94c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507808a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507c94c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f5078098740 con 0x7f507806d4d0 2024-04-27T03:21:08.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507808a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507659c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507808a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56740/0 (socket says 172.21.15.77:56740) 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507659c640 1 -- 172.21.15.77:0/1636229398 learned_addr learned my addr 172.21.15.77:0/1636229398 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507659c640 1 -- 172.21.15.77:0/1636229398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5078126ba0 con 0x7f507806d4d0 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507659c640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507808a200 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f506801a060 tx=0x7f506801ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f506effd640 1 -- 172.21.15.77:0/1636229398 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50680ba020 con 0x7f507806d4d0 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f506effd640 1 -- 172.21.15.77:0/1636229398 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50680b6420 con 0x7f507806d4d0 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507808b670 con 0x7f507806d4d0 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f506effd640 1 -- 172.21.15.77:0/1636229398 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50680b0040 con 0x7f507806d4d0 2024-04-27T03:21:08.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5078126e40 con 0x7f507806d4d0 2024-04-27T03:21:08.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f506effd640 1 -- 172.21.15.77:0/1636229398 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f50680bd020 con 0x7f507806d4d0 2024-04-27T03:21:08.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5078062e50 con 0x7f507806d4d0 2024-04-27T03:21:08.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f506effd640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5058035db0 0x7f5058038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:08.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f506effd640 1 -- 172.21.15.77:0/1636229398 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f50680e3490 con 0x7f507806d4d0 2024-04-27T03:21:08.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.422+0000 7f5075d9b640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5058035db0 0x7f5058038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:08.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.426+0000 7f5075d9b640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5058035db0 0x7f5058038270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5060001d20 tx=0x7f5060007d90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:08.439 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.430+0000 7f506effd640 1 -- 172.21.15.77:0/1636229398 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f50680c4030 con 0x7f507806d4d0 2024-04-27T03:21:08.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.582+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f507808b270 con 0x7f5058035db0 2024-04-27T03:21:08.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.742+0000 7f506effd640 1 -- 172.21.15.77:0/1636229398 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+665756 (secure 0 0 0) 0x7f506000c820 con 0x7f5058035db0 2024-04-27T03:21:08.761 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:21:08.767 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5058035db0 msgr2=0x7f5058038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5058035db0 0x7f5058038270 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5060001d20 tx=0x7f5060007d90 comp rx=0 tx=0).stop 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 msgr2=0x7f507808a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507808a200 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f506801a060 tx=0x7f506801ac70 comp rx=0 tx=0).stop 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 shutdown_connections 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5058035db0 0x7f5058038270 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 --2- 172.21.15.77:0/1636229398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f507806d4d0 0x7f507808a200 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 >> 172.21.15.77:0/1636229398 conn(0x7f5078099440 msgr2=0x7f507808ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 shutdown_connections 2024-04-27T03:21:08.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.750+0000 7f507c94c640 1 -- 172.21.15.77:0/1636229398 wait complete. 2024-04-27T03:21:08.782 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-27T03:21:07.732296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26412789,"num_objects":259,"num_object_clones":0,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":5177,"num_read_kb":4722,"num_write":2298,"num_write_kb":61373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4672,"ondisk_log_size":4672,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":190324,"kb_used_data":40660,"kb_used_omap":6,"kb_used_meta":149625,"kb_avail":377297036,"statfs":{"total":386547056640,"available":386352164864,"internally_reserved":0,"allocated":41635840,"data_stored":40495458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":153216790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25811687,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":510,"num_read_kb":465,"num_write":386,"num_write_kb":29372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005707"},"pg_stats":[{"pgid":"3.3f","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741206+0000","last_change":"2024-04-27T03:17:47.025622+0000","last_active":"2024-04-27T03:19:52.741206+0000","last_peered":"2024-04-27T03:19:52.741206+0000","last_clean":"2024-04-27T03:19:52.741206+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:19:52.741206+0000","last_undegraded":"2024-04-27T03:19:52.741206+0000","last_fullsized":"2024-04-27T03:19:52.741206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:56.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'5","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246474+0000","last_change":"2024-04-27T03:17:49.019025+0000","last_active":"2024-04-27T03:21:01.246474+0000","last_peered":"2024-04-27T03:21:01.246474+0000","last_clean":"2024-04-27T03:21:01.246474+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:01.246474+0000","last_undegraded":"2024-04-27T03:21:01.246474+0000","last_fullsized":"2024-04-27T03:21:01.246474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385062+0000","last_change":"2024-04-27T03:17:47.023458+0000","last_active":"2024-04-27T03:19:35.385062+0000","last_peered":"2024-04-27T03:19:35.385062+0000","last_clean":"2024-04-27T03:19:35.385062+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:19:35.385062+0000","last_undegraded":"2024-04-27T03:19:35.385062+0000","last_fullsized":"2024-04-27T03:19:35.385062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810083+0000","last_change":"2024-04-27T03:17:49.020940+0000","last_active":"2024-04-27T03:19:34.810083+0000","last_peered":"2024-04-27T03:19:34.810083+0000","last_clean":"2024-04-27T03:19:34.810083+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:19:34.810083+0000","last_undegraded":"2024-04-27T03:19:34.810083+0000","last_fullsized":"2024-04-27T03:19:34.810083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811946+0000","last_change":"2024-04-27T03:17:47.039594+0000","last_active":"2024-04-27T03:19:34.811946+0000","last_peered":"2024-04-27T03:19:34.811946+0000","last_clean":"2024-04-27T03:19:34.811946+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:19:34.811946+0000","last_undegraded":"2024-04-27T03:19:34.811946+0000","last_fullsized":"2024-04-27T03:19:34.811946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111615+0000","last_change":"2024-04-27T03:17:49.016480+0000","last_active":"2024-04-27T03:19:56.111615+0000","last_peered":"2024-04-27T03:19:56.111615+0000","last_clean":"2024-04-27T03:19:56.111615+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:19:56.111615+0000","last_undegraded":"2024-04-27T03:19:56.111615+0000","last_fullsized":"2024-04-27T03:19:56.111615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439532+0000","last_change":"2024-04-27T03:17:47.021409+0000","last_active":"2024-04-27T03:19:35.439532+0000","last_peered":"2024-04-27T03:19:35.439532+0000","last_clean":"2024-04-27T03:19:35.439532+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:19:35.439532+0000","last_undegraded":"2024-04-27T03:19:35.439532+0000","last_fullsized":"2024-04-27T03:19:35.439532+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:24.826641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948899+0000","last_change":"2024-04-27T03:17:49.021774+0000","last_active":"2024-04-27T03:21:00.948899+0000","last_peered":"2024-04-27T03:21:00.948899+0000","last_clean":"2024-04-27T03:21:00.948899+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:00.948899+0000","last_undegraded":"2024-04-27T03:21:00.948899+0000","last_fullsized":"2024-04-27T03:21:00.948899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817333+0000","last_change":"2024-04-27T03:17:47.015865+0000","last_active":"2024-04-27T03:19:34.817333+0000","last_peered":"2024-04-27T03:19:34.817333+0000","last_clean":"2024-04-27T03:19:34.817333+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:19:34.817333+0000","last_undegraded":"2024-04-27T03:19:34.817333+0000","last_fullsized":"2024-04-27T03:19:34.817333+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439003+0000","last_change":"2024-04-27T03:18:34.776780+0000","last_active":"2024-04-27T03:19:35.439003+0000","last_peered":"2024-04-27T03:19:35.439003+0000","last_clean":"2024-04-27T03:19:35.439003+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:19:35.439003+0000","last_undegraded":"2024-04-27T03:19:35.439003+0000","last_fullsized":"2024-04-27T03:19:35.439003+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:30.737857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.111562+0000","last_change":"2024-04-27T03:17:47.026206+0000","last_active":"2024-04-27T03:20:00.111562+0000","last_peered":"2024-04-27T03:20:00.111562+0000","last_clean":"2024-04-27T03:20:00.111562+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:20:00.111562+0000","last_undegraded":"2024-04-27T03:20:00.111562+0000","last_fullsized":"2024-04-27T03:20:00.111562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'11","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742400+0000","last_change":"2024-04-27T03:17:49.015210+0000","last_active":"2024-04-27T03:19:57.742400+0000","last_peered":"2024-04-27T03:19:57.742400+0000","last_clean":"2024-04-27T03:19:57.742400+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:19:57.742400+0000","last_undegraded":"2024-04-27T03:19:57.742400+0000","last_fullsized":"2024-04-27T03:19:57.742400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811270+0000","last_change":"2024-04-27T03:17:47.036460+0000","last_active":"2024-04-27T03:19:34.811270+0000","last_peered":"2024-04-27T03:19:34.811270+0000","last_clean":"2024-04-27T03:19:34.811270+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:19:34.811270+0000","last_undegraded":"2024-04-27T03:19:34.811270+0000","last_fullsized":"2024-04-27T03:19:34.811270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:56.284851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298837+0000","last_change":"2024-04-27T03:17:49.024190+0000","last_active":"2024-04-27T03:21:01.298837+0000","last_peered":"2024-04-27T03:21:01.298837+0000","last_clean":"2024-04-27T03:21:01.298837+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:01.298837+0000","last_undegraded":"2024-04-27T03:21:01.298837+0000","last_fullsized":"2024-04-27T03:21:01.298837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810250+0000","last_change":"2024-04-27T03:17:47.013877+0000","last_active":"2024-04-27T03:19:34.810250+0000","last_peered":"2024-04-27T03:19:34.810250+0000","last_clean":"2024-04-27T03:19:34.810250+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:19:34.810250+0000","last_undegraded":"2024-04-27T03:19:34.810250+0000","last_fullsized":"2024-04-27T03:19:34.810250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.444094+0000","last_change":"2024-04-27T03:17:49.020265+0000","last_active":"2024-04-27T03:19:35.444094+0000","last_peered":"2024-04-27T03:19:35.444094+0000","last_clean":"2024-04-27T03:19:35.444094+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:19:35.444094+0000","last_undegraded":"2024-04-27T03:19:35.444094+0000","last_fullsized":"2024-04-27T03:19:35.444094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:23.713924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298707+0000","last_change":"2024-04-27T03:17:49.019484+0000","last_active":"2024-04-27T03:21:01.298707+0000","last_peered":"2024-04-27T03:21:01.298707+0000","last_clean":"2024-04-27T03:21:01.298707+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:01.298707+0000","last_undegraded":"2024-04-27T03:21:01.298707+0000","last_fullsized":"2024-04-27T03:21:01.298707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385569+0000","last_change":"2024-04-27T03:17:47.027209+0000","last_active":"2024-04-27T03:19:35.385569+0000","last_peered":"2024-04-27T03:19:35.385569+0000","last_clean":"2024-04-27T03:19:35.385569+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:19:35.385569+0000","last_undegraded":"2024-04-27T03:19:35.385569+0000","last_fullsized":"2024-04-27T03:19:35.385569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814961+0000","last_change":"2024-04-27T03:17:49.013701+0000","last_active":"2024-04-27T03:19:34.814961+0000","last_peered":"2024-04-27T03:19:34.814961+0000","last_clean":"2024-04-27T03:19:34.814961+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:19:34.814961+0000","last_undegraded":"2024-04-27T03:19:34.814961+0000","last_fullsized":"2024-04-27T03:19:34.814961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.443009+0000","last_change":"2024-04-27T03:17:47.015536+0000","last_active":"2024-04-27T03:19:35.443009+0000","last_peered":"2024-04-27T03:19:35.443009+0000","last_clean":"2024-04-27T03:19:35.443009+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:19:35.443009+0000","last_undegraded":"2024-04-27T03:19:35.443009+0000","last_fullsized":"2024-04-27T03:19:35.443009+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:51:44.696219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.809825+0000","last_change":"2024-04-27T03:17:49.011747+0000","last_active":"2024-04-27T03:21:00.809825+0000","last_peered":"2024-04-27T03:21:00.809825+0000","last_clean":"2024-04-27T03:21:00.809825+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:21:00.809825+0000","last_undegraded":"2024-04-27T03:21:00.809825+0000","last_fullsized":"2024-04-27T03:21:00.809825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820563+0000","last_change":"2024-04-27T03:17:47.023848+0000","last_active":"2024-04-27T03:19:34.820563+0000","last_peered":"2024-04-27T03:19:34.820563+0000","last_clean":"2024-04-27T03:19:34.820563+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:19:34.820563+0000","last_undegraded":"2024-04-27T03:19:34.820563+0000","last_fullsized":"2024-04-27T03:19:34.820563+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.231670+0000","last_change":"2024-04-27T03:17:49.019463+0000","last_active":"2024-04-27T03:21:01.231670+0000","last_peered":"2024-04-27T03:21:01.231670+0000","last_clean":"2024-04-27T03:21:01.231670+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:01.231670+0000","last_undegraded":"2024-04-27T03:21:01.231670+0000","last_fullsized":"2024-04-27T03:21:01.231670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:53.154789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816701+0000","last_change":"2024-04-27T03:17:47.012402+0000","last_active":"2024-04-27T03:19:34.816701+0000","last_peered":"2024-04-27T03:19:34.816701+0000","last_clean":"2024-04-27T03:19:34.816701+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:19:34.816701+0000","last_undegraded":"2024-04-27T03:19:34.816701+0000","last_fullsized":"2024-04-27T03:19:34.816701+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146748+0000","last_change":"2024-04-27T03:17:49.008483+0000","last_active":"2024-04-27T03:21:01.146748+0000","last_peered":"2024-04-27T03:21:01.146748+0000","last_clean":"2024-04-27T03:21:01.146748+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:01.146748+0000","last_undegraded":"2024-04-27T03:21:01.146748+0000","last_fullsized":"2024-04-27T03:21:01.146748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:11.803924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812204+0000","last_change":"2024-04-27T03:17:47.036082+0000","last_active":"2024-04-27T03:19:34.812204+0000","last_peered":"2024-04-27T03:19:34.812204+0000","last_clean":"2024-04-27T03:19:34.812204+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:19:34.812204+0000","last_undegraded":"2024-04-27T03:19:34.812204+0000","last_fullsized":"2024-04-27T03:19:34.812204+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164153+0000","last_change":"2024-04-27T03:17:49.005822+0000","last_active":"2024-04-27T03:21:01.164153+0000","last_peered":"2024-04-27T03:21:01.164153+0000","last_clean":"2024-04-27T03:21:01.164153+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:01.164153+0000","last_undegraded":"2024-04-27T03:21:01.164153+0000","last_fullsized":"2024-04-27T03:21:01.164153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:40.042792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811583+0000","last_change":"2024-04-27T03:17:47.033466+0000","last_active":"2024-04-27T03:19:34.811583+0000","last_peered":"2024-04-27T03:19:34.811583+0000","last_clean":"2024-04-27T03:19:34.811583+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:19:34.811583+0000","last_undegraded":"2024-04-27T03:19:34.811583+0000","last_fullsized":"2024-04-27T03:19:34.811583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'37","reported_seq":105,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.190758+0000","last_change":"2024-04-27T03:17:57.088698+0000","last_active":"2024-04-27T03:21:01.190758+0000","last_peered":"2024-04-27T03:21:01.190758+0000","last_clean":"2024-04-27T03:21:01.190758+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:21:01.190758+0000","last_undegraded":"2024-04-27T03:21:01.190758+0000","last_fullsized":"2024-04-27T03:21:01.190758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:57.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233462+0000","last_change":"2024-04-27T03:17:49.015482+0000","last_active":"2024-04-27T03:21:01.233462+0000","last_peered":"2024-04-27T03:21:01.233462+0000","last_clean":"2024-04-27T03:21:01.233462+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:01.233462+0000","last_undegraded":"2024-04-27T03:21:01.233462+0000","last_fullsized":"2024-04-27T03:21:01.233462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811777+0000","last_change":"2024-04-27T03:17:47.018824+0000","last_active":"2024-04-27T03:19:34.811777+0000","last_peered":"2024-04-27T03:19:34.811777+0000","last_clean":"2024-04-27T03:19:34.811777+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:19:34.811777+0000","last_undegraded":"2024-04-27T03:19:34.811777+0000","last_fullsized":"2024-04-27T03:19:34.811777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815532+0000","last_change":"2024-04-27T03:17:51.032891+0000","last_active":"2024-04-27T03:19:34.815532+0000","last_peered":"2024-04-27T03:19:34.815532+0000","last_clean":"2024-04-27T03:19:34.815532+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:19:34.815532+0000","last_undegraded":"2024-04-27T03:19:34.815532+0000","last_fullsized":"2024-04-27T03:19:34.815532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'43","reported_seq":101,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.274412+0000","last_change":"2024-04-27T03:17:53.040688+0000","last_active":"2024-04-27T03:19:58.274412+0000","last_peered":"2024-04-27T03:19:58.274412+0000","last_clean":"2024-04-27T03:19:58.274412+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:19:58.274412+0000","last_undegraded":"2024-04-27T03:19:58.274412+0000","last_fullsized":"2024-04-27T03:19:58.274412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819799+0000","last_change":"2024-04-27T03:17:55.072849+0000","last_active":"2024-04-27T03:19:34.819799+0000","last_peered":"2024-04-27T03:19:34.819799+0000","last_clean":"2024-04-27T03:19:34.819799+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:19:34.819799+0000","last_undegraded":"2024-04-27T03:19:34.819799+0000","last_fullsized":"2024-04-27T03:19:34.819799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.848140+0000","last_change":"2024-04-27T03:17:57.061550+0000","last_active":"2024-04-27T03:21:00.848140+0000","last_peered":"2024-04-27T03:21:00.848140+0000","last_clean":"2024-04-27T03:21:00.848140+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:00.848140+0000","last_undegraded":"2024-04-27T03:21:00.848140+0000","last_fullsized":"2024-04-27T03:21:00.848140+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:54:11.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814870+0000","last_change":"2024-04-27T03:17:49.017246+0000","last_active":"2024-04-27T03:19:34.814870+0000","last_peered":"2024-04-27T03:19:34.814870+0000","last_clean":"2024-04-27T03:19:34.814870+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:19:34.814870+0000","last_undegraded":"2024-04-27T03:19:34.814870+0000","last_fullsized":"2024-04-27T03:19:34.814870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:15.229307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436921+0000","last_change":"2024-04-27T03:17:47.032159+0000","last_active":"2024-04-27T03:19:35.436921+0000","last_peered":"2024-04-27T03:19:35.436921+0000","last_clean":"2024-04-27T03:19:35.436921+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:19:35.436921+0000","last_undegraded":"2024-04-27T03:19:35.436921+0000","last_fullsized":"2024-04-27T03:19:35.436921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:43.992214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815036+0000","last_change":"2024-04-27T03:17:51.012211+0000","last_active":"2024-04-27T03:19:34.815036+0000","last_peered":"2024-04-27T03:19:34.815036+0000","last_clean":"2024-04-27T03:19:34.815036+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:19:34.815036+0000","last_undegraded":"2024-04-27T03:19:34.815036+0000","last_fullsized":"2024-04-27T03:19:34.815036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"27'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384958+0000","last_change":"2024-04-27T03:17:53.049040+0000","last_active":"2024-04-27T03:19:35.384958+0000","last_peered":"2024-04-27T03:19:35.384958+0000","last_clean":"2024-04-27T03:19:35.384958+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:19:35.384958+0000","last_undegraded":"2024-04-27T03:19:35.384958+0000","last_fullsized":"2024-04-27T03:19:35.384958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:34.768461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819788+0000","last_change":"2024-04-27T03:17:55.058791+0000","last_active":"2024-04-27T03:19:34.819788+0000","last_peered":"2024-04-27T03:19:34.819788+0000","last_clean":"2024-04-27T03:19:34.819788+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:19:34.819788+0000","last_undegraded":"2024-04-27T03:19:34.819788+0000","last_fullsized":"2024-04-27T03:19:34.819788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809843+0000","last_change":"2024-04-27T03:17:57.074831+0000","last_active":"2024-04-27T03:19:34.809843+0000","last_peered":"2024-04-27T03:19:34.809843+0000","last_clean":"2024-04-27T03:19:34.809843+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:19:34.809843+0000","last_undegraded":"2024-04-27T03:19:34.809843+0000","last_fullsized":"2024-04-27T03:19:34.809843+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'15","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.165698+0000","last_change":"2024-04-27T03:17:49.022969+0000","last_active":"2024-04-27T03:21:01.165698+0000","last_peered":"2024-04-27T03:21:01.165698+0000","last_clean":"2024-04-27T03:21:01.165698+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:01.165698+0000","last_undegraded":"2024-04-27T03:21:01.165698+0000","last_fullsized":"2024-04-27T03:21:01.165698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814358+0000","last_change":"2024-04-27T03:17:47.009721+0000","last_active":"2024-04-27T03:19:34.814358+0000","last_peered":"2024-04-27T03:19:34.814358+0000","last_clean":"2024-04-27T03:19:34.814358+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:19:34.814358+0000","last_undegraded":"2024-04-27T03:19:34.814358+0000","last_fullsized":"2024-04-27T03:19:34.814358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814852+0000","last_change":"2024-04-27T03:17:51.034093+0000","last_active":"2024-04-27T03:19:34.814852+0000","last_peered":"2024-04-27T03:19:34.814852+0000","last_clean":"2024-04-27T03:19:34.814852+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:19:34.814852+0000","last_undegraded":"2024-04-27T03:19:34.814852+0000","last_fullsized":"2024-04-27T03:19:34.814852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"27'22","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440526+0000","last_change":"2024-04-27T03:17:53.032443+0000","last_active":"2024-04-27T03:19:35.440526+0000","last_peered":"2024-04-27T03:19:35.440526+0000","last_clean":"2024-04-27T03:19:35.440526+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:19:35.440526+0000","last_undegraded":"2024-04-27T03:19:35.440526+0000","last_fullsized":"2024-04-27T03:19:35.440526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:35.232648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'282","reported_seq":426,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277953+0000","last_change":"2024-04-27T03:17:55.069049+0000","last_active":"2024-04-27T03:21:01.277953+0000","last_peered":"2024-04-27T03:21:01.277953+0000","last_clean":"2024-04-27T03:21:01.277953+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:01.277953+0000","last_undegraded":"2024-04-27T03:21:01.277953+0000","last_fullsized":"2024-04-27T03:21:01.277953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438345+0000","last_change":"2024-04-27T03:17:57.072058+0000","last_active":"2024-04-27T03:19:35.438345+0000","last_peered":"2024-04-27T03:19:35.438345+0000","last_clean":"2024-04-27T03:19:35.438345+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:19:35.438345+0000","last_undegraded":"2024-04-27T03:19:35.438345+0000","last_fullsized":"2024-04-27T03:19:35.438345+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:56.749229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950160+0000","last_change":"2024-04-27T03:17:49.017573+0000","last_active":"2024-04-27T03:21:00.950160+0000","last_peered":"2024-04-27T03:21:00.950160+0000","last_clean":"2024-04-27T03:21:00.950160+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:00.950160+0000","last_undegraded":"2024-04-27T03:21:00.950160+0000","last_fullsized":"2024-04-27T03:21:00.950160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:04.132727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813820+0000","last_change":"2024-04-27T03:17:47.003169+0000","last_active":"2024-04-27T03:19:34.813820+0000","last_peered":"2024-04-27T03:19:34.813820+0000","last_clean":"2024-04-27T03:19:34.813820+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:19:34.813820+0000","last_undegraded":"2024-04-27T03:19:34.813820+0000","last_fullsized":"2024-04-27T03:19:34.813820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822747+0000","last_change":"2024-04-27T03:17:51.020242+0000","last_active":"2024-04-27T03:19:34.822747+0000","last_peered":"2024-04-27T03:19:34.822747+0000","last_clean":"2024-04-27T03:19:34.822747+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:19:34.822747+0000","last_undegraded":"2024-04-27T03:19:34.822747+0000","last_fullsized":"2024-04-27T03:19:34.822747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'36","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.265097+0000","last_change":"2024-04-27T03:17:53.038866+0000","last_active":"2024-04-27T03:19:58.265097+0000","last_peered":"2024-04-27T03:19:58.265097+0000","last_clean":"2024-04-27T03:19:58.265097+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:19:58.265097+0000","last_undegraded":"2024-04-27T03:19:58.265097+0000","last_fullsized":"2024-04-27T03:19:58.265097+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:23.026213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'282","reported_seq":406,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.306917+0000","last_change":"2024-04-27T03:17:55.074657+0000","last_active":"2024-04-27T03:21:01.306917+0000","last_peered":"2024-04-27T03:21:01.306917+0000","last_clean":"2024-04-27T03:21:01.306917+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:01.306917+0000","last_undegraded":"2024-04-27T03:21:01.306917+0000","last_fullsized":"2024-04-27T03:21:01.306917+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'31","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246782+0000","last_change":"2024-04-27T03:18:34.790257+0000","last_active":"2024-04-27T03:21:01.246782+0000","last_peered":"2024-04-27T03:21:01.246782+0000","last_clean":"2024-04-27T03:21:01.246782+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:21:01.246782+0000","last_undegraded":"2024-04-27T03:21:01.246782+0000","last_fullsized":"2024-04-27T03:21:01.246782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:11.176662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.716875+0000","last_change":"2024-04-27T03:17:47.013403+0000","last_active":"2024-04-27T03:21:00.716875+0000","last_peered":"2024-04-27T03:21:00.716875+0000","last_clean":"2024-04-27T03:21:00.716875+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:21:00.716875+0000","last_undegraded":"2024-04-27T03:21:00.716875+0000","last_fullsized":"2024-04-27T03:21:00.716875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:47.849076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377293+0000","last_change":"2024-04-27T03:17:51.027190+0000","last_active":"2024-04-27T03:19:35.377293+0000","last_peered":"2024-04-27T03:19:35.377293+0000","last_clean":"2024-04-27T03:19:35.377293+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:19:35.377293+0000","last_undegraded":"2024-04-27T03:19:35.377293+0000","last_fullsized":"2024-04-27T03:19:35.377293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:50.708783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'47","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.221245+0000","last_change":"2024-04-27T03:17:53.060562+0000","last_active":"2024-04-27T03:20:00.221245+0000","last_peered":"2024-04-27T03:20:00.221245+0000","last_clean":"2024-04-27T03:20:00.221245+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:20:00.221245+0000","last_undegraded":"2024-04-27T03:20:00.221245+0000","last_fullsized":"2024-04-27T03:20:00.221245+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":41,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380552+0000","last_change":"2024-04-27T03:17:55.063723+0000","last_active":"2024-04-27T03:19:35.380552+0000","last_peered":"2024-04-27T03:19:35.380552+0000","last_clean":"2024-04-27T03:19:35.380552+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:19:35.380552+0000","last_undegraded":"2024-04-27T03:19:35.380552+0000","last_fullsized":"2024-04-27T03:19:35.380552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:59.616012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439014+0000","last_change":"2024-04-27T03:17:57.086011+0000","last_active":"2024-04-27T03:19:35.439014+0000","last_peered":"2024-04-27T03:19:35.439014+0000","last_clean":"2024-04-27T03:19:35.439014+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:19:35.439014+0000","last_undegraded":"2024-04-27T03:19:35.439014+0000","last_fullsized":"2024-04-27T03:19:35.439014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:11.219539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299288+0000","last_change":"2024-04-27T03:17:49.003206+0000","last_active":"2024-04-27T03:21:01.299288+0000","last_peered":"2024-04-27T03:21:01.299288+0000","last_clean":"2024-04-27T03:21:01.299288+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:01.299288+0000","last_undegraded":"2024-04-27T03:21:01.299288+0000","last_fullsized":"2024-04-27T03:21:01.299288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:42.242631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820229+0000","last_change":"2024-04-27T03:17:46.995080+0000","last_active":"2024-04-27T03:19:34.820229+0000","last_peered":"2024-04-27T03:19:34.820229+0000","last_clean":"2024-04-27T03:19:34.820229+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:19:34.820229+0000","last_undegraded":"2024-04-27T03:19:34.820229+0000","last_fullsized":"2024-04-27T03:19:34.820229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437771+0000","last_change":"2024-04-27T03:17:51.036969+0000","last_active":"2024-04-27T03:19:35.437771+0000","last_peered":"2024-04-27T03:19:35.437771+0000","last_clean":"2024-04-27T03:19:35.437771+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:19:35.437771+0000","last_undegraded":"2024-04-27T03:19:35.437771+0000","last_fullsized":"2024-04-27T03:19:35.437771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:48:00.532358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'37","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.245893+0000","last_change":"2024-04-27T03:17:53.041431+0000","last_active":"2024-04-27T03:19:58.245893+0000","last_peered":"2024-04-27T03:19:58.245893+0000","last_clean":"2024-04-27T03:19:58.245893+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:19:58.245893+0000","last_undegraded":"2024-04-27T03:19:58.245893+0000","last_fullsized":"2024-04-27T03:19:58.245893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809982+0000","last_change":"2024-04-27T03:17:55.075008+0000","last_active":"2024-04-27T03:19:34.809982+0000","last_peered":"2024-04-27T03:19:34.809982+0000","last_clean":"2024-04-27T03:19:34.809982+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:19:34.809982+0000","last_undegraded":"2024-04-27T03:19:34.809982+0000","last_fullsized":"2024-04-27T03:19:34.809982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:01.297073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"29'5","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382753+0000","last_change":"2024-04-27T03:17:57.084457+0000","last_active":"2024-04-27T03:19:35.382753+0000","last_peered":"2024-04-27T03:19:35.382753+0000","last_clean":"2024-04-27T03:19:35.382753+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:19:35.382753+0000","last_undegraded":"2024-04-27T03:19:35.382753+0000","last_fullsized":"2024-04-27T03:19:35.382753+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:34.604396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.827907+0000","last_change":"2024-04-27T03:18:34.767220+0000","last_active":"2024-04-27T03:21:00.827907+0000","last_peered":"2024-04-27T03:21:00.827907+0000","last_clean":"2024-04-27T03:21:00.827907+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:00.827907+0000","last_undegraded":"2024-04-27T03:21:00.827907+0000","last_fullsized":"2024-04-27T03:21:00.827907+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:38.334837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820841+0000","last_change":"2024-04-27T03:17:47.028376+0000","last_active":"2024-04-27T03:19:34.820841+0000","last_peered":"2024-04-27T03:19:34.820841+0000","last_clean":"2024-04-27T03:19:34.820841+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:19:34.820841+0000","last_undegraded":"2024-04-27T03:19:34.820841+0000","last_fullsized":"2024-04-27T03:19:34.820841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:18.893441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438808+0000","last_change":"2024-04-27T03:17:51.037525+0000","last_active":"2024-04-27T03:19:35.438808+0000","last_peered":"2024-04-27T03:19:35.438808+0000","last_clean":"2024-04-27T03:19:35.438808+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:19:35.438808+0000","last_undegraded":"2024-04-27T03:19:35.438808+0000","last_fullsized":"2024-04-27T03:19:35.438808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:38.895777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'48","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.284523+0000","last_change":"2024-04-27T03:18:34.800954+0000","last_active":"2024-04-27T03:19:58.284523+0000","last_peered":"2024-04-27T03:19:58.284523+0000","last_clean":"2024-04-27T03:19:58.284523+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:19:58.284523+0000","last_undegraded":"2024-04-27T03:19:58.284523+0000","last_fullsized":"2024-04-27T03:19:58.284523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:59.816977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'282","reported_seq":381,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.495106+0000","last_change":"2024-04-27T03:17:55.063480+0000","last_active":"2024-04-27T03:21:01.495106+0000","last_peered":"2024-04-27T03:21:01.495106+0000","last_clean":"2024-04-27T03:21:01.495106+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:01.495106+0000","last_undegraded":"2024-04-27T03:21:01.495106+0000","last_fullsized":"2024-04-27T03:21:01.495106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:25.995264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382973+0000","last_change":"2024-04-27T03:17:57.077044+0000","last_active":"2024-04-27T03:19:35.382973+0000","last_peered":"2024-04-27T03:19:35.382973+0000","last_clean":"2024-04-27T03:19:35.382973+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:19:35.382973+0000","last_undegraded":"2024-04-27T03:19:35.382973+0000","last_fullsized":"2024-04-27T03:19:35.382973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:03.311426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299661+0000","last_change":"2024-04-27T03:17:49.007911+0000","last_active":"2024-04-27T03:21:01.299661+0000","last_peered":"2024-04-27T03:21:01.299661+0000","last_clean":"2024-04-27T03:21:01.299661+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:01.299661+0000","last_undegraded":"2024-04-27T03:21:01.299661+0000","last_fullsized":"2024-04-27T03:21:01.299661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:07.206167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816213+0000","last_change":"2024-04-27T03:18:34.772611+0000","last_active":"2024-04-27T03:19:34.816213+0000","last_peered":"2024-04-27T03:19:34.816213+0000","last_clean":"2024-04-27T03:19:34.816213+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:19:34.816213+0000","last_undegraded":"2024-04-27T03:19:34.816213+0000","last_fullsized":"2024-04-27T03:19:34.816213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:47.500325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437638+0000","last_change":"2024-04-27T03:17:51.038976+0000","last_active":"2024-04-27T03:19:35.437638+0000","last_peered":"2024-04-27T03:19:35.437638+0000","last_clean":"2024-04-27T03:19:35.437638+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:19:35.437638+0000","last_undegraded":"2024-04-27T03:19:35.437638+0000","last_fullsized":"2024-04-27T03:19:35.437638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:17:48.170469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'66","reported_seq":138,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.287412+0000","last_change":"2024-04-27T03:17:53.050762+0000","last_active":"2024-04-27T03:19:58.287412+0000","last_peered":"2024-04-27T03:19:58.287412+0000","last_clean":"2024-04-27T03:19:58.287412+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:19:58.287412+0000","last_undegraded":"2024-04-27T03:19:58.287412+0000","last_fullsized":"2024-04-27T03:19:58.287412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":67,"num_read_kb":56,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436043+0000","last_change":"2024-04-27T03:17:55.057576+0000","last_active":"2024-04-27T03:19:35.436043+0000","last_peered":"2024-04-27T03:19:35.436043+0000","last_clean":"2024-04-27T03:19:35.436043+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:19:35.436043+0000","last_undegraded":"2024-04-27T03:19:35.436043+0000","last_fullsized":"2024-04-27T03:19:35.436043+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:41.241891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'18","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.456884+0000","last_change":"2024-04-27T03:17:57.062348+0000","last_active":"2024-04-27T03:21:01.456884+0000","last_peered":"2024-04-27T03:21:01.456884+0000","last_clean":"2024-04-27T03:21:01.456884+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:01.456884+0000","last_undegraded":"2024-04-27T03:21:01.456884+0000","last_fullsized":"2024-04-27T03:21:01.456884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:14.902153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232625+0000","last_change":"2024-04-27T03:17:49.009833+0000","last_active":"2024-04-27T03:21:01.232625+0000","last_peered":"2024-04-27T03:21:01.232625+0000","last_clean":"2024-04-27T03:21:01.232625+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:01.232625+0000","last_undegraded":"2024-04-27T03:21:01.232625+0000","last_fullsized":"2024-04-27T03:21:01.232625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:36:59.530315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437653+0000","last_change":"2024-04-27T03:18:34.767518+0000","last_active":"2024-04-27T03:19:35.437653+0000","last_peered":"2024-04-27T03:19:35.437653+0000","last_clean":"2024-04-27T03:19:35.437653+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:19:35.437653+0000","last_undegraded":"2024-04-27T03:19:35.437653+0000","last_fullsized":"2024-04-27T03:19:35.437653+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:02.579718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435824+0000","last_change":"2024-04-27T03:17:51.029195+0000","last_active":"2024-04-27T03:19:35.435824+0000","last_peered":"2024-04-27T03:19:35.435824+0000","last_clean":"2024-04-27T03:19:35.435824+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:19:35.435824+0000","last_undegraded":"2024-04-27T03:19:35.435824+0000","last_fullsized":"2024-04-27T03:19:35.435824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:42.059879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'40","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.253858+0000","last_change":"2024-04-27T03:17:53.049067+0000","last_active":"2024-04-27T03:19:58.253858+0000","last_peered":"2024-04-27T03:19:58.253858+0000","last_clean":"2024-04-27T03:19:58.253858+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:19:58.253858+0000","last_undegraded":"2024-04-27T03:19:58.253858+0000","last_fullsized":"2024-04-27T03:19:58.253858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:55:37.400674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":54,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815163+0000","last_change":"2024-04-27T03:17:55.072520+0000","last_active":"2024-04-27T03:19:34.815163+0000","last_peered":"2024-04-27T03:19:34.815163+0000","last_clean":"2024-04-27T03:19:34.815163+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:19:34.815163+0000","last_undegraded":"2024-04-27T03:19:34.815163+0000","last_fullsized":"2024-04-27T03:19:34.815163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'3","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.883363+0000","last_change":"2024-04-27T03:17:57.080943+0000","last_active":"2024-04-27T03:19:57.883363+0000","last_peered":"2024-04-27T03:19:57.883363+0000","last_clean":"2024-04-27T03:19:57.883363+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:19:57.883363+0000","last_undegraded":"2024-04-27T03:19:57.883363+0000","last_fullsized":"2024-04-27T03:19:57.883363+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822407+0000","last_change":"2024-04-27T03:17:49.016254+0000","last_active":"2024-04-27T03:19:34.822407+0000","last_peered":"2024-04-27T03:19:34.822407+0000","last_clean":"2024-04-27T03:19:34.822407+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:19:34.822407+0000","last_undegraded":"2024-04-27T03:19:34.822407+0000","last_fullsized":"2024-04-27T03:19:34.822407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817143+0000","last_change":"2024-04-27T03:17:47.026419+0000","last_active":"2024-04-27T03:19:34.817143+0000","last_peered":"2024-04-27T03:19:34.817143+0000","last_clean":"2024-04-27T03:19:34.817143+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:19:34.817143+0000","last_undegraded":"2024-04-27T03:19:34.817143+0000","last_fullsized":"2024-04-27T03:19:34.817143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:25.121818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377466+0000","last_change":"2024-04-27T03:17:51.036820+0000","last_active":"2024-04-27T03:19:35.377466+0000","last_peered":"2024-04-27T03:19:35.377466+0000","last_clean":"2024-04-27T03:19:35.377466+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:19:35.377466+0000","last_undegraded":"2024-04-27T03:19:35.377466+0000","last_fullsized":"2024-04-27T03:19:35.377466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:24.012671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.298340+0000","last_change":"2024-04-27T03:17:53.043412+0000","last_active":"2024-04-27T03:19:58.298340+0000","last_peered":"2024-04-27T03:19:58.298340+0000","last_clean":"2024-04-27T03:19:58.298340+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:19:58.298340+0000","last_undegraded":"2024-04-27T03:19:58.298340+0000","last_fullsized":"2024-04-27T03:19:58.298340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:37:03.560918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'282","reported_seq":408,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.467183+0000","last_change":"2024-04-27T03:17:55.070428+0000","last_active":"2024-04-27T03:21:01.467183+0000","last_peered":"2024-04-27T03:21:01.467183+0000","last_clean":"2024-04-27T03:21:01.467183+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:01.467183+0000","last_undegraded":"2024-04-27T03:21:01.467183+0000","last_fullsized":"2024-04-27T03:21:01.467183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:20.977192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'12","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.727848+0000","last_change":"2024-04-27T03:17:57.077926+0000","last_active":"2024-04-27T03:20:02.727848+0000","last_peered":"2024-04-27T03:20:02.727848+0000","last_clean":"2024-04-27T03:20:02.727848+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:20:02.727848+0000","last_undegraded":"2024-04-27T03:20:02.727848+0000","last_fullsized":"2024-04-27T03:20:02.727848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:14:58.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.441233+0000","last_change":"2024-04-27T03:17:49.008951+0000","last_active":"2024-04-27T03:19:35.441233+0000","last_peered":"2024-04-27T03:19:35.441233+0000","last_clean":"2024-04-27T03:19:35.441233+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:19:35.441233+0000","last_undegraded":"2024-04-27T03:19:35.441233+0000","last_fullsized":"2024-04-27T03:19:35.441233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:15.953778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810391+0000","last_change":"2024-04-27T03:17:47.036712+0000","last_active":"2024-04-27T03:19:34.810391+0000","last_peered":"2024-04-27T03:19:34.810391+0000","last_clean":"2024-04-27T03:19:34.810391+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:19:34.810391+0000","last_undegraded":"2024-04-27T03:19:34.810391+0000","last_fullsized":"2024-04-27T03:19:34.810391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'3","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724954+0000","last_change":"2024-04-27T03:17:51.037543+0000","last_active":"2024-04-27T03:19:59.724954+0000","last_peered":"2024-04-27T03:19:59.724954+0000","last_clean":"2024-04-27T03:19:59.724954+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:19:59.724954+0000","last_undegraded":"2024-04-27T03:19:59.724954+0000","last_fullsized":"2024-04-27T03:19:59.724954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":381,"num_read_kb":381,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"27'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814593+0000","last_change":"2024-04-27T03:17:53.042236+0000","last_active":"2024-04-27T03:19:34.814593+0000","last_peered":"2024-04-27T03:19:34.814593+0000","last_clean":"2024-04-27T03:19:34.814593+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:19:34.814593+0000","last_undegraded":"2024-04-27T03:19:34.814593+0000","last_fullsized":"2024-04-27T03:19:34.814593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'282","reported_seq":464,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.408184+0000","last_change":"2024-04-27T03:17:55.068988+0000","last_active":"2024-04-27T03:21:01.408184+0000","last_peered":"2024-04-27T03:21:01.408184+0000","last_clean":"2024-04-27T03:21:01.408184+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:01.408184+0000","last_undegraded":"2024-04-27T03:21:01.408184+0000","last_fullsized":"2024-04-27T03:21:01.408184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.880007+0000","last_change":"2024-04-27T03:17:57.074965+0000","last_active":"2024-04-27T03:19:59.880007+0000","last_peered":"2024-04-27T03:19:59.880007+0000","last_clean":"2024-04-27T03:19:59.880007+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:19:59.880007+0000","last_undegraded":"2024-04-27T03:19:59.880007+0000","last_fullsized":"2024-04-27T03:19:59.880007+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:30.053117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386155+0000","last_change":"2024-04-27T03:17:49.018519+0000","last_active":"2024-04-27T03:19:35.386155+0000","last_peered":"2024-04-27T03:19:35.386155+0000","last_clean":"2024-04-27T03:19:35.386155+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:19:35.386155+0000","last_undegraded":"2024-04-27T03:19:35.386155+0000","last_fullsized":"2024-04-27T03:19:35.386155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:46.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809539+0000","last_change":"2024-04-27T03:17:47.031891+0000","last_active":"2024-04-27T03:19:34.809539+0000","last_peered":"2024-04-27T03:19:34.809539+0000","last_clean":"2024-04-27T03:19:34.809539+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:19:34.809539+0000","last_undegraded":"2024-04-27T03:19:34.809539+0000","last_fullsized":"2024-04-27T03:19:34.809539+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815512+0000","last_change":"2024-04-27T03:17:51.033741+0000","last_active":"2024-04-27T03:19:34.815512+0000","last_peered":"2024-04-27T03:19:34.815512+0000","last_clean":"2024-04-27T03:19:34.815512+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:19:34.815512+0000","last_undegraded":"2024-04-27T03:19:34.815512+0000","last_fullsized":"2024-04-27T03:19:34.815512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'38","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.276705+0000","last_change":"2024-04-27T03:17:53.048795+0000","last_active":"2024-04-27T03:19:58.276705+0000","last_peered":"2024-04-27T03:19:58.276705+0000","last_clean":"2024-04-27T03:19:58.276705+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:19:58.276705+0000","last_undegraded":"2024-04-27T03:19:58.276705+0000","last_fullsized":"2024-04-27T03:19:58.276705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817470+0000","last_change":"2024-04-27T03:17:55.071142+0000","last_active":"2024-04-27T03:19:34.817470+0000","last_peered":"2024-04-27T03:19:34.817470+0000","last_clean":"2024-04-27T03:19:34.817470+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:19:34.817470+0000","last_undegraded":"2024-04-27T03:19:34.817470+0000","last_fullsized":"2024-04-27T03:19:34.817470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'8","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.368117+0000","last_change":"2024-04-27T03:17:57.071496+0000","last_active":"2024-04-27T03:21:01.368117+0000","last_peered":"2024-04-27T03:21:01.368117+0000","last_clean":"2024-04-27T03:21:01.368117+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:01.368117+0000","last_undegraded":"2024-04-27T03:21:01.368117+0000","last_fullsized":"2024-04-27T03:21:01.368117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:32:20.548838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.811995+0000","last_change":"2024-04-27T03:17:49.013855+0000","last_active":"2024-04-27T03:21:00.811995+0000","last_peered":"2024-04-27T03:21:00.811995+0000","last_clean":"2024-04-27T03:21:00.811995+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:00.811995+0000","last_undegraded":"2024-04-27T03:21:00.811995+0000","last_fullsized":"2024-04-27T03:21:00.811995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":201,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379585+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:19:35.379585+0000","last_peered":"2024-04-27T03:19:35.379585+0000","last_clean":"2024-04-27T03:19:35.379585+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:19:35.379585+0000","last_undegraded":"2024-04-27T03:19:35.379585+0000","last_fullsized":"2024-04-27T03:19:35.379585+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816708+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:19:34.816708+0000","last_peered":"2024-04-27T03:19:34.816708+0000","last_clean":"2024-04-27T03:19:34.816708+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:19:34.816708+0000","last_undegraded":"2024-04-27T03:19:34.816708+0000","last_fullsized":"2024-04-27T03:19:34.816708+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:25.777031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.100108+0000","last_change":"2024-04-27T03:17:47.039911+0000","last_active":"2024-04-27T03:21:01.100108+0000","last_peered":"2024-04-27T03:21:01.100108+0000","last_clean":"2024-04-27T03:21:01.100108+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:21:01.100108+0000","last_undegraded":"2024-04-27T03:21:01.100108+0000","last_fullsized":"2024-04-27T03:21:01.100108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382198+0000","last_change":"2024-04-27T03:17:51.036961+0000","last_active":"2024-04-27T03:19:35.382198+0000","last_peered":"2024-04-27T03:19:35.382198+0000","last_clean":"2024-04-27T03:19:35.382198+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:19:35.382198+0000","last_undegraded":"2024-04-27T03:19:35.382198+0000","last_fullsized":"2024-04-27T03:19:35.382198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:16.046086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'82","reported_seq":161,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.296556+0000","last_change":"2024-04-27T03:17:53.051108+0000","last_active":"2024-04-27T03:19:58.296556+0000","last_peered":"2024-04-27T03:19:58.296556+0000","last_clean":"2024-04-27T03:19:58.296556+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:19:58.296556+0000","last_undegraded":"2024-04-27T03:19:58.296556+0000","last_fullsized":"2024-04-27T03:19:58.296556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:32:18.585035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812224+0000","last_change":"2024-04-27T03:17:55.071697+0000","last_active":"2024-04-27T03:19:34.812224+0000","last_peered":"2024-04-27T03:19:34.812224+0000","last_clean":"2024-04-27T03:19:34.812224+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:19:34.812224+0000","last_undegraded":"2024-04-27T03:19:34.812224+0000","last_fullsized":"2024-04-27T03:19:34.812224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":167,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.324346+0000","last_change":"2024-04-27T03:17:57.075459+0000","last_active":"2024-04-27T03:21:01.324346+0000","last_peered":"2024-04-27T03:21:01.324346+0000","last_clean":"2024-04-27T03:21:01.324346+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:01.324346+0000","last_undegraded":"2024-04-27T03:21:01.324346+0000","last_fullsized":"2024-04-27T03:21:01.324346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:22.086532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233419+0000","last_change":"2024-04-27T03:17:49.007601+0000","last_active":"2024-04-27T03:21:01.233419+0000","last_peered":"2024-04-27T03:21:01.233419+0000","last_clean":"2024-04-27T03:21:01.233419+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:21:01.233419+0000","last_undegraded":"2024-04-27T03:21:01.233419+0000","last_fullsized":"2024-04-27T03:21:01.233419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815344+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:19:34.815344+0000","last_peered":"2024-04-27T03:19:34.815344+0000","last_clean":"2024-04-27T03:19:34.815344+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:19:34.815344+0000","last_undegraded":"2024-04-27T03:19:34.815344+0000","last_fullsized":"2024-04-27T03:19:34.815344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:47:23.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380638+0000","last_change":"2024-04-27T03:17:47.015528+0000","last_active":"2024-04-27T03:19:35.380638+0000","last_peered":"2024-04-27T03:19:35.380638+0000","last_clean":"2024-04-27T03:19:35.380638+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:19:35.380638+0000","last_undegraded":"2024-04-27T03:19:35.380638+0000","last_fullsized":"2024-04-27T03:19:35.380638+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822099+0000","last_change":"2024-04-27T03:17:51.034048+0000","last_active":"2024-04-27T03:19:34.822099+0000","last_peered":"2024-04-27T03:19:34.822099+0000","last_clean":"2024-04-27T03:19:34.822099+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:19:34.822099+0000","last_undegraded":"2024-04-27T03:19:34.822099+0000","last_fullsized":"2024-04-27T03:19:34.822099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'279","reported_seq":493,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.279126+0000","last_change":"2024-04-27T03:17:53.058256+0000","last_active":"2024-04-27T03:21:01.279126+0000","last_peered":"2024-04-27T03:21:01.279126+0000","last_clean":"2024-04-27T03:21:01.279126+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:01.279126+0000","last_undegraded":"2024-04-27T03:21:01.279126+0000","last_fullsized":"2024-04-27T03:21:01.279126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:08.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":366,"num_read_kb":350,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811204+0000","last_change":"2024-04-27T03:17:55.074606+0000","last_active":"2024-04-27T03:19:34.811204+0000","last_peered":"2024-04-27T03:19:34.811204+0000","last_clean":"2024-04-27T03:19:34.811204+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:19:34.811204+0000","last_undegraded":"2024-04-27T03:19:34.811204+0000","last_fullsized":"2024-04-27T03:19:34.811204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379830+0000","last_change":"2024-04-27T03:17:57.086888+0000","last_active":"2024-04-27T03:19:35.379830+0000","last_peered":"2024-04-27T03:19:35.379830+0000","last_clean":"2024-04-27T03:19:35.379830+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:19:35.379830+0000","last_undegraded":"2024-04-27T03:19:35.379830+0000","last_fullsized":"2024-04-27T03:19:35.379830+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:30.189251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232768+0000","last_change":"2024-04-27T03:17:49.014111+0000","last_active":"2024-04-27T03:21:01.232768+0000","last_peered":"2024-04-27T03:21:01.232768+0000","last_clean":"2024-04-27T03:21:01.232768+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:01.232768+0000","last_undegraded":"2024-04-27T03:21:01.232768+0000","last_fullsized":"2024-04-27T03:21:01.232768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:15.967813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436752+0000","last_change":"2024-04-27T03:17:39.633841+0000","last_active":"2024-04-27T03:19:35.436752+0000","last_peered":"2024-04-27T03:19:35.436752+0000","last_clean":"2024-04-27T03:19:35.436752+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:19:35.436752+0000","last_undegraded":"2024-04-27T03:19:35.436752+0000","last_fullsized":"2024-04-27T03:19:35.436752+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.559603+0000","last_change":"2024-04-27T03:17:47.011695+0000","last_active":"2024-04-27T03:21:00.559603+0000","last_peered":"2024-04-27T03:21:00.559603+0000","last_clean":"2024-04-27T03:21:00.559603+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:21:00.559603+0000","last_undegraded":"2024-04-27T03:21:00.559603+0000","last_fullsized":"2024-04-27T03:21:00.559603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"20'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435807+0000","last_change":"2024-04-27T03:17:51.031676+0000","last_active":"2024-04-27T03:19:35.435807+0000","last_peered":"2024-04-27T03:19:35.435807+0000","last_clean":"2024-04-27T03:19:35.435807+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:19:35.435807+0000","last_undegraded":"2024-04-27T03:19:35.435807+0000","last_fullsized":"2024-04-27T03:19:35.435807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:23.681601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'16","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.278287+0000","last_change":"2024-04-27T03:17:53.051185+0000","last_active":"2024-04-27T03:21:01.278287+0000","last_peered":"2024-04-27T03:21:01.278287+0000","last_clean":"2024-04-27T03:21:01.278287+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:01.278287+0000","last_undegraded":"2024-04-27T03:21:01.278287+0000","last_fullsized":"2024-04-27T03:21:01.278287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382273+0000","last_change":"2024-04-27T03:17:55.070180+0000","last_active":"2024-04-27T03:19:35.382273+0000","last_peered":"2024-04-27T03:19:35.382273+0000","last_clean":"2024-04-27T03:19:35.382273+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:19:35.382273+0000","last_undegraded":"2024-04-27T03:19:35.382273+0000","last_fullsized":"2024-04-27T03:19:35.382273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:15.975975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.206940+0000","last_change":"2024-04-27T03:17:57.071109+0000","last_active":"2024-04-27T03:21:01.206940+0000","last_peered":"2024-04-27T03:21:01.206940+0000","last_clean":"2024-04-27T03:21:01.206940+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:01.206940+0000","last_undegraded":"2024-04-27T03:21:01.206940+0000","last_fullsized":"2024-04-27T03:21:01.206940+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802933+0000","last_change":"2024-04-27T03:17:49.009764+0000","last_active":"2024-04-27T03:21:00.802933+0000","last_peered":"2024-04-27T03:21:00.802933+0000","last_clean":"2024-04-27T03:21:00.802933+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:00.802933+0000","last_undegraded":"2024-04-27T03:21:00.802933+0000","last_fullsized":"2024-04-27T03:21:00.802933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:31.202044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816459+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:19:34.816459+0000","last_peered":"2024-04-27T03:19:34.816459+0000","last_clean":"2024-04-27T03:19:34.816459+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:19:34.816459+0000","last_undegraded":"2024-04-27T03:19:34.816459+0000","last_fullsized":"2024-04-27T03:19:34.816459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.133613+0000","last_change":"2024-04-27T03:17:47.015018+0000","last_active":"2024-04-27T03:21:01.133613+0000","last_peered":"2024-04-27T03:21:01.133613+0000","last_clean":"2024-04-27T03:21:01.133613+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:21:01.133613+0000","last_undegraded":"2024-04-27T03:21:01.133613+0000","last_fullsized":"2024-04-27T03:21:01.133613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.378112+0000","last_change":"2024-04-27T03:17:51.029533+0000","last_active":"2024-04-27T03:19:35.378112+0000","last_peered":"2024-04-27T03:19:35.378112+0000","last_clean":"2024-04-27T03:19:35.378112+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:19:35.378112+0000","last_undegraded":"2024-04-27T03:19:35.378112+0000","last_fullsized":"2024-04-27T03:19:35.378112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:52.883679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"29'14","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.236793+0000","last_change":"2024-04-27T03:17:53.045958+0000","last_active":"2024-04-27T03:21:00.236793+0000","last_peered":"2024-04-27T03:21:00.236793+0000","last_clean":"2024-04-27T03:21:00.236793+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:00.236793+0000","last_undegraded":"2024-04-27T03:21:00.236793+0000","last_fullsized":"2024-04-27T03:21:00.236793+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815623+0000","last_change":"2024-04-27T03:17:55.078514+0000","last_active":"2024-04-27T03:19:34.815623+0000","last_peered":"2024-04-27T03:19:34.815623+0000","last_clean":"2024-04-27T03:19:34.815623+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:19:34.815623+0000","last_undegraded":"2024-04-27T03:19:34.815623+0000","last_fullsized":"2024-04-27T03:19:34.815623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.953374+0000","last_change":"2024-04-27T03:17:57.089242+0000","last_active":"2024-04-27T03:21:00.953374+0000","last_peered":"2024-04-27T03:21:00.953374+0000","last_clean":"2024-04-27T03:21:00.953374+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:21:00.953374+0000","last_undegraded":"2024-04-27T03:21:00.953374+0000","last_fullsized":"2024-04-27T03:21:00.953374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:12.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299342+0000","last_change":"2024-04-27T03:17:49.006116+0000","last_active":"2024-04-27T03:21:01.299342+0000","last_peered":"2024-04-27T03:21:01.299342+0000","last_clean":"2024-04-27T03:21:01.299342+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:01.299342+0000","last_undegraded":"2024-04-27T03:21:01.299342+0000","last_fullsized":"2024-04-27T03:21:01.299342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:05.838106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822302+0000","last_change":"2024-04-27T03:17:47.024776+0000","last_active":"2024-04-27T03:19:34.822302+0000","last_peered":"2024-04-27T03:19:34.822302+0000","last_clean":"2024-04-27T03:19:34.822302+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:19:34.822302+0000","last_undegraded":"2024-04-27T03:19:34.822302+0000","last_fullsized":"2024-04-27T03:19:34.822302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382818+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:19:35.382818+0000","last_peered":"2024-04-27T03:19:35.382818+0000","last_clean":"2024-04-27T03:19:35.382818+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:19:35.382818+0000","last_undegraded":"2024-04-27T03:19:35.382818+0000","last_fullsized":"2024-04-27T03:19:35.382818+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:01:48.896188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.733547+0000","last_change":"2024-04-27T03:17:51.037696+0000","last_active":"2024-04-27T03:19:57.733547+0000","last_peered":"2024-04-27T03:19:57.733547+0000","last_clean":"2024-04-27T03:19:57.733547+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:19:57.733547+0000","last_undegraded":"2024-04-27T03:19:57.733547+0000","last_fullsized":"2024-04-27T03:19:57.733547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:44.259718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'73","reported_seq":148,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.297267+0000","last_change":"2024-04-27T03:17:53.054938+0000","last_active":"2024-04-27T03:19:58.297267+0000","last_peered":"2024-04-27T03:19:58.297267+0000","last_clean":"2024-04-27T03:19:58.297267+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:19:58.297267+0000","last_undegraded":"2024-04-27T03:19:58.297267+0000","last_fullsized":"2024-04-27T03:19:58.297267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:38.318924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":63,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815814+0000","last_change":"2024-04-27T03:17:55.068676+0000","last_active":"2024-04-27T03:19:34.815814+0000","last_peered":"2024-04-27T03:19:34.815814+0000","last_clean":"2024-04-27T03:19:34.815814+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:19:34.815814+0000","last_undegraded":"2024-04-27T03:19:34.815814+0000","last_fullsized":"2024-04-27T03:19:34.815814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:04:55.186362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.082757+0000","last_change":"2024-04-27T03:17:57.068043+0000","last_active":"2024-04-27T03:20:00.082757+0000","last_peered":"2024-04-27T03:20:00.082757+0000","last_clean":"2024-04-27T03:20:00.082757+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:20:00.082757+0000","last_undegraded":"2024-04-27T03:20:00.082757+0000","last_fullsized":"2024-04-27T03:20:00.082757+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:14.279033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382103+0000","last_change":"2024-04-27T03:17:49.009882+0000","last_active":"2024-04-27T03:19:35.382103+0000","last_peered":"2024-04-27T03:19:35.382103+0000","last_clean":"2024-04-27T03:19:35.382103+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:19:35.382103+0000","last_undegraded":"2024-04-27T03:19:35.382103+0000","last_fullsized":"2024-04-27T03:19:35.382103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:36.878689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.286999+0000","last_change":"2024-04-27T03:17:47.030065+0000","last_active":"2024-04-27T03:21:01.286999+0000","last_peered":"2024-04-27T03:21:01.286999+0000","last_clean":"2024-04-27T03:21:01.286999+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:01.286999+0000","last_undegraded":"2024-04-27T03:21:01.286999+0000","last_fullsized":"2024-04-27T03:21:01.286999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:45:21.469900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":21,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820336+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:19:34.820336+0000","last_peered":"2024-04-27T03:19:34.820336+0000","last_clean":"2024-04-27T03:19:34.820336+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:19:34.820336+0000","last_undegraded":"2024-04-27T03:19:34.820336+0000","last_fullsized":"2024-04-27T03:19:34.820336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:48:41.511344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816118+0000","last_change":"2024-04-27T03:17:51.035808+0000","last_active":"2024-04-27T03:19:34.816118+0000","last_peered":"2024-04-27T03:19:34.816118+0000","last_clean":"2024-04-27T03:19:34.816118+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:19:34.816118+0000","last_undegraded":"2024-04-27T03:19:34.816118+0000","last_fullsized":"2024-04-27T03:19:34.816118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"27'20","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381588+0000","last_change":"2024-04-27T03:17:53.049471+0000","last_active":"2024-04-27T03:19:35.381588+0000","last_peered":"2024-04-27T03:19:35.381588+0000","last_clean":"2024-04-27T03:19:35.381588+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:19:35.381588+0000","last_undegraded":"2024-04-27T03:19:35.381588+0000","last_fullsized":"2024-04-27T03:19:35.381588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:42.365441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'282","reported_seq":361,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.481977+0000","last_change":"2024-04-27T03:17:55.077154+0000","last_active":"2024-04-27T03:21:01.481977+0000","last_peered":"2024-04-27T03:21:01.481977+0000","last_clean":"2024-04-27T03:21:01.481977+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:01.481977+0000","last_undegraded":"2024-04-27T03:21:01.481977+0000","last_fullsized":"2024-04-27T03:21:01.481977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:55:52.856782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.236855+0000","last_change":"2024-04-27T03:17:57.091210+0000","last_active":"2024-04-27T03:21:01.236855+0000","last_peered":"2024-04-27T03:21:01.236855+0000","last_clean":"2024-04-27T03:21:01.236855+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:21:01.236855+0000","last_undegraded":"2024-04-27T03:21:01.236855+0000","last_fullsized":"2024-04-27T03:21:01.236855+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:12.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948922+0000","last_change":"2024-04-27T03:17:49.007942+0000","last_active":"2024-04-27T03:21:00.948922+0000","last_peered":"2024-04-27T03:21:00.948922+0000","last_clean":"2024-04-27T03:21:00.948922+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:21:00.948922+0000","last_undegraded":"2024-04-27T03:21:00.948922+0000","last_fullsized":"2024-04-27T03:21:00.948922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815379+0000","last_change":"2024-04-27T03:17:47.039532+0000","last_active":"2024-04-27T03:19:34.815379+0000","last_peered":"2024-04-27T03:19:34.815379+0000","last_clean":"2024-04-27T03:19:34.815379+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:19:34.815379+0000","last_undegraded":"2024-04-27T03:19:34.815379+0000","last_fullsized":"2024-04-27T03:19:34.815379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384884+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:19:35.384884+0000","last_peered":"2024-04-27T03:19:35.384884+0000","last_clean":"2024-04-27T03:19:35.384884+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:19:35.384884+0000","last_undegraded":"2024-04-27T03:19:35.384884+0000","last_fullsized":"2024-04-27T03:19:35.384884+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:45.567442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815568+0000","last_change":"2024-04-27T03:17:51.020247+0000","last_active":"2024-04-27T03:19:34.815568+0000","last_peered":"2024-04-27T03:19:34.815568+0000","last_clean":"2024-04-27T03:19:34.815568+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:19:34.815568+0000","last_undegraded":"2024-04-27T03:19:34.815568+0000","last_fullsized":"2024-04-27T03:19:34.815568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812598+0000","last_change":"2024-04-27T03:17:53.048021+0000","last_active":"2024-04-27T03:19:34.812598+0000","last_peered":"2024-04-27T03:19:34.812598+0000","last_clean":"2024-04-27T03:19:34.812598+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:19:34.812598+0000","last_undegraded":"2024-04-27T03:19:34.812598+0000","last_fullsized":"2024-04-27T03:19:34.812598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:13.963398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'282","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.458427+0000","last_change":"2024-04-27T03:17:55.075368+0000","last_active":"2024-04-27T03:21:01.458427+0000","last_peered":"2024-04-27T03:21:01.458427+0000","last_clean":"2024-04-27T03:21:01.458427+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:01.458427+0000","last_undegraded":"2024-04-27T03:21:01.458427+0000","last_fullsized":"2024-04-27T03:21:01.458427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381777+0000","last_change":"2024-04-27T03:17:57.087004+0000","last_active":"2024-04-27T03:19:35.381777+0000","last_peered":"2024-04-27T03:19:35.381777+0000","last_clean":"2024-04-27T03:19:35.381777+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:19:35.381777+0000","last_undegraded":"2024-04-27T03:19:35.381777+0000","last_fullsized":"2024-04-27T03:19:35.381777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:17.983425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.193216+0000","last_change":"2024-04-27T03:17:49.004339+0000","last_active":"2024-04-27T03:20:00.193216+0000","last_peered":"2024-04-27T03:20:00.193216+0000","last_clean":"2024-04-27T03:20:00.193216+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:20:00.193216+0000","last_undegraded":"2024-04-27T03:20:00.193216+0000","last_fullsized":"2024-04-27T03:20:00.193216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:08.912344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.153066+0000","last_change":"2024-04-27T03:17:47.000780+0000","last_active":"2024-04-27T03:20:00.153066+0000","last_peered":"2024-04-27T03:20:00.153066+0000","last_clean":"2024-04-27T03:20:00.153066+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:20:00.153066+0000","last_undegraded":"2024-04-27T03:20:00.153066+0000","last_fullsized":"2024-04-27T03:20:00.153066+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819785+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:19:34.819785+0000","last_peered":"2024-04-27T03:19:34.819785+0000","last_clean":"2024-04-27T03:19:34.819785+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:19:34.819785+0000","last_undegraded":"2024-04-27T03:19:34.819785+0000","last_fullsized":"2024-04-27T03:19:34.819785+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:34.021988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810326+0000","last_change":"2024-04-27T03:17:51.033805+0000","last_active":"2024-04-27T03:19:34.810326+0000","last_peered":"2024-04-27T03:19:34.810326+0000","last_clean":"2024-04-27T03:19:34.810326+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:19:34.810326+0000","last_undegraded":"2024-04-27T03:19:34.810326+0000","last_fullsized":"2024-04-27T03:19:34.810326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'92","reported_seq":174,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.728448+0000","last_change":"2024-04-27T03:17:53.038345+0000","last_active":"2024-04-27T03:21:00.728448+0000","last_peered":"2024-04-27T03:21:00.728448+0000","last_clean":"2024-04-27T03:21:00.728448+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:00.728448+0000","last_undegraded":"2024-04-27T03:21:00.728448+0000","last_fullsized":"2024-04-27T03:21:00.728448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6026,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":86,"num_read_kb":71,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814813+0000","last_change":"2024-04-27T03:17:55.081924+0000","last_active":"2024-04-27T03:19:34.814813+0000","last_peered":"2024-04-27T03:19:34.814813+0000","last_clean":"2024-04-27T03:19:34.814813+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:19:34.814813+0000","last_undegraded":"2024-04-27T03:19:34.814813+0000","last_fullsized":"2024-04-27T03:19:34.814813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439932+0000","last_change":"2024-04-27T03:17:57.068568+0000","last_active":"2024-04-27T03:19:35.439932+0000","last_peered":"2024-04-27T03:19:35.439932+0000","last_clean":"2024-04-27T03:19:35.439932+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:19:35.439932+0000","last_undegraded":"2024-04-27T03:19:35.439932+0000","last_fullsized":"2024-04-27T03:19:35.439932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:43.785644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442918+0000","last_change":"2024-04-27T03:17:57.068516+0000","last_active":"2024-04-27T03:19:35.442918+0000","last_peered":"2024-04-27T03:19:35.442918+0000","last_clean":"2024-04-27T03:19:35.442918+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:19:35.442918+0000","last_undegraded":"2024-04-27T03:19:35.442918+0000","last_fullsized":"2024-04-27T03:19:35.442918+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:58.924125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802035+0000","last_change":"2024-04-27T03:17:49.009366+0000","last_active":"2024-04-27T03:21:00.802035+0000","last_peered":"2024-04-27T03:21:00.802035+0000","last_clean":"2024-04-27T03:21:00.802035+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:00.802035+0000","last_undegraded":"2024-04-27T03:21:00.802035+0000","last_fullsized":"2024-04-27T03:21:00.802035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:58.301334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.124874+0000","last_change":"2024-04-27T03:17:47.023352+0000","last_active":"2024-04-27T03:21:01.124874+0000","last_peered":"2024-04-27T03:21:01.124874+0000","last_clean":"2024-04-27T03:21:01.124874+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:01.124874+0000","last_undegraded":"2024-04-27T03:21:01.124874+0000","last_fullsized":"2024-04-27T03:21:01.124874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"20'1","reported_seq":222,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724858+0000","last_change":"2024-04-27T03:17:51.031849+0000","last_active":"2024-04-27T03:19:59.724858+0000","last_peered":"2024-04-27T03:19:59.724858+0000","last_clean":"2024-04-27T03:19:59.724858+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:19:59.724858+0000","last_undegraded":"2024-04-27T03:19:59.724858+0000","last_fullsized":"2024-04-27T03:19:59.724858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:05.179093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":282,"num_read_kb":282,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'38","reported_seq":91,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.283287+0000","last_change":"2024-04-27T03:17:53.041357+0000","last_active":"2024-04-27T03:19:58.283287+0000","last_peered":"2024-04-27T03:19:58.283287+0000","last_clean":"2024-04-27T03:19:58.283287+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:19:58.283287+0000","last_undegraded":"2024-04-27T03:19:58.283287+0000","last_fullsized":"2024-04-27T03:19:58.283287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:57.603435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809115+0000","last_change":"2024-04-27T03:17:55.074798+0000","last_active":"2024-04-27T03:19:34.809115+0000","last_peered":"2024-04-27T03:19:34.809115+0000","last_clean":"2024-04-27T03:19:34.809115+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:19:34.809115+0000","last_undegraded":"2024-04-27T03:19:34.809115+0000","last_fullsized":"2024-04-27T03:19:34.809115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:15.275458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.885734+0000","last_change":"2024-04-27T03:17:57.087180+0000","last_active":"2024-04-27T03:19:57.885734+0000","last_peered":"2024-04-27T03:19:57.885734+0000","last_clean":"2024-04-27T03:19:57.885734+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:19:57.885734+0000","last_undegraded":"2024-04-27T03:19:57.885734+0000","last_fullsized":"2024-04-27T03:19:57.885734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:50.266677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146716+0000","last_change":"2024-04-27T03:17:49.005188+0000","last_active":"2024-04-27T03:21:01.146716+0000","last_peered":"2024-04-27T03:21:01.146716+0000","last_clean":"2024-04-27T03:21:01.146716+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:01.146716+0000","last_undegraded":"2024-04-27T03:21:01.146716+0000","last_fullsized":"2024-04-27T03:21:01.146716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:24.812863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440214+0000","last_change":"2024-04-27T03:17:47.022474+0000","last_active":"2024-04-27T03:19:35.440214+0000","last_peered":"2024-04-27T03:19:35.440214+0000","last_clean":"2024-04-27T03:19:35.440214+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:19:35.440214+0000","last_undegraded":"2024-04-27T03:19:35.440214+0000","last_fullsized":"2024-04-27T03:19:35.440214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:13.293355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811181+0000","last_change":"2024-04-27T03:17:51.024948+0000","last_active":"2024-04-27T03:19:34.811181+0000","last_peered":"2024-04-27T03:19:34.811181+0000","last_clean":"2024-04-27T03:19:34.811181+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:19:34.811181+0000","last_undegraded":"2024-04-27T03:19:34.811181+0000","last_fullsized":"2024-04-27T03:19:34.811181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.271365+0000","last_change":"2024-04-27T03:17:53.053658+0000","last_active":"2024-04-27T03:21:00.271365+0000","last_peered":"2024-04-27T03:21:00.271365+0000","last_clean":"2024-04-27T03:21:00.271365+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:00.271365+0000","last_undegraded":"2024-04-27T03:21:00.271365+0000","last_fullsized":"2024-04-27T03:21:00.271365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:12.928546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438320+0000","last_change":"2024-04-27T03:17:55.051277+0000","last_active":"2024-04-27T03:19:35.438320+0000","last_peered":"2024-04-27T03:19:35.438320+0000","last_clean":"2024-04-27T03:19:35.438320+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:19:35.438320+0000","last_undegraded":"2024-04-27T03:19:35.438320+0000","last_fullsized":"2024-04-27T03:19:35.438320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:49.716708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382470+0000","last_change":"2024-04-27T03:17:57.089220+0000","last_active":"2024-04-27T03:19:35.382470+0000","last_peered":"2024-04-27T03:19:35.382470+0000","last_clean":"2024-04-27T03:19:35.382470+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:19:35.382470+0000","last_undegraded":"2024-04-27T03:19:35.382470+0000","last_fullsized":"2024-04-27T03:19:35.382470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:17.289812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949086+0000","last_change":"2024-04-27T03:18:34.799245+0000","last_active":"2024-04-27T03:21:00.949086+0000","last_peered":"2024-04-27T03:21:00.949086+0000","last_clean":"2024-04-27T03:21:00.949086+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:21:00.949086+0000","last_undegraded":"2024-04-27T03:21:00.949086+0000","last_fullsized":"2024-04-27T03:21:00.949086+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:17.961788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.107590+0000","last_change":"2024-04-27T03:17:47.012224+0000","last_active":"2024-04-27T03:21:01.107590+0000","last_peered":"2024-04-27T03:21:01.107590+0000","last_clean":"2024-04-27T03:21:01.107590+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:21:01.107590+0000","last_undegraded":"2024-04-27T03:21:01.107590+0000","last_fullsized":"2024-04-27T03:21:01.107590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813595+0000","last_change":"2024-04-27T03:17:51.037763+0000","last_active":"2024-04-27T03:19:34.813595+0000","last_peered":"2024-04-27T03:19:34.813595+0000","last_clean":"2024-04-27T03:19:34.813595+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:19:34.813595+0000","last_undegraded":"2024-04-27T03:19:34.813595+0000","last_fullsized":"2024-04-27T03:19:34.813595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:19.793632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'40","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.321256+0000","last_change":"2024-04-27T03:17:53.047021+0000","last_active":"2024-04-27T03:21:00.321256+0000","last_peered":"2024-04-27T03:21:00.321256+0000","last_clean":"2024-04-27T03:21:00.321256+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:00.321256+0000","last_undegraded":"2024-04-27T03:21:00.321256+0000","last_fullsized":"2024-04-27T03:21:00.321256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":61,"num_read_kb":64,"num_write":46,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379567+0000","last_change":"2024-04-27T03:17:55.075762+0000","last_active":"2024-04-27T03:19:35.379567+0000","last_peered":"2024-04-27T03:19:35.379567+0000","last_clean":"2024-04-27T03:19:35.379567+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:19:35.379567+0000","last_undegraded":"2024-04-27T03:19:35.379567+0000","last_fullsized":"2024-04-27T03:19:35.379567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:49:01.187937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812651+0000","last_change":"2024-04-27T03:17:57.086967+0000","last_active":"2024-04-27T03:19:34.812651+0000","last_peered":"2024-04-27T03:19:34.812651+0000","last_clean":"2024-04-27T03:19:34.812651+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:19:34.812651+0000","last_undegraded":"2024-04-27T03:19:34.812651+0000","last_fullsized":"2024-04-27T03:19:34.812651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:54.374770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.234017+0000","last_change":"2024-04-27T03:17:49.014835+0000","last_active":"2024-04-27T03:21:01.234017+0000","last_peered":"2024-04-27T03:21:01.234017+0000","last_clean":"2024-04-27T03:21:01.234017+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:01.234017+0000","last_undegraded":"2024-04-27T03:21:01.234017+0000","last_fullsized":"2024-04-27T03:21:01.234017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385250+0000","last_change":"2024-04-27T03:17:47.026622+0000","last_active":"2024-04-27T03:19:35.385250+0000","last_peered":"2024-04-27T03:19:35.385250+0000","last_clean":"2024-04-27T03:19:35.385250+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:19:35.385250+0000","last_undegraded":"2024-04-27T03:19:35.385250+0000","last_fullsized":"2024-04-27T03:19:35.385250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809701+0000","last_change":"2024-04-27T03:17:51.014069+0000","last_active":"2024-04-27T03:19:34.809701+0000","last_peered":"2024-04-27T03:19:34.809701+0000","last_clean":"2024-04-27T03:19:34.809701+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:19:34.809701+0000","last_undegraded":"2024-04-27T03:19:34.809701+0000","last_fullsized":"2024-04-27T03:19:34.809701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'31","reported_seq":77,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745864+0000","last_change":"2024-04-27T03:17:53.038792+0000","last_active":"2024-04-27T03:20:12.745864+0000","last_peered":"2024-04-27T03:20:12.745864+0000","last_clean":"2024-04-27T03:20:12.745864+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:20:12.745864+0000","last_undegraded":"2024-04-27T03:20:12.745864+0000","last_fullsized":"2024-04-27T03:20:12.745864+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438653+0000","last_change":"2024-04-27T03:17:55.055483+0000","last_active":"2024-04-27T03:19:35.438653+0000","last_peered":"2024-04-27T03:19:35.438653+0000","last_clean":"2024-04-27T03:19:35.438653+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:19:35.438653+0000","last_undegraded":"2024-04-27T03:19:35.438653+0000","last_fullsized":"2024-04-27T03:19:35.438653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:17.695216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'1","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742486+0000","last_change":"2024-04-27T03:17:49.022141+0000","last_active":"2024-04-27T03:19:57.742486+0000","last_peered":"2024-04-27T03:19:57.742486+0000","last_clean":"2024-04-27T03:19:57.742486+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:19:57.742486+0000","last_undegraded":"2024-04-27T03:19:57.742486+0000","last_fullsized":"2024-04-27T03:19:57.742486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.117502+0000","last_change":"2024-04-27T03:17:47.032082+0000","last_active":"2024-04-27T03:21:01.117502+0000","last_peered":"2024-04-27T03:21:01.117502+0000","last_clean":"2024-04-27T03:21:01.117502+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:21:01.117502+0000","last_undegraded":"2024-04-27T03:21:01.117502+0000","last_fullsized":"2024-04-27T03:21:01.117502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'4","reported_seq":230,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.696608+0000","last_change":"2024-04-27T03:17:51.024084+0000","last_active":"2024-04-27T03:19:59.696608+0000","last_peered":"2024-04-27T03:19:59.696608+0000","last_clean":"2024-04-27T03:19:59.696608+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:19:59.696608+0000","last_undegraded":"2024-04-27T03:19:59.696608+0000","last_fullsized":"2024-04-27T03:19:59.696608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:37.745198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":298,"num_read_kb":298,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'62","reported_seq":128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.293689+0000","last_change":"2024-04-27T03:17:53.035870+0000","last_active":"2024-04-27T03:19:58.293689+0000","last_peered":"2024-04-27T03:19:58.293689+0000","last_clean":"2024-04-27T03:19:58.293689+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:19:58.293689+0000","last_undegraded":"2024-04-27T03:19:58.293689+0000","last_fullsized":"2024-04-27T03:19:58.293689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:10.295737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":24,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439324+0000","last_change":"2024-04-27T03:17:55.075718+0000","last_active":"2024-04-27T03:19:35.439324+0000","last_peered":"2024-04-27T03:19:35.439324+0000","last_clean":"2024-04-27T03:19:35.439324+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:19:35.439324+0000","last_undegraded":"2024-04-27T03:19:35.439324+0000","last_fullsized":"2024-04-27T03:19:35.439324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:58.929938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.074643+0000","last_change":"2024-04-27T03:17:57.070510+0000","last_active":"2024-04-27T03:20:00.074643+0000","last_peered":"2024-04-27T03:20:00.074643+0000","last_clean":"2024-04-27T03:20:00.074643+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:20:00.074643+0000","last_undegraded":"2024-04-27T03:20:00.074643+0000","last_fullsized":"2024-04-27T03:20:00.074643+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111625+0000","last_change":"2024-04-27T03:17:49.000295+0000","last_active":"2024-04-27T03:19:56.111625+0000","last_peered":"2024-04-27T03:19:56.111625+0000","last_clean":"2024-04-27T03:19:56.111625+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:19:56.111625+0000","last_undegraded":"2024-04-27T03:19:56.111625+0000","last_fullsized":"2024-04-27T03:19:56.111625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379919+0000","last_change":"2024-04-27T03:17:47.021864+0000","last_active":"2024-04-27T03:19:35.379919+0000","last_peered":"2024-04-27T03:19:35.379919+0000","last_clean":"2024-04-27T03:19:35.379919+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:19:35.379919+0000","last_undegraded":"2024-04-27T03:19:35.379919+0000","last_fullsized":"2024-04-27T03:19:35.379919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:14.918204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820383+0000","last_change":"2024-04-27T03:17:51.033166+0000","last_active":"2024-04-27T03:19:34.820383+0000","last_peered":"2024-04-27T03:19:34.820383+0000","last_clean":"2024-04-27T03:19:34.820383+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:19:34.820383+0000","last_undegraded":"2024-04-27T03:19:34.820383+0000","last_fullsized":"2024-04-27T03:19:34.820383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:40.351925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.260042+0000","last_change":"2024-04-27T03:17:53.038085+0000","last_active":"2024-04-27T03:19:58.260042+0000","last_peered":"2024-04-27T03:19:58.260042+0000","last_clean":"2024-04-27T03:19:58.260042+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:19:58.260042+0000","last_undegraded":"2024-04-27T03:19:58.260042+0000","last_fullsized":"2024-04-27T03:19:58.260042+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.383103+0000","last_change":"2024-04-27T03:17:55.069433+0000","last_active":"2024-04-27T03:19:35.383103+0000","last_peered":"2024-04-27T03:19:35.383103+0000","last_clean":"2024-04-27T03:19:35.383103+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:19:35.383103+0000","last_undegraded":"2024-04-27T03:19:35.383103+0000","last_fullsized":"2024-04-27T03:19:35.383103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:17.345394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814942+0000","last_change":"2024-04-27T03:17:57.070067+0000","last_active":"2024-04-27T03:19:34.814942+0000","last_peered":"2024-04-27T03:19:34.814942+0000","last_clean":"2024-04-27T03:19:34.814942+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:19:34.814942+0000","last_undegraded":"2024-04-27T03:19:34.814942+0000","last_fullsized":"2024-04-27T03:19:34.814942+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814892+0000","last_change":"2024-04-27T03:17:49.004464+0000","last_active":"2024-04-27T03:19:34.814892+0000","last_peered":"2024-04-27T03:19:34.814892+0000","last_clean":"2024-04-27T03:19:34.814892+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:19:34.814892+0000","last_undegraded":"2024-04-27T03:19:34.814892+0000","last_fullsized":"2024-04-27T03:19:34.814892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.095190+0000","last_change":"2024-04-27T03:17:47.027156+0000","last_active":"2024-04-27T03:19:56.095190+0000","last_peered":"2024-04-27T03:19:56.095190+0000","last_clean":"2024-04-27T03:19:56.095190+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:19:56.095190+0000","last_undegraded":"2024-04-27T03:19:56.095190+0000","last_fullsized":"2024-04-27T03:19:56.095190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816982+0000","last_change":"2024-04-27T03:17:51.021222+0000","last_active":"2024-04-27T03:19:34.816982+0000","last_peered":"2024-04-27T03:19:34.816982+0000","last_clean":"2024-04-27T03:19:34.816982+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:19:34.816982+0000","last_undegraded":"2024-04-27T03:19:34.816982+0000","last_fullsized":"2024-04-27T03:19:34.816982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'39","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.872099+0000","last_change":"2024-04-27T03:17:53.041372+0000","last_active":"2024-04-27T03:19:57.872099+0000","last_peered":"2024-04-27T03:19:57.872099+0000","last_clean":"2024-04-27T03:19:57.872099+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:19:57.872099+0000","last_undegraded":"2024-04-27T03:19:57.872099+0000","last_fullsized":"2024-04-27T03:19:57.872099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384616+0000","last_change":"2024-04-27T03:17:55.077381+0000","last_active":"2024-04-27T03:19:35.384616+0000","last_peered":"2024-04-27T03:19:35.384616+0000","last_clean":"2024-04-27T03:19:35.384616+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:19:35.384616+0000","last_undegraded":"2024-04-27T03:19:35.384616+0000","last_fullsized":"2024-04-27T03:19:35.384616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:08:00.332438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.205156+0000","last_change":"2024-04-27T03:17:57.066218+0000","last_active":"2024-04-27T03:20:00.205156+0000","last_peered":"2024-04-27T03:20:00.205156+0000","last_clean":"2024-04-27T03:20:00.205156+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:20:00.205156+0000","last_undegraded":"2024-04-27T03:20:00.205156+0000","last_fullsized":"2024-04-27T03:20:00.205156+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.739365+0000","last_change":"2024-04-27T03:17:49.019967+0000","last_active":"2024-04-27T03:20:22.739365+0000","last_peered":"2024-04-27T03:20:22.739365+0000","last_clean":"2024-04-27T03:20:22.739365+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:20:22.739365+0000","last_undegraded":"2024-04-27T03:20:22.739365+0000","last_fullsized":"2024-04-27T03:20:22.739365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:45.458581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812087+0000","last_change":"2024-04-27T03:17:47.018817+0000","last_active":"2024-04-27T03:19:34.812087+0000","last_peered":"2024-04-27T03:19:34.812087+0000","last_clean":"2024-04-27T03:19:34.812087+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:19:34.812087+0000","last_undegraded":"2024-04-27T03:19:34.812087+0000","last_fullsized":"2024-04-27T03:19:34.812087+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377658+0000","last_change":"2024-04-27T03:17:51.030031+0000","last_active":"2024-04-27T03:19:35.377658+0000","last_peered":"2024-04-27T03:19:35.377658+0000","last_clean":"2024-04-27T03:19:35.377658+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:19:35.377658+0000","last_undegraded":"2024-04-27T03:19:35.377658+0000","last_fullsized":"2024-04-27T03:19:35.377658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:08.238587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814172+0000","last_change":"2024-04-27T03:17:53.046967+0000","last_active":"2024-04-27T03:19:34.814172+0000","last_peered":"2024-04-27T03:19:34.814172+0000","last_clean":"2024-04-27T03:19:34.814172+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:19:34.814172+0000","last_undegraded":"2024-04-27T03:19:34.814172+0000","last_fullsized":"2024-04-27T03:19:34.814172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'282","reported_seq":380,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277975+0000","last_change":"2024-04-27T03:17:55.060259+0000","last_active":"2024-04-27T03:21:01.277975+0000","last_peered":"2024-04-27T03:21:01.277975+0000","last_clean":"2024-04-27T03:21:01.277975+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:01.277975+0000","last_undegraded":"2024-04-27T03:21:01.277975+0000","last_fullsized":"2024-04-27T03:21:01.277975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:04:03.670215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813560+0000","last_change":"2024-04-27T03:17:57.084816+0000","last_active":"2024-04-27T03:19:34.813560+0000","last_peered":"2024-04-27T03:19:34.813560+0000","last_clean":"2024-04-27T03:19:34.813560+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:19:34.813560+0000","last_undegraded":"2024-04-27T03:19:34.813560+0000","last_fullsized":"2024-04-27T03:19:34.813560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:03.843907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248654+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.248654+0000","last_peered":"2024-04-27T03:21:01.248654+0000","last_clean":"2024-04-27T03:21:01.248654+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:01.248654+0000","last_undegraded":"2024-04-27T03:21:01.248654+0000","last_fullsized":"2024-04-27T03:21:01.248654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:22.252077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382608+0000","last_change":"2024-04-27T03:17:47.027621+0000","last_active":"2024-04-27T03:19:35.382608+0000","last_peered":"2024-04-27T03:19:35.382608+0000","last_clean":"2024-04-27T03:19:35.382608+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:19:35.382608+0000","last_undegraded":"2024-04-27T03:19:35.382608+0000","last_fullsized":"2024-04-27T03:19:35.382608+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377762+0000","last_change":"2024-04-27T03:17:51.032091+0000","last_active":"2024-04-27T03:19:35.377762+0000","last_peered":"2024-04-27T03:19:35.377762+0000","last_clean":"2024-04-27T03:19:35.377762+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:19:35.377762+0000","last_undegraded":"2024-04-27T03:19:35.377762+0000","last_fullsized":"2024-04-27T03:19:35.377762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:06.149814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'60","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.243349+0000","last_change":"2024-04-27T03:17:53.039949+0000","last_active":"2024-04-27T03:19:58.243349+0000","last_peered":"2024-04-27T03:19:58.243349+0000","last_clean":"2024-04-27T03:19:58.243349+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:19:58.243349+0000","last_undegraded":"2024-04-27T03:19:58.243349+0000","last_fullsized":"2024-04-27T03:19:58.243349+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.808839+0000","last_change":"2024-04-27T03:17:55.071558+0000","last_active":"2024-04-27T03:19:34.808839+0000","last_peered":"2024-04-27T03:19:34.808839+0000","last_clean":"2024-04-27T03:19:34.808839+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:19:34.808839+0000","last_undegraded":"2024-04-27T03:19:34.808839+0000","last_fullsized":"2024-04-27T03:19:34.808839+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:12.887800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.348234+0000","last_change":"2024-04-27T03:17:57.066613+0000","last_active":"2024-04-27T03:21:00.348234+0000","last_peered":"2024-04-27T03:21:00.348234+0000","last_clean":"2024-04-27T03:21:00.348234+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:21:00.348234+0000","last_undegraded":"2024-04-27T03:21:00.348234+0000","last_fullsized":"2024-04-27T03:21:00.348234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949296+0000","last_change":"2024-04-27T03:17:49.017718+0000","last_active":"2024-04-27T03:21:00.949296+0000","last_peered":"2024-04-27T03:21:00.949296+0000","last_clean":"2024-04-27T03:21:00.949296+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:00.949296+0000","last_undegraded":"2024-04-27T03:21:00.949296+0000","last_fullsized":"2024-04-27T03:21:00.949296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:40.378270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819810+0000","last_change":"2024-04-27T03:17:47.022941+0000","last_active":"2024-04-27T03:19:34.819810+0000","last_peered":"2024-04-27T03:19:34.819810+0000","last_clean":"2024-04-27T03:19:34.819810+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:19:34.819810+0000","last_undegraded":"2024-04-27T03:19:34.819810+0000","last_fullsized":"2024-04-27T03:19:34.819810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:09.787441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816507+0000","last_change":"2024-04-27T03:17:51.033578+0000","last_active":"2024-04-27T03:19:34.816507+0000","last_peered":"2024-04-27T03:19:34.816507+0000","last_clean":"2024-04-27T03:19:34.816507+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:19:34.816507+0000","last_undegraded":"2024-04-27T03:19:34.816507+0000","last_fullsized":"2024-04-27T03:19:34.816507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'38","reported_seq":90,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.244009+0000","last_change":"2024-04-27T03:17:53.040115+0000","last_active":"2024-04-27T03:19:58.244009+0000","last_peered":"2024-04-27T03:19:58.244009+0000","last_clean":"2024-04-27T03:19:58.244009+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:19:58.244009+0000","last_undegraded":"2024-04-27T03:19:58.244009+0000","last_fullsized":"2024-04-27T03:19:58.244009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:28.843466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815617+0000","last_change":"2024-04-27T03:17:55.058615+0000","last_active":"2024-04-27T03:19:34.815617+0000","last_peered":"2024-04-27T03:19:34.815617+0000","last_clean":"2024-04-27T03:19:34.815617+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:19:34.815617+0000","last_undegraded":"2024-04-27T03:19:34.815617+0000","last_fullsized":"2024-04-27T03:19:34.815617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.401928+0000","last_change":"2024-04-27T03:17:57.087706+0000","last_active":"2024-04-27T03:21:01.401928+0000","last_peered":"2024-04-27T03:21:01.401928+0000","last_clean":"2024-04-27T03:21:01.401928+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:21:01.401928+0000","last_undegraded":"2024-04-27T03:21:01.401928+0000","last_fullsized":"2024-04-27T03:21:01.401928+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:57.341894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950567+0000","last_change":"2024-04-27T03:17:49.011902+0000","last_active":"2024-04-27T03:21:00.950567+0000","last_peered":"2024-04-27T03:21:00.950567+0000","last_clean":"2024-04-27T03:21:00.950567+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:21:00.950567+0000","last_undegraded":"2024-04-27T03:21:00.950567+0000","last_fullsized":"2024-04-27T03:21:00.950567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.735456+0000","last_change":"2024-04-27T03:17:47.016344+0000","last_active":"2024-04-27T03:19:47.735456+0000","last_peered":"2024-04-27T03:19:47.735456+0000","last_clean":"2024-04-27T03:19:47.735456+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:19:47.735456+0000","last_undegraded":"2024-04-27T03:19:47.735456+0000","last_fullsized":"2024-04-27T03:19:47.735456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377191+0000","last_change":"2024-04-27T03:17:51.029804+0000","last_active":"2024-04-27T03:19:35.377191+0000","last_peered":"2024-04-27T03:19:35.377191+0000","last_clean":"2024-04-27T03:19:35.377191+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:19:35.377191+0000","last_undegraded":"2024-04-27T03:19:35.377191+0000","last_fullsized":"2024-04-27T03:19:35.377191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:13.111551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'32","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.286319+0000","last_change":"2024-04-27T03:17:53.041146+0000","last_active":"2024-04-27T03:19:58.286319+0000","last_peered":"2024-04-27T03:19:58.286319+0000","last_clean":"2024-04-27T03:19:58.286319+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:19:58.286319+0000","last_undegraded":"2024-04-27T03:19:58.286319+0000","last_fullsized":"2024-04-27T03:19:58.286319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:15.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816372+0000","last_change":"2024-04-27T03:17:55.071034+0000","last_active":"2024-04-27T03:19:34.816372+0000","last_peered":"2024-04-27T03:19:34.816372+0000","last_clean":"2024-04-27T03:19:34.816372+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:19:34.816372+0000","last_undegraded":"2024-04-27T03:19:34.816372+0000","last_fullsized":"2024-04-27T03:19:34.816372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.302058+0000","last_change":"2024-04-27T03:17:57.086785+0000","last_active":"2024-04-27T03:19:58.302058+0000","last_peered":"2024-04-27T03:19:58.302058+0000","last_clean":"2024-04-27T03:19:58.302058+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:19:58.302058+0000","last_undegraded":"2024-04-27T03:19:58.302058+0000","last_fullsized":"2024-04-27T03:19:58.302058+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:25:08.779674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397096+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.397096+0000","last_peered":"2024-04-27T03:21:01.397096+0000","last_clean":"2024-04-27T03:21:01.397096+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:01.397096+0000","last_undegraded":"2024-04-27T03:21:01.397096+0000","last_fullsized":"2024-04-27T03:21:01.397096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.019895+0000","last_change":"2024-04-27T03:17:47.002631+0000","last_active":"2024-04-27T03:20:00.019895+0000","last_peered":"2024-04-27T03:20:00.019895+0000","last_clean":"2024-04-27T03:20:00.019895+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:20:00.019895+0000","last_undegraded":"2024-04-27T03:20:00.019895+0000","last_fullsized":"2024-04-27T03:20:00.019895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'2","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.673843+0000","last_change":"2024-04-27T03:18:34.794992+0000","last_active":"2024-04-27T03:19:56.673843+0000","last_peered":"2024-04-27T03:19:56.673843+0000","last_clean":"2024-04-27T03:19:56.673843+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:19:56.673843+0000","last_undegraded":"2024-04-27T03:19:56.673843+0000","last_fullsized":"2024-04-27T03:19:56.673843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:56:59.713686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'44","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.246215+0000","last_change":"2024-04-27T03:17:53.039095+0000","last_active":"2024-04-27T03:21:00.246215+0000","last_peered":"2024-04-27T03:21:00.246215+0000","last_clean":"2024-04-27T03:21:00.246215+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:00.246215+0000","last_undegraded":"2024-04-27T03:21:00.246215+0000","last_fullsized":"2024-04-27T03:21:00.246215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":30,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438638+0000","last_change":"2024-04-27T03:17:55.063864+0000","last_active":"2024-04-27T03:19:35.438638+0000","last_peered":"2024-04-27T03:19:35.438638+0000","last_clean":"2024-04-27T03:19:35.438638+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:19:35.438638+0000","last_undegraded":"2024-04-27T03:19:35.438638+0000","last_fullsized":"2024-04-27T03:19:35.438638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:46.421450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.307196+0000","last_change":"2024-04-27T03:17:57.061982+0000","last_active":"2024-04-27T03:19:58.307196+0000","last_peered":"2024-04-27T03:19:58.307196+0000","last_clean":"2024-04-27T03:19:58.307196+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:19:58.307196+0000","last_undegraded":"2024-04-27T03:19:58.307196+0000","last_fullsized":"2024-04-27T03:19:58.307196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386099+0000","last_change":"2024-04-27T03:17:49.001808+0000","last_active":"2024-04-27T03:19:35.386099+0000","last_peered":"2024-04-27T03:19:35.386099+0000","last_clean":"2024-04-27T03:19:35.386099+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:19:35.386099+0000","last_undegraded":"2024-04-27T03:19:35.386099+0000","last_fullsized":"2024-04-27T03:19:35.386099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:33.807122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436337+0000","last_change":"2024-04-27T03:17:47.032632+0000","last_active":"2024-04-27T03:19:35.436337+0000","last_peered":"2024-04-27T03:19:35.436337+0000","last_clean":"2024-04-27T03:19:35.436337+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:19:35.436337+0000","last_undegraded":"2024-04-27T03:19:35.436337+0000","last_fullsized":"2024-04-27T03:19:35.436337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:53:43.154787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385434+0000","last_change":"2024-04-27T03:17:49.011651+0000","last_active":"2024-04-27T03:19:35.385434+0000","last_peered":"2024-04-27T03:19:35.385434+0000","last_clean":"2024-04-27T03:19:35.385434+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:19:35.385434+0000","last_undegraded":"2024-04-27T03:19:35.385434+0000","last_fullsized":"2024-04-27T03:19:35.385434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'1","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.732801+0000","last_change":"2024-04-27T03:18:34.773740+0000","last_active":"2024-04-27T03:19:52.732801+0000","last_peered":"2024-04-27T03:19:52.732801+0000","last_clean":"2024-04-27T03:19:52.732801+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:19:52.732801+0000","last_undegraded":"2024-04-27T03:19:52.732801+0000","last_fullsized":"2024-04-27T03:19:52.732801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:04.248651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639872,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111155+0000","last_change":"2024-04-27T03:18:34.784905+0000","last_active":"2024-04-27T03:19:56.111155+0000","last_peered":"2024-04-27T03:19:56.111155+0000","last_clean":"2024-04-27T03:19:56.111155+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:19:56.111155+0000","last_undegraded":"2024-04-27T03:19:56.111155+0000","last_fullsized":"2024-04-27T03:19:56.111155+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:28.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810481+0000","last_change":"2024-04-27T03:17:47.036148+0000","last_active":"2024-04-27T03:19:34.810481+0000","last_peered":"2024-04-27T03:19:34.810481+0000","last_clean":"2024-04-27T03:19:34.810481+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:19:34.810481+0000","last_undegraded":"2024-04-27T03:19:34.810481+0000","last_fullsized":"2024-04-27T03:19:34.810481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382422+0000","last_change":"2024-04-27T03:17:49.017582+0000","last_active":"2024-04-27T03:19:35.382422+0000","last_peered":"2024-04-27T03:19:35.382422+0000","last_clean":"2024-04-27T03:19:35.382422+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:19:35.382422+0000","last_undegraded":"2024-04-27T03:19:35.382422+0000","last_fullsized":"2024-04-27T03:19:35.382422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:20.541354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438828+0000","last_change":"2024-04-27T03:17:47.016123+0000","last_active":"2024-04-27T03:19:35.438828+0000","last_peered":"2024-04-27T03:19:35.438828+0000","last_clean":"2024-04-27T03:19:35.438828+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:19:35.438828+0000","last_undegraded":"2024-04-27T03:19:35.438828+0000","last_fullsized":"2024-04-27T03:19:35.438828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:59.109823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300171+0000","last_change":"2024-04-27T03:17:49.021184+0000","last_active":"2024-04-27T03:21:01.300171+0000","last_peered":"2024-04-27T03:21:01.300171+0000","last_clean":"2024-04-27T03:21:01.300171+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:01.300171+0000","last_undegraded":"2024-04-27T03:21:01.300171+0000","last_fullsized":"2024-04-27T03:21:01.300171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811874+0000","last_change":"2024-04-27T03:17:47.039423+0000","last_active":"2024-04-27T03:19:34.811874+0000","last_peered":"2024-04-27T03:19:34.811874+0000","last_clean":"2024-04-27T03:19:34.811874+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:19:34.811874+0000","last_undegraded":"2024-04-27T03:19:34.811874+0000","last_fullsized":"2024-04-27T03:19:34.811874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381685+0000","last_change":"2024-04-27T03:17:49.006801+0000","last_active":"2024-04-27T03:19:35.381685+0000","last_peered":"2024-04-27T03:19:35.381685+0000","last_clean":"2024-04-27T03:19:35.381685+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:19:35.381685+0000","last_undegraded":"2024-04-27T03:19:35.381685+0000","last_fullsized":"2024-04-27T03:19:35.381685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:39:13.312552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.947064+0000","last_change":"2024-04-27T03:17:47.030288+0000","last_active":"2024-04-27T03:19:59.947064+0000","last_peered":"2024-04-27T03:19:59.947064+0000","last_clean":"2024-04-27T03:19:59.947064+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:19:59.947064+0000","last_undegraded":"2024-04-27T03:19:59.947064+0000","last_fullsized":"2024-04-27T03:19:59.947064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:52:43.758948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300194+0000","last_change":"2024-04-27T03:17:49.015796+0000","last_active":"2024-04-27T03:21:01.300194+0000","last_peered":"2024-04-27T03:21:01.300194+0000","last_clean":"2024-04-27T03:21:01.300194+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:01.300194+0000","last_undegraded":"2024-04-27T03:21:01.300194+0000","last_fullsized":"2024-04-27T03:21:01.300194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.756327+0000","last_change":"2024-04-27T03:17:47.031857+0000","last_active":"2024-04-27T03:21:00.756327+0000","last_peered":"2024-04-27T03:21:00.756327+0000","last_clean":"2024-04-27T03:21:00.756327+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:00.756327+0000","last_undegraded":"2024-04-27T03:21:00.756327+0000","last_fullsized":"2024-04-27T03:21:00.756327+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":78,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949139+0000","last_change":"2024-04-27T03:17:49.017220+0000","last_active":"2024-04-27T03:21:00.949139+0000","last_peered":"2024-04-27T03:21:00.949139+0000","last_clean":"2024-04-27T03:21:00.949139+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:00.949139+0000","last_undegraded":"2024-04-27T03:21:00.949139+0000","last_fullsized":"2024-04-27T03:21:00.949139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815261+0000","last_change":"2024-04-27T03:17:47.039005+0000","last_active":"2024-04-27T03:19:34.815261+0000","last_peered":"2024-04-27T03:19:34.815261+0000","last_clean":"2024-04-27T03:19:34.815261+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:19:34.815261+0000","last_undegraded":"2024-04-27T03:19:34.815261+0000","last_fullsized":"2024-04-27T03:19:34.815261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299434+0000","last_change":"2024-04-27T03:17:49.011708+0000","last_active":"2024-04-27T03:21:01.299434+0000","last_peered":"2024-04-27T03:21:01.299434+0000","last_clean":"2024-04-27T03:21:01.299434+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:01.299434+0000","last_undegraded":"2024-04-27T03:21:01.299434+0000","last_fullsized":"2024-04-27T03:21:01.299434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:52.139159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809399+0000","last_change":"2024-04-27T03:17:47.016024+0000","last_active":"2024-04-27T03:19:34.809399+0000","last_peered":"2024-04-27T03:19:34.809399+0000","last_clean":"2024-04-27T03:19:34.809399+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:19:34.809399+0000","last_undegraded":"2024-04-27T03:19:34.809399+0000","last_fullsized":"2024-04-27T03:19:34.809399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300134+0000","last_change":"2024-04-27T03:17:49.010865+0000","last_active":"2024-04-27T03:21:01.300134+0000","last_peered":"2024-04-27T03:21:01.300134+0000","last_clean":"2024-04-27T03:21:01.300134+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:21:01.300134+0000","last_undegraded":"2024-04-27T03:21:01.300134+0000","last_fullsized":"2024-04-27T03:21:01.300134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810923+0000","last_change":"2024-04-27T03:17:47.019986+0000","last_active":"2024-04-27T03:19:34.810923+0000","last_peered":"2024-04-27T03:19:34.810923+0000","last_clean":"2024-04-27T03:19:34.810923+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:19:34.810923+0000","last_undegraded":"2024-04-27T03:19:34.810923+0000","last_fullsized":"2024-04-27T03:19:34.810923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949386+0000","last_change":"2024-04-27T03:17:49.005667+0000","last_active":"2024-04-27T03:21:00.949386+0000","last_peered":"2024-04-27T03:21:00.949386+0000","last_clean":"2024-04-27T03:21:00.949386+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:00.949386+0000","last_undegraded":"2024-04-27T03:21:00.949386+0000","last_fullsized":"2024-04-27T03:21:00.949386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:07.432944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817074+0000","last_change":"2024-04-27T03:17:47.023505+0000","last_active":"2024-04-27T03:19:34.817074+0000","last_peered":"2024-04-27T03:19:34.817074+0000","last_clean":"2024-04-27T03:19:34.817074+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:19:34.817074+0000","last_undegraded":"2024-04-27T03:19:34.817074+0000","last_fullsized":"2024-04-27T03:19:34.817074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382540+0000","last_change":"2024-04-27T03:17:49.009462+0000","last_active":"2024-04-27T03:19:35.382540+0000","last_peered":"2024-04-27T03:19:35.382540+0000","last_clean":"2024-04-27T03:19:35.382540+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:19:35.382540+0000","last_undegraded":"2024-04-27T03:19:35.382540+0000","last_fullsized":"2024-04-27T03:19:35.382540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:32.201166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382267+0000","last_change":"2024-04-27T03:17:47.015754+0000","last_active":"2024-04-27T03:19:35.382267+0000","last_peered":"2024-04-27T03:19:35.382267+0000","last_clean":"2024-04-27T03:19:35.382267+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:19:35.382267+0000","last_undegraded":"2024-04-27T03:19:35.382267+0000","last_fullsized":"2024-04-27T03:19:35.382267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'7","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744638+0000","last_change":"2024-04-27T03:17:49.017194+0000","last_active":"2024-04-27T03:20:07.744638+0000","last_peered":"2024-04-27T03:20:07.744638+0000","last_clean":"2024-04-27T03:20:07.744638+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:20:07.744638+0000","last_undegraded":"2024-04-27T03:20:07.744638+0000","last_fullsized":"2024-04-27T03:20:07.744638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812147+0000","last_change":"2024-04-27T03:17:47.038806+0000","last_active":"2024-04-27T03:19:34.812147+0000","last_peered":"2024-04-27T03:19:34.812147+0000","last_clean":"2024-04-27T03:19:34.812147+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:19:34.812147+0000","last_undegraded":"2024-04-27T03:19:34.812147+0000","last_fullsized":"2024-04-27T03:19:34.812147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.808067+0000","last_change":"2024-04-27T03:17:49.006535+0000","last_active":"2024-04-27T03:21:00.808067+0000","last_peered":"2024-04-27T03:21:00.808067+0000","last_clean":"2024-04-27T03:21:00.808067+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:00.808067+0000","last_undegraded":"2024-04-27T03:21:00.808067+0000","last_fullsized":"2024-04-27T03:21:00.808067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:21.664514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442247+0000","last_change":"2024-04-27T03:19:35.441020+0000","last_active":"2024-04-27T03:19:35.442247+0000","last_peered":"2024-04-27T03:19:35.442247+0000","last_clean":"2024-04-27T03:19:35.442247+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:19:35.442247+0000","last_undegraded":"2024-04-27T03:19:35.442247+0000","last_fullsized":"2024-04-27T03:19:35.442247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:39:26.708815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814170+0000","last_change":"2024-04-27T03:17:49.016545+0000","last_active":"2024-04-27T03:19:34.814170+0000","last_peered":"2024-04-27T03:19:34.814170+0000","last_clean":"2024-04-27T03:19:34.814170+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:19:34.814170+0000","last_undegraded":"2024-04-27T03:19:34.814170+0000","last_fullsized":"2024-04-27T03:19:34.814170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:52.886758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442831+0000","last_change":"2024-04-27T03:17:47.022441+0000","last_active":"2024-04-27T03:19:35.442831+0000","last_peered":"2024-04-27T03:19:35.442831+0000","last_clean":"2024-04-27T03:19:35.442831+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:19:35.442831+0000","last_undegraded":"2024-04-27T03:19:35.442831+0000","last_fullsized":"2024-04-27T03:19:35.442831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146608+0000","last_change":"2024-04-27T03:17:49.018017+0000","last_active":"2024-04-27T03:21:01.146608+0000","last_peered":"2024-04-27T03:21:01.146608+0000","last_clean":"2024-04-27T03:21:01.146608+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:01.146608+0000","last_undegraded":"2024-04-27T03:21:01.146608+0000","last_fullsized":"2024-04-27T03:21:01.146608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815679+0000","last_change":"2024-04-27T03:17:47.026063+0000","last_active":"2024-04-27T03:19:34.815679+0000","last_peered":"2024-04-27T03:19:34.815679+0000","last_clean":"2024-04-27T03:19:34.815679+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:19:34.815679+0000","last_undegraded":"2024-04-27T03:19:34.815679+0000","last_fullsized":"2024-04-27T03:19:34.815679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814107+0000","last_change":"2024-04-27T03:17:47.002843+0000","last_active":"2024-04-27T03:19:34.814107+0000","last_peered":"2024-04-27T03:19:34.814107+0000","last_clean":"2024-04-27T03:19:34.814107+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:19:34.814107+0000","last_undegraded":"2024-04-27T03:19:34.814107+0000","last_fullsized":"2024-04-27T03:19:34.814107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'11","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164859+0000","last_change":"2024-04-27T03:17:49.014133+0000","last_active":"2024-04-27T03:21:01.164859+0000","last_peered":"2024-04-27T03:21:01.164859+0000","last_clean":"2024-04-27T03:21:01.164859+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:01.164859+0000","last_undegraded":"2024-04-27T03:21:01.164859+0000","last_fullsized":"2024-04-27T03:21:01.164859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:38.383961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379782+0000","last_change":"2024-04-27T03:17:47.022513+0000","last_active":"2024-04-27T03:19:35.379782+0000","last_peered":"2024-04-27T03:19:35.379782+0000","last_clean":"2024-04-27T03:19:35.379782+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:19:35.379782+0000","last_undegraded":"2024-04-27T03:19:35.379782+0000","last_fullsized":"2024-04-27T03:19:35.379782+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:24.322205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385155+0000","last_change":"2024-04-27T03:17:49.012955+0000","last_active":"2024-04-27T03:19:35.385155+0000","last_peered":"2024-04-27T03:19:35.385155+0000","last_clean":"2024-04-27T03:19:35.385155+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:19:35.385155+0000","last_undegraded":"2024-04-27T03:19:35.385155+0000","last_fullsized":"2024-04-27T03:19:35.385155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2256,"ondisk_log_size":2256,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":13384,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1690,"num_read_kb":1444,"num_write":723,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":10824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1470,"ondisk_log_size":1470,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":961,"num_read_kb":961,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":473,"ondisk_log_size":473,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":25805696,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":309,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38719488,"data_stored":38719488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46056,"kb_used_data":6560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94325784,"statfs":{"total":96636764160,"available":96589602816,"internally_reserved":0,"allocated":6717440,"data_stored":6440144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57224,"kb_used_data":13568,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94314616,"statfs":{"total":96636764160,"available":96578166784,"internally_reserved":0,"allocated":13893632,"data_stored":13644798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":9524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":9752576,"data_stored":9451527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50504,"kb_used_data":11008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94321336,"statfs":{"total":96636764160,"available":96585048064,"internally_reserved":0,"allocated":11272192,"data_stored":10958989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10809344,"data_stored":10809344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8712192,"data_stored":8712192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":12906496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":4422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:21:08.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.934+0000 7f16ce769640 1 Processor -- start 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.934+0000 7f16ce769640 1 -- start start 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16ce769640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16c80963b0 0x7f16c8096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16ce769640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16c8098d10 con 0x7f16c806d4d0 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16ce769640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f16c8098610 con 0x7f16c80963b0 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16c80963b0 0x7f16c8096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c7fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f16c80963b0 0x7f16c8096790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56756/0 (socket says 172.21.15.77:56756) 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c7fff640 1 -- 172.21.15.77:0/3554862394 learned_addr learned my addr 172.21.15.77:0/3554862394 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:08.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c77fe640 1 --1- 172.21.15.77:0/3554862394 >> v1:172.21.15.77:6789/0 conn(0x7f16c806d4d0 0x7f16c806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46692/0 (socket says 172.21.15.77:46692) 2024-04-27T03:21:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c7fff640 1 -- 172.21.15.77:0/3554862394 >> v1:172.21.15.77:6789/0 conn(0x7f16c806d4d0 legacy=0x7f16c806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c7fff640 1 -- 172.21.15.77:0/3554862394 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c8071f90 con 0x7f16c80963b0 2024-04-27T03:21:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c7fff640 1 --2- 172.21.15.77:0/3554862394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c80963b0 0x7f16c8096790 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009e50 tx=0x7f16c001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5cb18861a7fff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c6ffd640 1 -- 172.21.15.77:0/3554862394 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c0009200 con 0x7f16c80963b0 2024-04-27T03:21:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c6ffd640 1 -- 172.21.15.77:0/3554862394 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c00a5590 con 0x7f16c80963b0 2024-04-27T03:21:08.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16c6ffd640 1 -- 172.21.15.77:0/3554862394 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c00a5890 con 0x7f16c80963b0 2024-04-27T03:21:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16ce769640 1 -- 172.21.15.77:0/3554862394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c80963b0 msgr2=0x7f16c8096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.938+0000 7f16ce769640 1 --2- 172.21.15.77:0/3554862394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c80963b0 0x7f16c8096790 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009e50 tx=0x7f16c001ca10 comp rx=0 tx=0).stop 2024-04-27T03:21:08.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- 172.21.15.77:0/3554862394 shutdown_connections 2024-04-27T03:21:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 --2- 172.21.15.77:0/3554862394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c80963b0 0x7f16c8096790 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- 172.21.15.77:0/3554862394 >> 172.21.15.77:0/3554862394 conn(0x7f16c8099440 msgr2=0x7f16c8099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- 172.21.15.77:0/3554862394 shutdown_connections 2024-04-27T03:21:08.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- 172.21.15.77:0/3554862394 wait complete. 2024-04-27T03:21:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 Processor -- start 2024-04-27T03:21:08.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- start start 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c806d4d0 0x7f16c81c4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f16c8134d80 con 0x7f16c806d4d0 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c806d4d0 0x7f16c81c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c7fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c806d4d0 0x7f16c81c4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56770/0 (socket says 172.21.15.77:56770) 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c7fff640 1 -- 172.21.15.77:0/570626373 learned_addr learned my addr 172.21.15.77:0/570626373 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c7fff640 1 -- 172.21.15.77:0/570626373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c81234a0 con 0x7f16c806d4d0 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c7fff640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c806d4d0 0x7f16c81c4550 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f16c001a640 tx=0x7f16c00197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c4ff9640 1 -- 172.21.15.77:0/570626373 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c00bc020 con 0x7f16c806d4d0 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c4ff9640 1 -- 172.21.15.77:0/570626373 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c00b8470 con 0x7f16c806d4d0 2024-04-27T03:21:08.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c4ff9640 1 -- 172.21.15.77:0/570626373 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c00b8750 con 0x7f16c806d4d0 2024-04-27T03:21:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c8127130 con 0x7f16c806d4d0 2024-04-27T03:21:08.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c81245b0 con 0x7f16c806d4d0 2024-04-27T03:21:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c80960f0 con 0x7f16c806d4d0 2024-04-27T03:21:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c4ff9640 1 -- 172.21.15.77:0/570626373 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f16c00bf020 con 0x7f16c806d4d0 2024-04-27T03:21:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c4ff9640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f16a8035db0 0x7f16a8038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c4ff9640 1 -- 172.21.15.77:0/570626373 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f16c00e4380 con 0x7f16c806d4d0 2024-04-27T03:21:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.942+0000 7f16c77fe640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f16a8035db0 0x7f16a8038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:08.956 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.946+0000 7f16c77fe640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f16a8035db0 0x7f16a8038270 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f16b0000f70 tx=0x7f16b0007a90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:08.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:08.946+0000 7f16c4ff9640 1 -- 172.21.15.77:0/570626373 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f16c00c6030 con 0x7f16c806d4d0 2024-04-27T03:21:09.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.110+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f16c8062e50 con 0x7f16a8035db0 2024-04-27T03:21:09.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.162+0000 7f16c4ff9640 1 -- 172.21.15.77:0/570626373 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+665756 (secure 0 0 0) 0x7f16c8062e50 con 0x7f16a8035db0 2024-04-27T03:21:09.177 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:21:09.183 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:21:09.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f16a8035db0 msgr2=0x7f16a8038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f16a8035db0 0x7f16a8038270 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f16b0000f70 tx=0x7f16b0007a90 comp rx=0 tx=0).stop 2024-04-27T03:21:09.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c806d4d0 msgr2=0x7f16c81c4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c806d4d0 0x7f16c81c4550 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f16c001a640 tx=0x7f16c00197b0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 shutdown_connections 2024-04-27T03:21:09.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f16a8035db0 0x7f16a8038270 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 --2- 172.21.15.77:0/570626373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f16c806d4d0 0x7f16c81c4550 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 >> 172.21.15.77:0/570626373 conn(0x7f16c8099440 msgr2=0x7f16c809a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:09.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 shutdown_connections 2024-04-27T03:21:09.185 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.166+0000 7f16ce769640 1 -- 172.21.15.77:0/570626373 wait complete. 2024-04-27T03:21:09.203 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-27T03:21:07.732296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26412789,"num_objects":259,"num_object_clones":0,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":5177,"num_read_kb":4722,"num_write":2298,"num_write_kb":61373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4672,"ondisk_log_size":4672,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":190324,"kb_used_data":40660,"kb_used_omap":6,"kb_used_meta":149625,"kb_avail":377297036,"statfs":{"total":386547056640,"available":386352164864,"internally_reserved":0,"allocated":41635840,"data_stored":40495458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":153216790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25811687,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":510,"num_read_kb":465,"num_write":386,"num_write_kb":29372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005707"},"pg_stats":[{"pgid":"3.3f","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741206+0000","last_change":"2024-04-27T03:17:47.025622+0000","last_active":"2024-04-27T03:19:52.741206+0000","last_peered":"2024-04-27T03:19:52.741206+0000","last_clean":"2024-04-27T03:19:52.741206+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:19:52.741206+0000","last_undegraded":"2024-04-27T03:19:52.741206+0000","last_fullsized":"2024-04-27T03:19:52.741206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:56.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'5","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246474+0000","last_change":"2024-04-27T03:17:49.019025+0000","last_active":"2024-04-27T03:21:01.246474+0000","last_peered":"2024-04-27T03:21:01.246474+0000","last_clean":"2024-04-27T03:21:01.246474+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:01.246474+0000","last_undegraded":"2024-04-27T03:21:01.246474+0000","last_fullsized":"2024-04-27T03:21:01.246474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385062+0000","last_change":"2024-04-27T03:17:47.023458+0000","last_active":"2024-04-27T03:19:35.385062+0000","last_peered":"2024-04-27T03:19:35.385062+0000","last_clean":"2024-04-27T03:19:35.385062+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:19:35.385062+0000","last_undegraded":"2024-04-27T03:19:35.385062+0000","last_fullsized":"2024-04-27T03:19:35.385062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810083+0000","last_change":"2024-04-27T03:17:49.020940+0000","last_active":"2024-04-27T03:19:34.810083+0000","last_peered":"2024-04-27T03:19:34.810083+0000","last_clean":"2024-04-27T03:19:34.810083+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:19:34.810083+0000","last_undegraded":"2024-04-27T03:19:34.810083+0000","last_fullsized":"2024-04-27T03:19:34.810083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811946+0000","last_change":"2024-04-27T03:17:47.039594+0000","last_active":"2024-04-27T03:19:34.811946+0000","last_peered":"2024-04-27T03:19:34.811946+0000","last_clean":"2024-04-27T03:19:34.811946+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:19:34.811946+0000","last_undegraded":"2024-04-27T03:19:34.811946+0000","last_fullsized":"2024-04-27T03:19:34.811946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111615+0000","last_change":"2024-04-27T03:17:49.016480+0000","last_active":"2024-04-27T03:19:56.111615+0000","last_peered":"2024-04-27T03:19:56.111615+0000","last_clean":"2024-04-27T03:19:56.111615+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:19:56.111615+0000","last_undegraded":"2024-04-27T03:19:56.111615+0000","last_fullsized":"2024-04-27T03:19:56.111615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439532+0000","last_change":"2024-04-27T03:17:47.021409+0000","last_active":"2024-04-27T03:19:35.439532+0000","last_peered":"2024-04-27T03:19:35.439532+0000","last_clean":"2024-04-27T03:19:35.439532+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:19:35.439532+0000","last_undegraded":"2024-04-27T03:19:35.439532+0000","last_fullsized":"2024-04-27T03:19:35.439532+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:24.826641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948899+0000","last_change":"2024-04-27T03:17:49.021774+0000","last_active":"2024-04-27T03:21:00.948899+0000","last_peered":"2024-04-27T03:21:00.948899+0000","last_clean":"2024-04-27T03:21:00.948899+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:00.948899+0000","last_undegraded":"2024-04-27T03:21:00.948899+0000","last_fullsized":"2024-04-27T03:21:00.948899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817333+0000","last_change":"2024-04-27T03:17:47.015865+0000","last_active":"2024-04-27T03:19:34.817333+0000","last_peered":"2024-04-27T03:19:34.817333+0000","last_clean":"2024-04-27T03:19:34.817333+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:19:34.817333+0000","last_undegraded":"2024-04-27T03:19:34.817333+0000","last_fullsized":"2024-04-27T03:19:34.817333+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439003+0000","last_change":"2024-04-27T03:18:34.776780+0000","last_active":"2024-04-27T03:19:35.439003+0000","last_peered":"2024-04-27T03:19:35.439003+0000","last_clean":"2024-04-27T03:19:35.439003+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:19:35.439003+0000","last_undegraded":"2024-04-27T03:19:35.439003+0000","last_fullsized":"2024-04-27T03:19:35.439003+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:30.737857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.111562+0000","last_change":"2024-04-27T03:17:47.026206+0000","last_active":"2024-04-27T03:20:00.111562+0000","last_peered":"2024-04-27T03:20:00.111562+0000","last_clean":"2024-04-27T03:20:00.111562+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:20:00.111562+0000","last_undegraded":"2024-04-27T03:20:00.111562+0000","last_fullsized":"2024-04-27T03:20:00.111562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'11","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742400+0000","last_change":"2024-04-27T03:17:49.015210+0000","last_active":"2024-04-27T03:19:57.742400+0000","last_peered":"2024-04-27T03:19:57.742400+0000","last_clean":"2024-04-27T03:19:57.742400+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:19:57.742400+0000","last_undegraded":"2024-04-27T03:19:57.742400+0000","last_fullsized":"2024-04-27T03:19:57.742400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811270+0000","last_change":"2024-04-27T03:17:47.036460+0000","last_active":"2024-04-27T03:19:34.811270+0000","last_peered":"2024-04-27T03:19:34.811270+0000","last_clean":"2024-04-27T03:19:34.811270+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:19:34.811270+0000","last_undegraded":"2024-04-27T03:19:34.811270+0000","last_fullsized":"2024-04-27T03:19:34.811270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:56.284851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298837+0000","last_change":"2024-04-27T03:17:49.024190+0000","last_active":"2024-04-27T03:21:01.298837+0000","last_peered":"2024-04-27T03:21:01.298837+0000","last_clean":"2024-04-27T03:21:01.298837+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:01.298837+0000","last_undegraded":"2024-04-27T03:21:01.298837+0000","last_fullsized":"2024-04-27T03:21:01.298837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810250+0000","last_change":"2024-04-27T03:17:47.013877+0000","last_active":"2024-04-27T03:19:34.810250+0000","last_peered":"2024-04-27T03:19:34.810250+0000","last_clean":"2024-04-27T03:19:34.810250+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:19:34.810250+0000","last_undegraded":"2024-04-27T03:19:34.810250+0000","last_fullsized":"2024-04-27T03:19:34.810250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.444094+0000","last_change":"2024-04-27T03:17:49.020265+0000","last_active":"2024-04-27T03:19:35.444094+0000","last_peered":"2024-04-27T03:19:35.444094+0000","last_clean":"2024-04-27T03:19:35.444094+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:19:35.444094+0000","last_undegraded":"2024-04-27T03:19:35.444094+0000","last_fullsized":"2024-04-27T03:19:35.444094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:23.713924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298707+0000","last_change":"2024-04-27T03:17:49.019484+0000","last_active":"2024-04-27T03:21:01.298707+0000","last_peered":"2024-04-27T03:21:01.298707+0000","last_clean":"2024-04-27T03:21:01.298707+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:01.298707+0000","last_undegraded":"2024-04-27T03:21:01.298707+0000","last_fullsized":"2024-04-27T03:21:01.298707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385569+0000","last_change":"2024-04-27T03:17:47.027209+0000","last_active":"2024-04-27T03:19:35.385569+0000","last_peered":"2024-04-27T03:19:35.385569+0000","last_clean":"2024-04-27T03:19:35.385569+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:19:35.385569+0000","last_undegraded":"2024-04-27T03:19:35.385569+0000","last_fullsized":"2024-04-27T03:19:35.385569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814961+0000","last_change":"2024-04-27T03:17:49.013701+0000","last_active":"2024-04-27T03:19:34.814961+0000","last_peered":"2024-04-27T03:19:34.814961+0000","last_clean":"2024-04-27T03:19:34.814961+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:19:34.814961+0000","last_undegraded":"2024-04-27T03:19:34.814961+0000","last_fullsized":"2024-04-27T03:19:34.814961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.443009+0000","last_change":"2024-04-27T03:17:47.015536+0000","last_active":"2024-04-27T03:19:35.443009+0000","last_peered":"2024-04-27T03:19:35.443009+0000","last_clean":"2024-04-27T03:19:35.443009+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:19:35.443009+0000","last_undegraded":"2024-04-27T03:19:35.443009+0000","last_fullsized":"2024-04-27T03:19:35.443009+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:51:44.696219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.809825+0000","last_change":"2024-04-27T03:17:49.011747+0000","last_active":"2024-04-27T03:21:00.809825+0000","last_peered":"2024-04-27T03:21:00.809825+0000","last_clean":"2024-04-27T03:21:00.809825+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:21:00.809825+0000","last_undegraded":"2024-04-27T03:21:00.809825+0000","last_fullsized":"2024-04-27T03:21:00.809825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820563+0000","last_change":"2024-04-27T03:17:47.023848+0000","last_active":"2024-04-27T03:19:34.820563+0000","last_peered":"2024-04-27T03:19:34.820563+0000","last_clean":"2024-04-27T03:19:34.820563+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:19:34.820563+0000","last_undegraded":"2024-04-27T03:19:34.820563+0000","last_fullsized":"2024-04-27T03:19:34.820563+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.231670+0000","last_change":"2024-04-27T03:17:49.019463+0000","last_active":"2024-04-27T03:21:01.231670+0000","last_peered":"2024-04-27T03:21:01.231670+0000","last_clean":"2024-04-27T03:21:01.231670+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:01.231670+0000","last_undegraded":"2024-04-27T03:21:01.231670+0000","last_fullsized":"2024-04-27T03:21:01.231670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:53.154789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816701+0000","last_change":"2024-04-27T03:17:47.012402+0000","last_active":"2024-04-27T03:19:34.816701+0000","last_peered":"2024-04-27T03:19:34.816701+0000","last_clean":"2024-04-27T03:19:34.816701+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:19:34.816701+0000","last_undegraded":"2024-04-27T03:19:34.816701+0000","last_fullsized":"2024-04-27T03:19:34.816701+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146748+0000","last_change":"2024-04-27T03:17:49.008483+0000","last_active":"2024-04-27T03:21:01.146748+0000","last_peered":"2024-04-27T03:21:01.146748+0000","last_clean":"2024-04-27T03:21:01.146748+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:01.146748+0000","last_undegraded":"2024-04-27T03:21:01.146748+0000","last_fullsized":"2024-04-27T03:21:01.146748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:11.803924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812204+0000","last_change":"2024-04-27T03:17:47.036082+0000","last_active":"2024-04-27T03:19:34.812204+0000","last_peered":"2024-04-27T03:19:34.812204+0000","last_clean":"2024-04-27T03:19:34.812204+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:19:34.812204+0000","last_undegraded":"2024-04-27T03:19:34.812204+0000","last_fullsized":"2024-04-27T03:19:34.812204+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164153+0000","last_change":"2024-04-27T03:17:49.005822+0000","last_active":"2024-04-27T03:21:01.164153+0000","last_peered":"2024-04-27T03:21:01.164153+0000","last_clean":"2024-04-27T03:21:01.164153+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:01.164153+0000","last_undegraded":"2024-04-27T03:21:01.164153+0000","last_fullsized":"2024-04-27T03:21:01.164153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:40.042792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811583+0000","last_change":"2024-04-27T03:17:47.033466+0000","last_active":"2024-04-27T03:19:34.811583+0000","last_peered":"2024-04-27T03:19:34.811583+0000","last_clean":"2024-04-27T03:19:34.811583+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:19:34.811583+0000","last_undegraded":"2024-04-27T03:19:34.811583+0000","last_fullsized":"2024-04-27T03:19:34.811583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'37","reported_seq":105,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.190758+0000","last_change":"2024-04-27T03:17:57.088698+0000","last_active":"2024-04-27T03:21:01.190758+0000","last_peered":"2024-04-27T03:21:01.190758+0000","last_clean":"2024-04-27T03:21:01.190758+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:21:01.190758+0000","last_undegraded":"2024-04-27T03:21:01.190758+0000","last_fullsized":"2024-04-27T03:21:01.190758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:57.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233462+0000","last_change":"2024-04-27T03:17:49.015482+0000","last_active":"2024-04-27T03:21:01.233462+0000","last_peered":"2024-04-27T03:21:01.233462+0000","last_clean":"2024-04-27T03:21:01.233462+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:01.233462+0000","last_undegraded":"2024-04-27T03:21:01.233462+0000","last_fullsized":"2024-04-27T03:21:01.233462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811777+0000","last_change":"2024-04-27T03:17:47.018824+0000","last_active":"2024-04-27T03:19:34.811777+0000","last_peered":"2024-04-27T03:19:34.811777+0000","last_clean":"2024-04-27T03:19:34.811777+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:19:34.811777+0000","last_undegraded":"2024-04-27T03:19:34.811777+0000","last_fullsized":"2024-04-27T03:19:34.811777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815532+0000","last_change":"2024-04-27T03:17:51.032891+0000","last_active":"2024-04-27T03:19:34.815532+0000","last_peered":"2024-04-27T03:19:34.815532+0000","last_clean":"2024-04-27T03:19:34.815532+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:19:34.815532+0000","last_undegraded":"2024-04-27T03:19:34.815532+0000","last_fullsized":"2024-04-27T03:19:34.815532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'43","reported_seq":101,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.274412+0000","last_change":"2024-04-27T03:17:53.040688+0000","last_active":"2024-04-27T03:19:58.274412+0000","last_peered":"2024-04-27T03:19:58.274412+0000","last_clean":"2024-04-27T03:19:58.274412+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:19:58.274412+0000","last_undegraded":"2024-04-27T03:19:58.274412+0000","last_fullsized":"2024-04-27T03:19:58.274412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819799+0000","last_change":"2024-04-27T03:17:55.072849+0000","last_active":"2024-04-27T03:19:34.819799+0000","last_peered":"2024-04-27T03:19:34.819799+0000","last_clean":"2024-04-27T03:19:34.819799+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:19:34.819799+0000","last_undegraded":"2024-04-27T03:19:34.819799+0000","last_fullsized":"2024-04-27T03:19:34.819799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.848140+0000","last_change":"2024-04-27T03:17:57.061550+0000","last_active":"2024-04-27T03:21:00.848140+0000","last_peered":"2024-04-27T03:21:00.848140+0000","last_clean":"2024-04-27T03:21:00.848140+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:00.848140+0000","last_undegraded":"2024-04-27T03:21:00.848140+0000","last_fullsized":"2024-04-27T03:21:00.848140+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:54:11.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814870+0000","last_change":"2024-04-27T03:17:49.017246+0000","last_active":"2024-04-27T03:19:34.814870+0000","last_peered":"2024-04-27T03:19:34.814870+0000","last_clean":"2024-04-27T03:19:34.814870+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:19:34.814870+0000","last_undegraded":"2024-04-27T03:19:34.814870+0000","last_fullsized":"2024-04-27T03:19:34.814870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:15.229307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436921+0000","last_change":"2024-04-27T03:17:47.032159+0000","last_active":"2024-04-27T03:19:35.436921+0000","last_peered":"2024-04-27T03:19:35.436921+0000","last_clean":"2024-04-27T03:19:35.436921+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:19:35.436921+0000","last_undegraded":"2024-04-27T03:19:35.436921+0000","last_fullsized":"2024-04-27T03:19:35.436921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:43.992214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815036+0000","last_change":"2024-04-27T03:17:51.012211+0000","last_active":"2024-04-27T03:19:34.815036+0000","last_peered":"2024-04-27T03:19:34.815036+0000","last_clean":"2024-04-27T03:19:34.815036+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:19:34.815036+0000","last_undegraded":"2024-04-27T03:19:34.815036+0000","last_fullsized":"2024-04-27T03:19:34.815036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"27'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384958+0000","last_change":"2024-04-27T03:17:53.049040+0000","last_active":"2024-04-27T03:19:35.384958+0000","last_peered":"2024-04-27T03:19:35.384958+0000","last_clean":"2024-04-27T03:19:35.384958+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:19:35.384958+0000","last_undegraded":"2024-04-27T03:19:35.384958+0000","last_fullsized":"2024-04-27T03:19:35.384958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:34.768461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819788+0000","last_change":"2024-04-27T03:17:55.058791+0000","last_active":"2024-04-27T03:19:34.819788+0000","last_peered":"2024-04-27T03:19:34.819788+0000","last_clean":"2024-04-27T03:19:34.819788+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:19:34.819788+0000","last_undegraded":"2024-04-27T03:19:34.819788+0000","last_fullsized":"2024-04-27T03:19:34.819788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809843+0000","last_change":"2024-04-27T03:17:57.074831+0000","last_active":"2024-04-27T03:19:34.809843+0000","last_peered":"2024-04-27T03:19:34.809843+0000","last_clean":"2024-04-27T03:19:34.809843+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:19:34.809843+0000","last_undegraded":"2024-04-27T03:19:34.809843+0000","last_fullsized":"2024-04-27T03:19:34.809843+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'15","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.165698+0000","last_change":"2024-04-27T03:17:49.022969+0000","last_active":"2024-04-27T03:21:01.165698+0000","last_peered":"2024-04-27T03:21:01.165698+0000","last_clean":"2024-04-27T03:21:01.165698+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:01.165698+0000","last_undegraded":"2024-04-27T03:21:01.165698+0000","last_fullsized":"2024-04-27T03:21:01.165698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814358+0000","last_change":"2024-04-27T03:17:47.009721+0000","last_active":"2024-04-27T03:19:34.814358+0000","last_peered":"2024-04-27T03:19:34.814358+0000","last_clean":"2024-04-27T03:19:34.814358+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:19:34.814358+0000","last_undegraded":"2024-04-27T03:19:34.814358+0000","last_fullsized":"2024-04-27T03:19:34.814358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814852+0000","last_change":"2024-04-27T03:17:51.034093+0000","last_active":"2024-04-27T03:19:34.814852+0000","last_peered":"2024-04-27T03:19:34.814852+0000","last_clean":"2024-04-27T03:19:34.814852+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:19:34.814852+0000","last_undegraded":"2024-04-27T03:19:34.814852+0000","last_fullsized":"2024-04-27T03:19:34.814852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"27'22","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440526+0000","last_change":"2024-04-27T03:17:53.032443+0000","last_active":"2024-04-27T03:19:35.440526+0000","last_peered":"2024-04-27T03:19:35.440526+0000","last_clean":"2024-04-27T03:19:35.440526+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:19:35.440526+0000","last_undegraded":"2024-04-27T03:19:35.440526+0000","last_fullsized":"2024-04-27T03:19:35.440526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:35.232648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'282","reported_seq":426,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277953+0000","last_change":"2024-04-27T03:17:55.069049+0000","last_active":"2024-04-27T03:21:01.277953+0000","last_peered":"2024-04-27T03:21:01.277953+0000","last_clean":"2024-04-27T03:21:01.277953+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:01.277953+0000","last_undegraded":"2024-04-27T03:21:01.277953+0000","last_fullsized":"2024-04-27T03:21:01.277953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438345+0000","last_change":"2024-04-27T03:17:57.072058+0000","last_active":"2024-04-27T03:19:35.438345+0000","last_peered":"2024-04-27T03:19:35.438345+0000","last_clean":"2024-04-27T03:19:35.438345+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:19:35.438345+0000","last_undegraded":"2024-04-27T03:19:35.438345+0000","last_fullsized":"2024-04-27T03:19:35.438345+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:56.749229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950160+0000","last_change":"2024-04-27T03:17:49.017573+0000","last_active":"2024-04-27T03:21:00.950160+0000","last_peered":"2024-04-27T03:21:00.950160+0000","last_clean":"2024-04-27T03:21:00.950160+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:00.950160+0000","last_undegraded":"2024-04-27T03:21:00.950160+0000","last_fullsized":"2024-04-27T03:21:00.950160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:04.132727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813820+0000","last_change":"2024-04-27T03:17:47.003169+0000","last_active":"2024-04-27T03:19:34.813820+0000","last_peered":"2024-04-27T03:19:34.813820+0000","last_clean":"2024-04-27T03:19:34.813820+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:19:34.813820+0000","last_undegraded":"2024-04-27T03:19:34.813820+0000","last_fullsized":"2024-04-27T03:19:34.813820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822747+0000","last_change":"2024-04-27T03:17:51.020242+0000","last_active":"2024-04-27T03:19:34.822747+0000","last_peered":"2024-04-27T03:19:34.822747+0000","last_clean":"2024-04-27T03:19:34.822747+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:19:34.822747+0000","last_undegraded":"2024-04-27T03:19:34.822747+0000","last_fullsized":"2024-04-27T03:19:34.822747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'36","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.265097+0000","last_change":"2024-04-27T03:17:53.038866+0000","last_active":"2024-04-27T03:19:58.265097+0000","last_peered":"2024-04-27T03:19:58.265097+0000","last_clean":"2024-04-27T03:19:58.265097+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:19:58.265097+0000","last_undegraded":"2024-04-27T03:19:58.265097+0000","last_fullsized":"2024-04-27T03:19:58.265097+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:23.026213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'282","reported_seq":406,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.306917+0000","last_change":"2024-04-27T03:17:55.074657+0000","last_active":"2024-04-27T03:21:01.306917+0000","last_peered":"2024-04-27T03:21:01.306917+0000","last_clean":"2024-04-27T03:21:01.306917+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:01.306917+0000","last_undegraded":"2024-04-27T03:21:01.306917+0000","last_fullsized":"2024-04-27T03:21:01.306917+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'31","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246782+0000","last_change":"2024-04-27T03:18:34.790257+0000","last_active":"2024-04-27T03:21:01.246782+0000","last_peered":"2024-04-27T03:21:01.246782+0000","last_clean":"2024-04-27T03:21:01.246782+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:21:01.246782+0000","last_undegraded":"2024-04-27T03:21:01.246782+0000","last_fullsized":"2024-04-27T03:21:01.246782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:11.176662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.716875+0000","last_change":"2024-04-27T03:17:47.013403+0000","last_active":"2024-04-27T03:21:00.716875+0000","last_peered":"2024-04-27T03:21:00.716875+0000","last_clean":"2024-04-27T03:21:00.716875+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:21:00.716875+0000","last_undegraded":"2024-04-27T03:21:00.716875+0000","last_fullsized":"2024-04-27T03:21:00.716875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:47.849076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377293+0000","last_change":"2024-04-27T03:17:51.027190+0000","last_active":"2024-04-27T03:19:35.377293+0000","last_peered":"2024-04-27T03:19:35.377293+0000","last_clean":"2024-04-27T03:19:35.377293+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:19:35.377293+0000","last_undegraded":"2024-04-27T03:19:35.377293+0000","last_fullsized":"2024-04-27T03:19:35.377293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:50.708783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'47","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.221245+0000","last_change":"2024-04-27T03:17:53.060562+0000","last_active":"2024-04-27T03:20:00.221245+0000","last_peered":"2024-04-27T03:20:00.221245+0000","last_clean":"2024-04-27T03:20:00.221245+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:20:00.221245+0000","last_undegraded":"2024-04-27T03:20:00.221245+0000","last_fullsized":"2024-04-27T03:20:00.221245+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":41,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380552+0000","last_change":"2024-04-27T03:17:55.063723+0000","last_active":"2024-04-27T03:19:35.380552+0000","last_peered":"2024-04-27T03:19:35.380552+0000","last_clean":"2024-04-27T03:19:35.380552+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:19:35.380552+0000","last_undegraded":"2024-04-27T03:19:35.380552+0000","last_fullsized":"2024-04-27T03:19:35.380552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:59.616012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439014+0000","last_change":"2024-04-27T03:17:57.086011+0000","last_active":"2024-04-27T03:19:35.439014+0000","last_peered":"2024-04-27T03:19:35.439014+0000","last_clean":"2024-04-27T03:19:35.439014+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:19:35.439014+0000","last_undegraded":"2024-04-27T03:19:35.439014+0000","last_fullsized":"2024-04-27T03:19:35.439014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:11.219539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299288+0000","last_change":"2024-04-27T03:17:49.003206+0000","last_active":"2024-04-27T03:21:01.299288+0000","last_peered":"2024-04-27T03:21:01.299288+0000","last_clean":"2024-04-27T03:21:01.299288+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:01.299288+0000","last_undegraded":"2024-04-27T03:21:01.299288+0000","last_fullsized":"2024-04-27T03:21:01.299288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:42.242631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820229+0000","last_change":"2024-04-27T03:17:46.995080+0000","last_active":"2024-04-27T03:19:34.820229+0000","last_peered":"2024-04-27T03:19:34.820229+0000","last_clean":"2024-04-27T03:19:34.820229+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:19:34.820229+0000","last_undegraded":"2024-04-27T03:19:34.820229+0000","last_fullsized":"2024-04-27T03:19:34.820229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437771+0000","last_change":"2024-04-27T03:17:51.036969+0000","last_active":"2024-04-27T03:19:35.437771+0000","last_peered":"2024-04-27T03:19:35.437771+0000","last_clean":"2024-04-27T03:19:35.437771+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:19:35.437771+0000","last_undegraded":"2024-04-27T03:19:35.437771+0000","last_fullsized":"2024-04-27T03:19:35.437771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:48:00.532358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'37","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.245893+0000","last_change":"2024-04-27T03:17:53.041431+0000","last_active":"2024-04-27T03:19:58.245893+0000","last_peered":"2024-04-27T03:19:58.245893+0000","last_clean":"2024-04-27T03:19:58.245893+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:19:58.245893+0000","last_undegraded":"2024-04-27T03:19:58.245893+0000","last_fullsized":"2024-04-27T03:19:58.245893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809982+0000","last_change":"2024-04-27T03:17:55.075008+0000","last_active":"2024-04-27T03:19:34.809982+0000","last_peered":"2024-04-27T03:19:34.809982+0000","last_clean":"2024-04-27T03:19:34.809982+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:19:34.809982+0000","last_undegraded":"2024-04-27T03:19:34.809982+0000","last_fullsized":"2024-04-27T03:19:34.809982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:01.297073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"29'5","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382753+0000","last_change":"2024-04-27T03:17:57.084457+0000","last_active":"2024-04-27T03:19:35.382753+0000","last_peered":"2024-04-27T03:19:35.382753+0000","last_clean":"2024-04-27T03:19:35.382753+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:19:35.382753+0000","last_undegraded":"2024-04-27T03:19:35.382753+0000","last_fullsized":"2024-04-27T03:19:35.382753+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:34.604396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.827907+0000","last_change":"2024-04-27T03:18:34.767220+0000","last_active":"2024-04-27T03:21:00.827907+0000","last_peered":"2024-04-27T03:21:00.827907+0000","last_clean":"2024-04-27T03:21:00.827907+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:00.827907+0000","last_undegraded":"2024-04-27T03:21:00.827907+0000","last_fullsized":"2024-04-27T03:21:00.827907+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:38.334837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820841+0000","last_change":"2024-04-27T03:17:47.028376+0000","last_active":"2024-04-27T03:19:34.820841+0000","last_peered":"2024-04-27T03:19:34.820841+0000","last_clean":"2024-04-27T03:19:34.820841+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:19:34.820841+0000","last_undegraded":"2024-04-27T03:19:34.820841+0000","last_fullsized":"2024-04-27T03:19:34.820841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:18.893441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438808+0000","last_change":"2024-04-27T03:17:51.037525+0000","last_active":"2024-04-27T03:19:35.438808+0000","last_peered":"2024-04-27T03:19:35.438808+0000","last_clean":"2024-04-27T03:19:35.438808+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:19:35.438808+0000","last_undegraded":"2024-04-27T03:19:35.438808+0000","last_fullsized":"2024-04-27T03:19:35.438808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:38.895777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'48","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.284523+0000","last_change":"2024-04-27T03:18:34.800954+0000","last_active":"2024-04-27T03:19:58.284523+0000","last_peered":"2024-04-27T03:19:58.284523+0000","last_clean":"2024-04-27T03:19:58.284523+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:19:58.284523+0000","last_undegraded":"2024-04-27T03:19:58.284523+0000","last_fullsized":"2024-04-27T03:19:58.284523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:59.816977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'282","reported_seq":381,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.495106+0000","last_change":"2024-04-27T03:17:55.063480+0000","last_active":"2024-04-27T03:21:01.495106+0000","last_peered":"2024-04-27T03:21:01.495106+0000","last_clean":"2024-04-27T03:21:01.495106+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:01.495106+0000","last_undegraded":"2024-04-27T03:21:01.495106+0000","last_fullsized":"2024-04-27T03:21:01.495106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:25.995264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382973+0000","last_change":"2024-04-27T03:17:57.077044+0000","last_active":"2024-04-27T03:19:35.382973+0000","last_peered":"2024-04-27T03:19:35.382973+0000","last_clean":"2024-04-27T03:19:35.382973+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:19:35.382973+0000","last_undegraded":"2024-04-27T03:19:35.382973+0000","last_fullsized":"2024-04-27T03:19:35.382973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:03.311426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299661+0000","last_change":"2024-04-27T03:17:49.007911+0000","last_active":"2024-04-27T03:21:01.299661+0000","last_peered":"2024-04-27T03:21:01.299661+0000","last_clean":"2024-04-27T03:21:01.299661+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:01.299661+0000","last_undegraded":"2024-04-27T03:21:01.299661+0000","last_fullsized":"2024-04-27T03:21:01.299661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:07.206167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816213+0000","last_change":"2024-04-27T03:18:34.772611+0000","last_active":"2024-04-27T03:19:34.816213+0000","last_peered":"2024-04-27T03:19:34.816213+0000","last_clean":"2024-04-27T03:19:34.816213+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:19:34.816213+0000","last_undegraded":"2024-04-27T03:19:34.816213+0000","last_fullsized":"2024-04-27T03:19:34.816213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:47.500325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437638+0000","last_change":"2024-04-27T03:17:51.038976+0000","last_active":"2024-04-27T03:19:35.437638+0000","last_peered":"2024-04-27T03:19:35.437638+0000","last_clean":"2024-04-27T03:19:35.437638+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:19:35.437638+0000","last_undegraded":"2024-04-27T03:19:35.437638+0000","last_fullsized":"2024-04-27T03:19:35.437638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:17:48.170469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'66","reported_seq":138,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.287412+0000","last_change":"2024-04-27T03:17:53.050762+0000","last_active":"2024-04-27T03:19:58.287412+0000","last_peered":"2024-04-27T03:19:58.287412+0000","last_clean":"2024-04-27T03:19:58.287412+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:19:58.287412+0000","last_undegraded":"2024-04-27T03:19:58.287412+0000","last_fullsized":"2024-04-27T03:19:58.287412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":67,"num_read_kb":56,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436043+0000","last_change":"2024-04-27T03:17:55.057576+0000","last_active":"2024-04-27T03:19:35.436043+0000","last_peered":"2024-04-27T03:19:35.436043+0000","last_clean":"2024-04-27T03:19:35.436043+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:19:35.436043+0000","last_undegraded":"2024-04-27T03:19:35.436043+0000","last_fullsized":"2024-04-27T03:19:35.436043+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:41.241891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'18","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.456884+0000","last_change":"2024-04-27T03:17:57.062348+0000","last_active":"2024-04-27T03:21:01.456884+0000","last_peered":"2024-04-27T03:21:01.456884+0000","last_clean":"2024-04-27T03:21:01.456884+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:01.456884+0000","last_undegraded":"2024-04-27T03:21:01.456884+0000","last_fullsized":"2024-04-27T03:21:01.456884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:14.902153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232625+0000","last_change":"2024-04-27T03:17:49.009833+0000","last_active":"2024-04-27T03:21:01.232625+0000","last_peered":"2024-04-27T03:21:01.232625+0000","last_clean":"2024-04-27T03:21:01.232625+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:01.232625+0000","last_undegraded":"2024-04-27T03:21:01.232625+0000","last_fullsized":"2024-04-27T03:21:01.232625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:36:59.530315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437653+0000","last_change":"2024-04-27T03:18:34.767518+0000","last_active":"2024-04-27T03:19:35.437653+0000","last_peered":"2024-04-27T03:19:35.437653+0000","last_clean":"2024-04-27T03:19:35.437653+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:19:35.437653+0000","last_undegraded":"2024-04-27T03:19:35.437653+0000","last_fullsized":"2024-04-27T03:19:35.437653+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:02.579718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435824+0000","last_change":"2024-04-27T03:17:51.029195+0000","last_active":"2024-04-27T03:19:35.435824+0000","last_peered":"2024-04-27T03:19:35.435824+0000","last_clean":"2024-04-27T03:19:35.435824+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:19:35.435824+0000","last_undegraded":"2024-04-27T03:19:35.435824+0000","last_fullsized":"2024-04-27T03:19:35.435824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:42.059879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'40","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.253858+0000","last_change":"2024-04-27T03:17:53.049067+0000","last_active":"2024-04-27T03:19:58.253858+0000","last_peered":"2024-04-27T03:19:58.253858+0000","last_clean":"2024-04-27T03:19:58.253858+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:19:58.253858+0000","last_undegraded":"2024-04-27T03:19:58.253858+0000","last_fullsized":"2024-04-27T03:19:58.253858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:55:37.400674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":54,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815163+0000","last_change":"2024-04-27T03:17:55.072520+0000","last_active":"2024-04-27T03:19:34.815163+0000","last_peered":"2024-04-27T03:19:34.815163+0000","last_clean":"2024-04-27T03:19:34.815163+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:19:34.815163+0000","last_undegraded":"2024-04-27T03:19:34.815163+0000","last_fullsized":"2024-04-27T03:19:34.815163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'3","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.883363+0000","last_change":"2024-04-27T03:17:57.080943+0000","last_active":"2024-04-27T03:19:57.883363+0000","last_peered":"2024-04-27T03:19:57.883363+0000","last_clean":"2024-04-27T03:19:57.883363+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:19:57.883363+0000","last_undegraded":"2024-04-27T03:19:57.883363+0000","last_fullsized":"2024-04-27T03:19:57.883363+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822407+0000","last_change":"2024-04-27T03:17:49.016254+0000","last_active":"2024-04-27T03:19:34.822407+0000","last_peered":"2024-04-27T03:19:34.822407+0000","last_clean":"2024-04-27T03:19:34.822407+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:19:34.822407+0000","last_undegraded":"2024-04-27T03:19:34.822407+0000","last_fullsized":"2024-04-27T03:19:34.822407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817143+0000","last_change":"2024-04-27T03:17:47.026419+0000","last_active":"2024-04-27T03:19:34.817143+0000","last_peered":"2024-04-27T03:19:34.817143+0000","last_clean":"2024-04-27T03:19:34.817143+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:19:34.817143+0000","last_undegraded":"2024-04-27T03:19:34.817143+0000","last_fullsized":"2024-04-27T03:19:34.817143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:25.121818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377466+0000","last_change":"2024-04-27T03:17:51.036820+0000","last_active":"2024-04-27T03:19:35.377466+0000","last_peered":"2024-04-27T03:19:35.377466+0000","last_clean":"2024-04-27T03:19:35.377466+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:19:35.377466+0000","last_undegraded":"2024-04-27T03:19:35.377466+0000","last_fullsized":"2024-04-27T03:19:35.377466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:24.012671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.298340+0000","last_change":"2024-04-27T03:17:53.043412+0000","last_active":"2024-04-27T03:19:58.298340+0000","last_peered":"2024-04-27T03:19:58.298340+0000","last_clean":"2024-04-27T03:19:58.298340+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:19:58.298340+0000","last_undegraded":"2024-04-27T03:19:58.298340+0000","last_fullsized":"2024-04-27T03:19:58.298340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:37:03.560918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'282","reported_seq":408,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.467183+0000","last_change":"2024-04-27T03:17:55.070428+0000","last_active":"2024-04-27T03:21:01.467183+0000","last_peered":"2024-04-27T03:21:01.467183+0000","last_clean":"2024-04-27T03:21:01.467183+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:01.467183+0000","last_undegraded":"2024-04-27T03:21:01.467183+0000","last_fullsized":"2024-04-27T03:21:01.467183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:20.977192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'12","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.727848+0000","last_change":"2024-04-27T03:17:57.077926+0000","last_active":"2024-04-27T03:20:02.727848+0000","last_peered":"2024-04-27T03:20:02.727848+0000","last_clean":"2024-04-27T03:20:02.727848+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:20:02.727848+0000","last_undegraded":"2024-04-27T03:20:02.727848+0000","last_fullsized":"2024-04-27T03:20:02.727848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:14:58.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.441233+0000","last_change":"2024-04-27T03:17:49.008951+0000","last_active":"2024-04-27T03:19:35.441233+0000","last_peered":"2024-04-27T03:19:35.441233+0000","last_clean":"2024-04-27T03:19:35.441233+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:19:35.441233+0000","last_undegraded":"2024-04-27T03:19:35.441233+0000","last_fullsized":"2024-04-27T03:19:35.441233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:15.953778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810391+0000","last_change":"2024-04-27T03:17:47.036712+0000","last_active":"2024-04-27T03:19:34.810391+0000","last_peered":"2024-04-27T03:19:34.810391+0000","last_clean":"2024-04-27T03:19:34.810391+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:19:34.810391+0000","last_undegraded":"2024-04-27T03:19:34.810391+0000","last_fullsized":"2024-04-27T03:19:34.810391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'3","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724954+0000","last_change":"2024-04-27T03:17:51.037543+0000","last_active":"2024-04-27T03:19:59.724954+0000","last_peered":"2024-04-27T03:19:59.724954+0000","last_clean":"2024-04-27T03:19:59.724954+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:19:59.724954+0000","last_undegraded":"2024-04-27T03:19:59.724954+0000","last_fullsized":"2024-04-27T03:19:59.724954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":381,"num_read_kb":381,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"27'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814593+0000","last_change":"2024-04-27T03:17:53.042236+0000","last_active":"2024-04-27T03:19:34.814593+0000","last_peered":"2024-04-27T03:19:34.814593+0000","last_clean":"2024-04-27T03:19:34.814593+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:19:34.814593+0000","last_undegraded":"2024-04-27T03:19:34.814593+0000","last_fullsized":"2024-04-27T03:19:34.814593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'282","reported_seq":464,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.408184+0000","last_change":"2024-04-27T03:17:55.068988+0000","last_active":"2024-04-27T03:21:01.408184+0000","last_peered":"2024-04-27T03:21:01.408184+0000","last_clean":"2024-04-27T03:21:01.408184+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:01.408184+0000","last_undegraded":"2024-04-27T03:21:01.408184+0000","last_fullsized":"2024-04-27T03:21:01.408184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.880007+0000","last_change":"2024-04-27T03:17:57.074965+0000","last_active":"2024-04-27T03:19:59.880007+0000","last_peered":"2024-04-27T03:19:59.880007+0000","last_clean":"2024-04-27T03:19:59.880007+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:19:59.880007+0000","last_undegraded":"2024-04-27T03:19:59.880007+0000","last_fullsized":"2024-04-27T03:19:59.880007+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:30.053117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386155+0000","last_change":"2024-04-27T03:17:49.018519+0000","last_active":"2024-04-27T03:19:35.386155+0000","last_peered":"2024-04-27T03:19:35.386155+0000","last_clean":"2024-04-27T03:19:35.386155+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:19:35.386155+0000","last_undegraded":"2024-04-27T03:19:35.386155+0000","last_fullsized":"2024-04-27T03:19:35.386155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:46.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809539+0000","last_change":"2024-04-27T03:17:47.031891+0000","last_active":"2024-04-27T03:19:34.809539+0000","last_peered":"2024-04-27T03:19:34.809539+0000","last_clean":"2024-04-27T03:19:34.809539+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:19:34.809539+0000","last_undegraded":"2024-04-27T03:19:34.809539+0000","last_fullsized":"2024-04-27T03:19:34.809539+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815512+0000","last_change":"2024-04-27T03:17:51.033741+0000","last_active":"2024-04-27T03:19:34.815512+0000","last_peered":"2024-04-27T03:19:34.815512+0000","last_clean":"2024-04-27T03:19:34.815512+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:19:34.815512+0000","last_undegraded":"2024-04-27T03:19:34.815512+0000","last_fullsized":"2024-04-27T03:19:34.815512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'38","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.276705+0000","last_change":"2024-04-27T03:17:53.048795+0000","last_active":"2024-04-27T03:19:58.276705+0000","last_peered":"2024-04-27T03:19:58.276705+0000","last_clean":"2024-04-27T03:19:58.276705+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:19:58.276705+0000","last_undegraded":"2024-04-27T03:19:58.276705+0000","last_fullsized":"2024-04-27T03:19:58.276705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817470+0000","last_change":"2024-04-27T03:17:55.071142+0000","last_active":"2024-04-27T03:19:34.817470+0000","last_peered":"2024-04-27T03:19:34.817470+0000","last_clean":"2024-04-27T03:19:34.817470+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:19:34.817470+0000","last_undegraded":"2024-04-27T03:19:34.817470+0000","last_fullsized":"2024-04-27T03:19:34.817470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'8","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.368117+0000","last_change":"2024-04-27T03:17:57.071496+0000","last_active":"2024-04-27T03:21:01.368117+0000","last_peered":"2024-04-27T03:21:01.368117+0000","last_clean":"2024-04-27T03:21:01.368117+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:01.368117+0000","last_undegraded":"2024-04-27T03:21:01.368117+0000","last_fullsized":"2024-04-27T03:21:01.368117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:32:20.548838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.811995+0000","last_change":"2024-04-27T03:17:49.013855+0000","last_active":"2024-04-27T03:21:00.811995+0000","last_peered":"2024-04-27T03:21:00.811995+0000","last_clean":"2024-04-27T03:21:00.811995+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:00.811995+0000","last_undegraded":"2024-04-27T03:21:00.811995+0000","last_fullsized":"2024-04-27T03:21:00.811995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":201,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379585+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:19:35.379585+0000","last_peered":"2024-04-27T03:19:35.379585+0000","last_clean":"2024-04-27T03:19:35.379585+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:19:35.379585+0000","last_undegraded":"2024-04-27T03:19:35.379585+0000","last_fullsized":"2024-04-27T03:19:35.379585+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816708+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:19:34.816708+0000","last_peered":"2024-04-27T03:19:34.816708+0000","last_clean":"2024-04-27T03:19:34.816708+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:19:34.816708+0000","last_undegraded":"2024-04-27T03:19:34.816708+0000","last_fullsized":"2024-04-27T03:19:34.816708+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:25.777031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.100108+0000","last_change":"2024-04-27T03:17:47.039911+0000","last_active":"2024-04-27T03:21:01.100108+0000","last_peered":"2024-04-27T03:21:01.100108+0000","last_clean":"2024-04-27T03:21:01.100108+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:21:01.100108+0000","last_undegraded":"2024-04-27T03:21:01.100108+0000","last_fullsized":"2024-04-27T03:21:01.100108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382198+0000","last_change":"2024-04-27T03:17:51.036961+0000","last_active":"2024-04-27T03:19:35.382198+0000","last_peered":"2024-04-27T03:19:35.382198+0000","last_clean":"2024-04-27T03:19:35.382198+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:19:35.382198+0000","last_undegraded":"2024-04-27T03:19:35.382198+0000","last_fullsized":"2024-04-27T03:19:35.382198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:16.046086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'82","reported_seq":161,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.296556+0000","last_change":"2024-04-27T03:17:53.051108+0000","last_active":"2024-04-27T03:19:58.296556+0000","last_peered":"2024-04-27T03:19:58.296556+0000","last_clean":"2024-04-27T03:19:58.296556+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:19:58.296556+0000","last_undegraded":"2024-04-27T03:19:58.296556+0000","last_fullsized":"2024-04-27T03:19:58.296556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:32:18.585035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812224+0000","last_change":"2024-04-27T03:17:55.071697+0000","last_active":"2024-04-27T03:19:34.812224+0000","last_peered":"2024-04-27T03:19:34.812224+0000","last_clean":"2024-04-27T03:19:34.812224+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:19:34.812224+0000","last_undegraded":"2024-04-27T03:19:34.812224+0000","last_fullsized":"2024-04-27T03:19:34.812224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":167,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.324346+0000","last_change":"2024-04-27T03:17:57.075459+0000","last_active":"2024-04-27T03:21:01.324346+0000","last_peered":"2024-04-27T03:21:01.324346+0000","last_clean":"2024-04-27T03:21:01.324346+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:01.324346+0000","last_undegraded":"2024-04-27T03:21:01.324346+0000","last_fullsized":"2024-04-27T03:21:01.324346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:22.086532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233419+0000","last_change":"2024-04-27T03:17:49.007601+0000","last_active":"2024-04-27T03:21:01.233419+0000","last_peered":"2024-04-27T03:21:01.233419+0000","last_clean":"2024-04-27T03:21:01.233419+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:21:01.233419+0000","last_undegraded":"2024-04-27T03:21:01.233419+0000","last_fullsized":"2024-04-27T03:21:01.233419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815344+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:19:34.815344+0000","last_peered":"2024-04-27T03:19:34.815344+0000","last_clean":"2024-04-27T03:19:34.815344+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:19:34.815344+0000","last_undegraded":"2024-04-27T03:19:34.815344+0000","last_fullsized":"2024-04-27T03:19:34.815344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:47:23.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380638+0000","last_change":"2024-04-27T03:17:47.015528+0000","last_active":"2024-04-27T03:19:35.380638+0000","last_peered":"2024-04-27T03:19:35.380638+0000","last_clean":"2024-04-27T03:19:35.380638+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:19:35.380638+0000","last_undegraded":"2024-04-27T03:19:35.380638+0000","last_fullsized":"2024-04-27T03:19:35.380638+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822099+0000","last_change":"2024-04-27T03:17:51.034048+0000","last_active":"2024-04-27T03:19:34.822099+0000","last_peered":"2024-04-27T03:19:34.822099+0000","last_clean":"2024-04-27T03:19:34.822099+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:19:34.822099+0000","last_undegraded":"2024-04-27T03:19:34.822099+0000","last_fullsized":"2024-04-27T03:19:34.822099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'279","reported_seq":493,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.279126+0000","last_change":"2024-04-27T03:17:53.058256+0000","last_active":"2024-04-27T03:21:01.279126+0000","last_peered":"2024-04-27T03:21:01.279126+0000","last_clean":"2024-04-27T03:21:01.279126+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:01.279126+0000","last_undegraded":"2024-04-27T03:21:01.279126+0000","last_fullsized":"2024-04-27T03:21:01.279126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:08.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":366,"num_read_kb":350,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811204+0000","last_change":"2024-04-27T03:17:55.074606+0000","last_active":"2024-04-27T03:19:34.811204+0000","last_peered":"2024-04-27T03:19:34.811204+0000","last_clean":"2024-04-27T03:19:34.811204+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:19:34.811204+0000","last_undegraded":"2024-04-27T03:19:34.811204+0000","last_fullsized":"2024-04-27T03:19:34.811204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379830+0000","last_change":"2024-04-27T03:17:57.086888+0000","last_active":"2024-04-27T03:19:35.379830+0000","last_peered":"2024-04-27T03:19:35.379830+0000","last_clean":"2024-04-27T03:19:35.379830+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:19:35.379830+0000","last_undegraded":"2024-04-27T03:19:35.379830+0000","last_fullsized":"2024-04-27T03:19:35.379830+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:30.189251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232768+0000","last_change":"2024-04-27T03:17:49.014111+0000","last_active":"2024-04-27T03:21:01.232768+0000","last_peered":"2024-04-27T03:21:01.232768+0000","last_clean":"2024-04-27T03:21:01.232768+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:01.232768+0000","last_undegraded":"2024-04-27T03:21:01.232768+0000","last_fullsized":"2024-04-27T03:21:01.232768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:15.967813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436752+0000","last_change":"2024-04-27T03:17:39.633841+0000","last_active":"2024-04-27T03:19:35.436752+0000","last_peered":"2024-04-27T03:19:35.436752+0000","last_clean":"2024-04-27T03:19:35.436752+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:19:35.436752+0000","last_undegraded":"2024-04-27T03:19:35.436752+0000","last_fullsized":"2024-04-27T03:19:35.436752+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.559603+0000","last_change":"2024-04-27T03:17:47.011695+0000","last_active":"2024-04-27T03:21:00.559603+0000","last_peered":"2024-04-27T03:21:00.559603+0000","last_clean":"2024-04-27T03:21:00.559603+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:21:00.559603+0000","last_undegraded":"2024-04-27T03:21:00.559603+0000","last_fullsized":"2024-04-27T03:21:00.559603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"20'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435807+0000","last_change":"2024-04-27T03:17:51.031676+0000","last_active":"2024-04-27T03:19:35.435807+0000","last_peered":"2024-04-27T03:19:35.435807+0000","last_clean":"2024-04-27T03:19:35.435807+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:19:35.435807+0000","last_undegraded":"2024-04-27T03:19:35.435807+0000","last_fullsized":"2024-04-27T03:19:35.435807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:23.681601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'16","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.278287+0000","last_change":"2024-04-27T03:17:53.051185+0000","last_active":"2024-04-27T03:21:01.278287+0000","last_peered":"2024-04-27T03:21:01.278287+0000","last_clean":"2024-04-27T03:21:01.278287+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:01.278287+0000","last_undegraded":"2024-04-27T03:21:01.278287+0000","last_fullsized":"2024-04-27T03:21:01.278287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382273+0000","last_change":"2024-04-27T03:17:55.070180+0000","last_active":"2024-04-27T03:19:35.382273+0000","last_peered":"2024-04-27T03:19:35.382273+0000","last_clean":"2024-04-27T03:19:35.382273+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:19:35.382273+0000","last_undegraded":"2024-04-27T03:19:35.382273+0000","last_fullsized":"2024-04-27T03:19:35.382273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:15.975975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.206940+0000","last_change":"2024-04-27T03:17:57.071109+0000","last_active":"2024-04-27T03:21:01.206940+0000","last_peered":"2024-04-27T03:21:01.206940+0000","last_clean":"2024-04-27T03:21:01.206940+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:01.206940+0000","last_undegraded":"2024-04-27T03:21:01.206940+0000","last_fullsized":"2024-04-27T03:21:01.206940+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802933+0000","last_change":"2024-04-27T03:17:49.009764+0000","last_active":"2024-04-27T03:21:00.802933+0000","last_peered":"2024-04-27T03:21:00.802933+0000","last_clean":"2024-04-27T03:21:00.802933+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:00.802933+0000","last_undegraded":"2024-04-27T03:21:00.802933+0000","last_fullsized":"2024-04-27T03:21:00.802933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:31.202044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816459+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:19:34.816459+0000","last_peered":"2024-04-27T03:19:34.816459+0000","last_clean":"2024-04-27T03:19:34.816459+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:19:34.816459+0000","last_undegraded":"2024-04-27T03:19:34.816459+0000","last_fullsized":"2024-04-27T03:19:34.816459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.133613+0000","last_change":"2024-04-27T03:17:47.015018+0000","last_active":"2024-04-27T03:21:01.133613+0000","last_peered":"2024-04-27T03:21:01.133613+0000","last_clean":"2024-04-27T03:21:01.133613+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:21:01.133613+0000","last_undegraded":"2024-04-27T03:21:01.133613+0000","last_fullsized":"2024-04-27T03:21:01.133613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.378112+0000","last_change":"2024-04-27T03:17:51.029533+0000","last_active":"2024-04-27T03:19:35.378112+0000","last_peered":"2024-04-27T03:19:35.378112+0000","last_clean":"2024-04-27T03:19:35.378112+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:19:35.378112+0000","last_undegraded":"2024-04-27T03:19:35.378112+0000","last_fullsized":"2024-04-27T03:19:35.378112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:52.883679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"29'14","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.236793+0000","last_change":"2024-04-27T03:17:53.045958+0000","last_active":"2024-04-27T03:21:00.236793+0000","last_peered":"2024-04-27T03:21:00.236793+0000","last_clean":"2024-04-27T03:21:00.236793+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:00.236793+0000","last_undegraded":"2024-04-27T03:21:00.236793+0000","last_fullsized":"2024-04-27T03:21:00.236793+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815623+0000","last_change":"2024-04-27T03:17:55.078514+0000","last_active":"2024-04-27T03:19:34.815623+0000","last_peered":"2024-04-27T03:19:34.815623+0000","last_clean":"2024-04-27T03:19:34.815623+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:19:34.815623+0000","last_undegraded":"2024-04-27T03:19:34.815623+0000","last_fullsized":"2024-04-27T03:19:34.815623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.953374+0000","last_change":"2024-04-27T03:17:57.089242+0000","last_active":"2024-04-27T03:21:00.953374+0000","last_peered":"2024-04-27T03:21:00.953374+0000","last_clean":"2024-04-27T03:21:00.953374+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:21:00.953374+0000","last_undegraded":"2024-04-27T03:21:00.953374+0000","last_fullsized":"2024-04-27T03:21:00.953374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:12.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299342+0000","last_change":"2024-04-27T03:17:49.006116+0000","last_active":"2024-04-27T03:21:01.299342+0000","last_peered":"2024-04-27T03:21:01.299342+0000","last_clean":"2024-04-27T03:21:01.299342+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:01.299342+0000","last_undegraded":"2024-04-27T03:21:01.299342+0000","last_fullsized":"2024-04-27T03:21:01.299342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:05.838106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822302+0000","last_change":"2024-04-27T03:17:47.024776+0000","last_active":"2024-04-27T03:19:34.822302+0000","last_peered":"2024-04-27T03:19:34.822302+0000","last_clean":"2024-04-27T03:19:34.822302+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:19:34.822302+0000","last_undegraded":"2024-04-27T03:19:34.822302+0000","last_fullsized":"2024-04-27T03:19:34.822302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382818+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:19:35.382818+0000","last_peered":"2024-04-27T03:19:35.382818+0000","last_clean":"2024-04-27T03:19:35.382818+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:19:35.382818+0000","last_undegraded":"2024-04-27T03:19:35.382818+0000","last_fullsized":"2024-04-27T03:19:35.382818+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:01:48.896188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.733547+0000","last_change":"2024-04-27T03:17:51.037696+0000","last_active":"2024-04-27T03:19:57.733547+0000","last_peered":"2024-04-27T03:19:57.733547+0000","last_clean":"2024-04-27T03:19:57.733547+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:19:57.733547+0000","last_undegraded":"2024-04-27T03:19:57.733547+0000","last_fullsized":"2024-04-27T03:19:57.733547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:44.259718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'73","reported_seq":148,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.297267+0000","last_change":"2024-04-27T03:17:53.054938+0000","last_active":"2024-04-27T03:19:58.297267+0000","last_peered":"2024-04-27T03:19:58.297267+0000","last_clean":"2024-04-27T03:19:58.297267+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:19:58.297267+0000","last_undegraded":"2024-04-27T03:19:58.297267+0000","last_fullsized":"2024-04-27T03:19:58.297267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:38.318924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":63,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815814+0000","last_change":"2024-04-27T03:17:55.068676+0000","last_active":"2024-04-27T03:19:34.815814+0000","last_peered":"2024-04-27T03:19:34.815814+0000","last_clean":"2024-04-27T03:19:34.815814+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:19:34.815814+0000","last_undegraded":"2024-04-27T03:19:34.815814+0000","last_fullsized":"2024-04-27T03:19:34.815814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:04:55.186362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.082757+0000","last_change":"2024-04-27T03:17:57.068043+0000","last_active":"2024-04-27T03:20:00.082757+0000","last_peered":"2024-04-27T03:20:00.082757+0000","last_clean":"2024-04-27T03:20:00.082757+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:20:00.082757+0000","last_undegraded":"2024-04-27T03:20:00.082757+0000","last_fullsized":"2024-04-27T03:20:00.082757+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:14.279033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382103+0000","last_change":"2024-04-27T03:17:49.009882+0000","last_active":"2024-04-27T03:19:35.382103+0000","last_peered":"2024-04-27T03:19:35.382103+0000","last_clean":"2024-04-27T03:19:35.382103+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:19:35.382103+0000","last_undegraded":"2024-04-27T03:19:35.382103+0000","last_fullsized":"2024-04-27T03:19:35.382103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:36.878689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.286999+0000","last_change":"2024-04-27T03:17:47.030065+0000","last_active":"2024-04-27T03:21:01.286999+0000","last_peered":"2024-04-27T03:21:01.286999+0000","last_clean":"2024-04-27T03:21:01.286999+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:01.286999+0000","last_undegraded":"2024-04-27T03:21:01.286999+0000","last_fullsized":"2024-04-27T03:21:01.286999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:45:21.469900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":21,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820336+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:19:34.820336+0000","last_peered":"2024-04-27T03:19:34.820336+0000","last_clean":"2024-04-27T03:19:34.820336+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:19:34.820336+0000","last_undegraded":"2024-04-27T03:19:34.820336+0000","last_fullsized":"2024-04-27T03:19:34.820336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:48:41.511344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816118+0000","last_change":"2024-04-27T03:17:51.035808+0000","last_active":"2024-04-27T03:19:34.816118+0000","last_peered":"2024-04-27T03:19:34.816118+0000","last_clean":"2024-04-27T03:19:34.816118+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:19:34.816118+0000","last_undegraded":"2024-04-27T03:19:34.816118+0000","last_fullsized":"2024-04-27T03:19:34.816118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"27'20","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381588+0000","last_change":"2024-04-27T03:17:53.049471+0000","last_active":"2024-04-27T03:19:35.381588+0000","last_peered":"2024-04-27T03:19:35.381588+0000","last_clean":"2024-04-27T03:19:35.381588+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:19:35.381588+0000","last_undegraded":"2024-04-27T03:19:35.381588+0000","last_fullsized":"2024-04-27T03:19:35.381588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:42.365441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'282","reported_seq":361,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.481977+0000","last_change":"2024-04-27T03:17:55.077154+0000","last_active":"2024-04-27T03:21:01.481977+0000","last_peered":"2024-04-27T03:21:01.481977+0000","last_clean":"2024-04-27T03:21:01.481977+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:01.481977+0000","last_undegraded":"2024-04-27T03:21:01.481977+0000","last_fullsized":"2024-04-27T03:21:01.481977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:55:52.856782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.236855+0000","last_change":"2024-04-27T03:17:57.091210+0000","last_active":"2024-04-27T03:21:01.236855+0000","last_peered":"2024-04-27T03:21:01.236855+0000","last_clean":"2024-04-27T03:21:01.236855+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:21:01.236855+0000","last_undegraded":"2024-04-27T03:21:01.236855+0000","last_fullsized":"2024-04-27T03:21:01.236855+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:12.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948922+0000","last_change":"2024-04-27T03:17:49.007942+0000","last_active":"2024-04-27T03:21:00.948922+0000","last_peered":"2024-04-27T03:21:00.948922+0000","last_clean":"2024-04-27T03:21:00.948922+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:21:00.948922+0000","last_undegraded":"2024-04-27T03:21:00.948922+0000","last_fullsized":"2024-04-27T03:21:00.948922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815379+0000","last_change":"2024-04-27T03:17:47.039532+0000","last_active":"2024-04-27T03:19:34.815379+0000","last_peered":"2024-04-27T03:19:34.815379+0000","last_clean":"2024-04-27T03:19:34.815379+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:19:34.815379+0000","last_undegraded":"2024-04-27T03:19:34.815379+0000","last_fullsized":"2024-04-27T03:19:34.815379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384884+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:19:35.384884+0000","last_peered":"2024-04-27T03:19:35.384884+0000","last_clean":"2024-04-27T03:19:35.384884+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:19:35.384884+0000","last_undegraded":"2024-04-27T03:19:35.384884+0000","last_fullsized":"2024-04-27T03:19:35.384884+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:45.567442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815568+0000","last_change":"2024-04-27T03:17:51.020247+0000","last_active":"2024-04-27T03:19:34.815568+0000","last_peered":"2024-04-27T03:19:34.815568+0000","last_clean":"2024-04-27T03:19:34.815568+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:19:34.815568+0000","last_undegraded":"2024-04-27T03:19:34.815568+0000","last_fullsized":"2024-04-27T03:19:34.815568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812598+0000","last_change":"2024-04-27T03:17:53.048021+0000","last_active":"2024-04-27T03:19:34.812598+0000","last_peered":"2024-04-27T03:19:34.812598+0000","last_clean":"2024-04-27T03:19:34.812598+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:19:34.812598+0000","last_undegraded":"2024-04-27T03:19:34.812598+0000","last_fullsized":"2024-04-27T03:19:34.812598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:13.963398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'282","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.458427+0000","last_change":"2024-04-27T03:17:55.075368+0000","last_active":"2024-04-27T03:21:01.458427+0000","last_peered":"2024-04-27T03:21:01.458427+0000","last_clean":"2024-04-27T03:21:01.458427+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:01.458427+0000","last_undegraded":"2024-04-27T03:21:01.458427+0000","last_fullsized":"2024-04-27T03:21:01.458427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381777+0000","last_change":"2024-04-27T03:17:57.087004+0000","last_active":"2024-04-27T03:19:35.381777+0000","last_peered":"2024-04-27T03:19:35.381777+0000","last_clean":"2024-04-27T03:19:35.381777+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:19:35.381777+0000","last_undegraded":"2024-04-27T03:19:35.381777+0000","last_fullsized":"2024-04-27T03:19:35.381777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:17.983425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.193216+0000","last_change":"2024-04-27T03:17:49.004339+0000","last_active":"2024-04-27T03:20:00.193216+0000","last_peered":"2024-04-27T03:20:00.193216+0000","last_clean":"2024-04-27T03:20:00.193216+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:20:00.193216+0000","last_undegraded":"2024-04-27T03:20:00.193216+0000","last_fullsized":"2024-04-27T03:20:00.193216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:08.912344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.153066+0000","last_change":"2024-04-27T03:17:47.000780+0000","last_active":"2024-04-27T03:20:00.153066+0000","last_peered":"2024-04-27T03:20:00.153066+0000","last_clean":"2024-04-27T03:20:00.153066+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:20:00.153066+0000","last_undegraded":"2024-04-27T03:20:00.153066+0000","last_fullsized":"2024-04-27T03:20:00.153066+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819785+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:19:34.819785+0000","last_peered":"2024-04-27T03:19:34.819785+0000","last_clean":"2024-04-27T03:19:34.819785+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:19:34.819785+0000","last_undegraded":"2024-04-27T03:19:34.819785+0000","last_fullsized":"2024-04-27T03:19:34.819785+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:34.021988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810326+0000","last_change":"2024-04-27T03:17:51.033805+0000","last_active":"2024-04-27T03:19:34.810326+0000","last_peered":"2024-04-27T03:19:34.810326+0000","last_clean":"2024-04-27T03:19:34.810326+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:19:34.810326+0000","last_undegraded":"2024-04-27T03:19:34.810326+0000","last_fullsized":"2024-04-27T03:19:34.810326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'92","reported_seq":174,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.728448+0000","last_change":"2024-04-27T03:17:53.038345+0000","last_active":"2024-04-27T03:21:00.728448+0000","last_peered":"2024-04-27T03:21:00.728448+0000","last_clean":"2024-04-27T03:21:00.728448+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:00.728448+0000","last_undegraded":"2024-04-27T03:21:00.728448+0000","last_fullsized":"2024-04-27T03:21:00.728448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6026,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":86,"num_read_kb":71,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814813+0000","last_change":"2024-04-27T03:17:55.081924+0000","last_active":"2024-04-27T03:19:34.814813+0000","last_peered":"2024-04-27T03:19:34.814813+0000","last_clean":"2024-04-27T03:19:34.814813+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:19:34.814813+0000","last_undegraded":"2024-04-27T03:19:34.814813+0000","last_fullsized":"2024-04-27T03:19:34.814813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439932+0000","last_change":"2024-04-27T03:17:57.068568+0000","last_active":"2024-04-27T03:19:35.439932+0000","last_peered":"2024-04-27T03:19:35.439932+0000","last_clean":"2024-04-27T03:19:35.439932+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:19:35.439932+0000","last_undegraded":"2024-04-27T03:19:35.439932+0000","last_fullsized":"2024-04-27T03:19:35.439932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:43.785644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442918+0000","last_change":"2024-04-27T03:17:57.068516+0000","last_active":"2024-04-27T03:19:35.442918+0000","last_peered":"2024-04-27T03:19:35.442918+0000","last_clean":"2024-04-27T03:19:35.442918+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:19:35.442918+0000","last_undegraded":"2024-04-27T03:19:35.442918+0000","last_fullsized":"2024-04-27T03:19:35.442918+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:58.924125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802035+0000","last_change":"2024-04-27T03:17:49.009366+0000","last_active":"2024-04-27T03:21:00.802035+0000","last_peered":"2024-04-27T03:21:00.802035+0000","last_clean":"2024-04-27T03:21:00.802035+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:00.802035+0000","last_undegraded":"2024-04-27T03:21:00.802035+0000","last_fullsized":"2024-04-27T03:21:00.802035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:58.301334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.124874+0000","last_change":"2024-04-27T03:17:47.023352+0000","last_active":"2024-04-27T03:21:01.124874+0000","last_peered":"2024-04-27T03:21:01.124874+0000","last_clean":"2024-04-27T03:21:01.124874+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:01.124874+0000","last_undegraded":"2024-04-27T03:21:01.124874+0000","last_fullsized":"2024-04-27T03:21:01.124874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"20'1","reported_seq":222,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724858+0000","last_change":"2024-04-27T03:17:51.031849+0000","last_active":"2024-04-27T03:19:59.724858+0000","last_peered":"2024-04-27T03:19:59.724858+0000","last_clean":"2024-04-27T03:19:59.724858+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:19:59.724858+0000","last_undegraded":"2024-04-27T03:19:59.724858+0000","last_fullsized":"2024-04-27T03:19:59.724858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:05.179093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":282,"num_read_kb":282,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'38","reported_seq":91,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.283287+0000","last_change":"2024-04-27T03:17:53.041357+0000","last_active":"2024-04-27T03:19:58.283287+0000","last_peered":"2024-04-27T03:19:58.283287+0000","last_clean":"2024-04-27T03:19:58.283287+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:19:58.283287+0000","last_undegraded":"2024-04-27T03:19:58.283287+0000","last_fullsized":"2024-04-27T03:19:58.283287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:57.603435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809115+0000","last_change":"2024-04-27T03:17:55.074798+0000","last_active":"2024-04-27T03:19:34.809115+0000","last_peered":"2024-04-27T03:19:34.809115+0000","last_clean":"2024-04-27T03:19:34.809115+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:19:34.809115+0000","last_undegraded":"2024-04-27T03:19:34.809115+0000","last_fullsized":"2024-04-27T03:19:34.809115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:15.275458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.885734+0000","last_change":"2024-04-27T03:17:57.087180+0000","last_active":"2024-04-27T03:19:57.885734+0000","last_peered":"2024-04-27T03:19:57.885734+0000","last_clean":"2024-04-27T03:19:57.885734+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:19:57.885734+0000","last_undegraded":"2024-04-27T03:19:57.885734+0000","last_fullsized":"2024-04-27T03:19:57.885734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:50.266677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146716+0000","last_change":"2024-04-27T03:17:49.005188+0000","last_active":"2024-04-27T03:21:01.146716+0000","last_peered":"2024-04-27T03:21:01.146716+0000","last_clean":"2024-04-27T03:21:01.146716+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:01.146716+0000","last_undegraded":"2024-04-27T03:21:01.146716+0000","last_fullsized":"2024-04-27T03:21:01.146716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:24.812863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440214+0000","last_change":"2024-04-27T03:17:47.022474+0000","last_active":"2024-04-27T03:19:35.440214+0000","last_peered":"2024-04-27T03:19:35.440214+0000","last_clean":"2024-04-27T03:19:35.440214+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:19:35.440214+0000","last_undegraded":"2024-04-27T03:19:35.440214+0000","last_fullsized":"2024-04-27T03:19:35.440214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:13.293355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811181+0000","last_change":"2024-04-27T03:17:51.024948+0000","last_active":"2024-04-27T03:19:34.811181+0000","last_peered":"2024-04-27T03:19:34.811181+0000","last_clean":"2024-04-27T03:19:34.811181+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:19:34.811181+0000","last_undegraded":"2024-04-27T03:19:34.811181+0000","last_fullsized":"2024-04-27T03:19:34.811181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.271365+0000","last_change":"2024-04-27T03:17:53.053658+0000","last_active":"2024-04-27T03:21:00.271365+0000","last_peered":"2024-04-27T03:21:00.271365+0000","last_clean":"2024-04-27T03:21:00.271365+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:00.271365+0000","last_undegraded":"2024-04-27T03:21:00.271365+0000","last_fullsized":"2024-04-27T03:21:00.271365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:12.928546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438320+0000","last_change":"2024-04-27T03:17:55.051277+0000","last_active":"2024-04-27T03:19:35.438320+0000","last_peered":"2024-04-27T03:19:35.438320+0000","last_clean":"2024-04-27T03:19:35.438320+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:19:35.438320+0000","last_undegraded":"2024-04-27T03:19:35.438320+0000","last_fullsized":"2024-04-27T03:19:35.438320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:49.716708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382470+0000","last_change":"2024-04-27T03:17:57.089220+0000","last_active":"2024-04-27T03:19:35.382470+0000","last_peered":"2024-04-27T03:19:35.382470+0000","last_clean":"2024-04-27T03:19:35.382470+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:19:35.382470+0000","last_undegraded":"2024-04-27T03:19:35.382470+0000","last_fullsized":"2024-04-27T03:19:35.382470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:17.289812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949086+0000","last_change":"2024-04-27T03:18:34.799245+0000","last_active":"2024-04-27T03:21:00.949086+0000","last_peered":"2024-04-27T03:21:00.949086+0000","last_clean":"2024-04-27T03:21:00.949086+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:21:00.949086+0000","last_undegraded":"2024-04-27T03:21:00.949086+0000","last_fullsized":"2024-04-27T03:21:00.949086+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:17.961788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.107590+0000","last_change":"2024-04-27T03:17:47.012224+0000","last_active":"2024-04-27T03:21:01.107590+0000","last_peered":"2024-04-27T03:21:01.107590+0000","last_clean":"2024-04-27T03:21:01.107590+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:21:01.107590+0000","last_undegraded":"2024-04-27T03:21:01.107590+0000","last_fullsized":"2024-04-27T03:21:01.107590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813595+0000","last_change":"2024-04-27T03:17:51.037763+0000","last_active":"2024-04-27T03:19:34.813595+0000","last_peered":"2024-04-27T03:19:34.813595+0000","last_clean":"2024-04-27T03:19:34.813595+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:19:34.813595+0000","last_undegraded":"2024-04-27T03:19:34.813595+0000","last_fullsized":"2024-04-27T03:19:34.813595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:19.793632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'40","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.321256+0000","last_change":"2024-04-27T03:17:53.047021+0000","last_active":"2024-04-27T03:21:00.321256+0000","last_peered":"2024-04-27T03:21:00.321256+0000","last_clean":"2024-04-27T03:21:00.321256+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:00.321256+0000","last_undegraded":"2024-04-27T03:21:00.321256+0000","last_fullsized":"2024-04-27T03:21:00.321256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":61,"num_read_kb":64,"num_write":46,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379567+0000","last_change":"2024-04-27T03:17:55.075762+0000","last_active":"2024-04-27T03:19:35.379567+0000","last_peered":"2024-04-27T03:19:35.379567+0000","last_clean":"2024-04-27T03:19:35.379567+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:19:35.379567+0000","last_undegraded":"2024-04-27T03:19:35.379567+0000","last_fullsized":"2024-04-27T03:19:35.379567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:49:01.187937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812651+0000","last_change":"2024-04-27T03:17:57.086967+0000","last_active":"2024-04-27T03:19:34.812651+0000","last_peered":"2024-04-27T03:19:34.812651+0000","last_clean":"2024-04-27T03:19:34.812651+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:19:34.812651+0000","last_undegraded":"2024-04-27T03:19:34.812651+0000","last_fullsized":"2024-04-27T03:19:34.812651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:54.374770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.234017+0000","last_change":"2024-04-27T03:17:49.014835+0000","last_active":"2024-04-27T03:21:01.234017+0000","last_peered":"2024-04-27T03:21:01.234017+0000","last_clean":"2024-04-27T03:21:01.234017+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:01.234017+0000","last_undegraded":"2024-04-27T03:21:01.234017+0000","last_fullsized":"2024-04-27T03:21:01.234017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385250+0000","last_change":"2024-04-27T03:17:47.026622+0000","last_active":"2024-04-27T03:19:35.385250+0000","last_peered":"2024-04-27T03:19:35.385250+0000","last_clean":"2024-04-27T03:19:35.385250+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:19:35.385250+0000","last_undegraded":"2024-04-27T03:19:35.385250+0000","last_fullsized":"2024-04-27T03:19:35.385250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809701+0000","last_change":"2024-04-27T03:17:51.014069+0000","last_active":"2024-04-27T03:19:34.809701+0000","last_peered":"2024-04-27T03:19:34.809701+0000","last_clean":"2024-04-27T03:19:34.809701+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:19:34.809701+0000","last_undegraded":"2024-04-27T03:19:34.809701+0000","last_fullsized":"2024-04-27T03:19:34.809701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'31","reported_seq":77,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745864+0000","last_change":"2024-04-27T03:17:53.038792+0000","last_active":"2024-04-27T03:20:12.745864+0000","last_peered":"2024-04-27T03:20:12.745864+0000","last_clean":"2024-04-27T03:20:12.745864+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:20:12.745864+0000","last_undegraded":"2024-04-27T03:20:12.745864+0000","last_fullsized":"2024-04-27T03:20:12.745864+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438653+0000","last_change":"2024-04-27T03:17:55.055483+0000","last_active":"2024-04-27T03:19:35.438653+0000","last_peered":"2024-04-27T03:19:35.438653+0000","last_clean":"2024-04-27T03:19:35.438653+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:19:35.438653+0000","last_undegraded":"2024-04-27T03:19:35.438653+0000","last_fullsized":"2024-04-27T03:19:35.438653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:17.695216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'1","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742486+0000","last_change":"2024-04-27T03:17:49.022141+0000","last_active":"2024-04-27T03:19:57.742486+0000","last_peered":"2024-04-27T03:19:57.742486+0000","last_clean":"2024-04-27T03:19:57.742486+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:19:57.742486+0000","last_undegraded":"2024-04-27T03:19:57.742486+0000","last_fullsized":"2024-04-27T03:19:57.742486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.117502+0000","last_change":"2024-04-27T03:17:47.032082+0000","last_active":"2024-04-27T03:21:01.117502+0000","last_peered":"2024-04-27T03:21:01.117502+0000","last_clean":"2024-04-27T03:21:01.117502+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:21:01.117502+0000","last_undegraded":"2024-04-27T03:21:01.117502+0000","last_fullsized":"2024-04-27T03:21:01.117502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'4","reported_seq":230,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.696608+0000","last_change":"2024-04-27T03:17:51.024084+0000","last_active":"2024-04-27T03:19:59.696608+0000","last_peered":"2024-04-27T03:19:59.696608+0000","last_clean":"2024-04-27T03:19:59.696608+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:19:59.696608+0000","last_undegraded":"2024-04-27T03:19:59.696608+0000","last_fullsized":"2024-04-27T03:19:59.696608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:37.745198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":298,"num_read_kb":298,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'62","reported_seq":128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.293689+0000","last_change":"2024-04-27T03:17:53.035870+0000","last_active":"2024-04-27T03:19:58.293689+0000","last_peered":"2024-04-27T03:19:58.293689+0000","last_clean":"2024-04-27T03:19:58.293689+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:19:58.293689+0000","last_undegraded":"2024-04-27T03:19:58.293689+0000","last_fullsized":"2024-04-27T03:19:58.293689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:10.295737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":24,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439324+0000","last_change":"2024-04-27T03:17:55.075718+0000","last_active":"2024-04-27T03:19:35.439324+0000","last_peered":"2024-04-27T03:19:35.439324+0000","last_clean":"2024-04-27T03:19:35.439324+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:19:35.439324+0000","last_undegraded":"2024-04-27T03:19:35.439324+0000","last_fullsized":"2024-04-27T03:19:35.439324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:58.929938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.074643+0000","last_change":"2024-04-27T03:17:57.070510+0000","last_active":"2024-04-27T03:20:00.074643+0000","last_peered":"2024-04-27T03:20:00.074643+0000","last_clean":"2024-04-27T03:20:00.074643+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:20:00.074643+0000","last_undegraded":"2024-04-27T03:20:00.074643+0000","last_fullsized":"2024-04-27T03:20:00.074643+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111625+0000","last_change":"2024-04-27T03:17:49.000295+0000","last_active":"2024-04-27T03:19:56.111625+0000","last_peered":"2024-04-27T03:19:56.111625+0000","last_clean":"2024-04-27T03:19:56.111625+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:19:56.111625+0000","last_undegraded":"2024-04-27T03:19:56.111625+0000","last_fullsized":"2024-04-27T03:19:56.111625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379919+0000","last_change":"2024-04-27T03:17:47.021864+0000","last_active":"2024-04-27T03:19:35.379919+0000","last_peered":"2024-04-27T03:19:35.379919+0000","last_clean":"2024-04-27T03:19:35.379919+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:19:35.379919+0000","last_undegraded":"2024-04-27T03:19:35.379919+0000","last_fullsized":"2024-04-27T03:19:35.379919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:14.918204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820383+0000","last_change":"2024-04-27T03:17:51.033166+0000","last_active":"2024-04-27T03:19:34.820383+0000","last_peered":"2024-04-27T03:19:34.820383+0000","last_clean":"2024-04-27T03:19:34.820383+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:19:34.820383+0000","last_undegraded":"2024-04-27T03:19:34.820383+0000","last_fullsized":"2024-04-27T03:19:34.820383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:40.351925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.260042+0000","last_change":"2024-04-27T03:17:53.038085+0000","last_active":"2024-04-27T03:19:58.260042+0000","last_peered":"2024-04-27T03:19:58.260042+0000","last_clean":"2024-04-27T03:19:58.260042+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:19:58.260042+0000","last_undegraded":"2024-04-27T03:19:58.260042+0000","last_fullsized":"2024-04-27T03:19:58.260042+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.383103+0000","last_change":"2024-04-27T03:17:55.069433+0000","last_active":"2024-04-27T03:19:35.383103+0000","last_peered":"2024-04-27T03:19:35.383103+0000","last_clean":"2024-04-27T03:19:35.383103+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:19:35.383103+0000","last_undegraded":"2024-04-27T03:19:35.383103+0000","last_fullsized":"2024-04-27T03:19:35.383103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:17.345394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814942+0000","last_change":"2024-04-27T03:17:57.070067+0000","last_active":"2024-04-27T03:19:34.814942+0000","last_peered":"2024-04-27T03:19:34.814942+0000","last_clean":"2024-04-27T03:19:34.814942+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:19:34.814942+0000","last_undegraded":"2024-04-27T03:19:34.814942+0000","last_fullsized":"2024-04-27T03:19:34.814942+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814892+0000","last_change":"2024-04-27T03:17:49.004464+0000","last_active":"2024-04-27T03:19:34.814892+0000","last_peered":"2024-04-27T03:19:34.814892+0000","last_clean":"2024-04-27T03:19:34.814892+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:19:34.814892+0000","last_undegraded":"2024-04-27T03:19:34.814892+0000","last_fullsized":"2024-04-27T03:19:34.814892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.095190+0000","last_change":"2024-04-27T03:17:47.027156+0000","last_active":"2024-04-27T03:19:56.095190+0000","last_peered":"2024-04-27T03:19:56.095190+0000","last_clean":"2024-04-27T03:19:56.095190+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:19:56.095190+0000","last_undegraded":"2024-04-27T03:19:56.095190+0000","last_fullsized":"2024-04-27T03:19:56.095190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816982+0000","last_change":"2024-04-27T03:17:51.021222+0000","last_active":"2024-04-27T03:19:34.816982+0000","last_peered":"2024-04-27T03:19:34.816982+0000","last_clean":"2024-04-27T03:19:34.816982+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:19:34.816982+0000","last_undegraded":"2024-04-27T03:19:34.816982+0000","last_fullsized":"2024-04-27T03:19:34.816982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'39","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.872099+0000","last_change":"2024-04-27T03:17:53.041372+0000","last_active":"2024-04-27T03:19:57.872099+0000","last_peered":"2024-04-27T03:19:57.872099+0000","last_clean":"2024-04-27T03:19:57.872099+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:19:57.872099+0000","last_undegraded":"2024-04-27T03:19:57.872099+0000","last_fullsized":"2024-04-27T03:19:57.872099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384616+0000","last_change":"2024-04-27T03:17:55.077381+0000","last_active":"2024-04-27T03:19:35.384616+0000","last_peered":"2024-04-27T03:19:35.384616+0000","last_clean":"2024-04-27T03:19:35.384616+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:19:35.384616+0000","last_undegraded":"2024-04-27T03:19:35.384616+0000","last_fullsized":"2024-04-27T03:19:35.384616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:08:00.332438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.205156+0000","last_change":"2024-04-27T03:17:57.066218+0000","last_active":"2024-04-27T03:20:00.205156+0000","last_peered":"2024-04-27T03:20:00.205156+0000","last_clean":"2024-04-27T03:20:00.205156+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:20:00.205156+0000","last_undegraded":"2024-04-27T03:20:00.205156+0000","last_fullsized":"2024-04-27T03:20:00.205156+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.739365+0000","last_change":"2024-04-27T03:17:49.019967+0000","last_active":"2024-04-27T03:20:22.739365+0000","last_peered":"2024-04-27T03:20:22.739365+0000","last_clean":"2024-04-27T03:20:22.739365+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:20:22.739365+0000","last_undegraded":"2024-04-27T03:20:22.739365+0000","last_fullsized":"2024-04-27T03:20:22.739365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:45.458581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812087+0000","last_change":"2024-04-27T03:17:47.018817+0000","last_active":"2024-04-27T03:19:34.812087+0000","last_peered":"2024-04-27T03:19:34.812087+0000","last_clean":"2024-04-27T03:19:34.812087+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:19:34.812087+0000","last_undegraded":"2024-04-27T03:19:34.812087+0000","last_fullsized":"2024-04-27T03:19:34.812087+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377658+0000","last_change":"2024-04-27T03:17:51.030031+0000","last_active":"2024-04-27T03:19:35.377658+0000","last_peered":"2024-04-27T03:19:35.377658+0000","last_clean":"2024-04-27T03:19:35.377658+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:19:35.377658+0000","last_undegraded":"2024-04-27T03:19:35.377658+0000","last_fullsized":"2024-04-27T03:19:35.377658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:08.238587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814172+0000","last_change":"2024-04-27T03:17:53.046967+0000","last_active":"2024-04-27T03:19:34.814172+0000","last_peered":"2024-04-27T03:19:34.814172+0000","last_clean":"2024-04-27T03:19:34.814172+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:19:34.814172+0000","last_undegraded":"2024-04-27T03:19:34.814172+0000","last_fullsized":"2024-04-27T03:19:34.814172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'282","reported_seq":380,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277975+0000","last_change":"2024-04-27T03:17:55.060259+0000","last_active":"2024-04-27T03:21:01.277975+0000","last_peered":"2024-04-27T03:21:01.277975+0000","last_clean":"2024-04-27T03:21:01.277975+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:01.277975+0000","last_undegraded":"2024-04-27T03:21:01.277975+0000","last_fullsized":"2024-04-27T03:21:01.277975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:04:03.670215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813560+0000","last_change":"2024-04-27T03:17:57.084816+0000","last_active":"2024-04-27T03:19:34.813560+0000","last_peered":"2024-04-27T03:19:34.813560+0000","last_clean":"2024-04-27T03:19:34.813560+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:19:34.813560+0000","last_undegraded":"2024-04-27T03:19:34.813560+0000","last_fullsized":"2024-04-27T03:19:34.813560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:03.843907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248654+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.248654+0000","last_peered":"2024-04-27T03:21:01.248654+0000","last_clean":"2024-04-27T03:21:01.248654+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:01.248654+0000","last_undegraded":"2024-04-27T03:21:01.248654+0000","last_fullsized":"2024-04-27T03:21:01.248654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:22.252077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382608+0000","last_change":"2024-04-27T03:17:47.027621+0000","last_active":"2024-04-27T03:19:35.382608+0000","last_peered":"2024-04-27T03:19:35.382608+0000","last_clean":"2024-04-27T03:19:35.382608+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:19:35.382608+0000","last_undegraded":"2024-04-27T03:19:35.382608+0000","last_fullsized":"2024-04-27T03:19:35.382608+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377762+0000","last_change":"2024-04-27T03:17:51.032091+0000","last_active":"2024-04-27T03:19:35.377762+0000","last_peered":"2024-04-27T03:19:35.377762+0000","last_clean":"2024-04-27T03:19:35.377762+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:19:35.377762+0000","last_undegraded":"2024-04-27T03:19:35.377762+0000","last_fullsized":"2024-04-27T03:19:35.377762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:06.149814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'60","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.243349+0000","last_change":"2024-04-27T03:17:53.039949+0000","last_active":"2024-04-27T03:19:58.243349+0000","last_peered":"2024-04-27T03:19:58.243349+0000","last_clean":"2024-04-27T03:19:58.243349+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:19:58.243349+0000","last_undegraded":"2024-04-27T03:19:58.243349+0000","last_fullsized":"2024-04-27T03:19:58.243349+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.808839+0000","last_change":"2024-04-27T03:17:55.071558+0000","last_active":"2024-04-27T03:19:34.808839+0000","last_peered":"2024-04-27T03:19:34.808839+0000","last_clean":"2024-04-27T03:19:34.808839+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:19:34.808839+0000","last_undegraded":"2024-04-27T03:19:34.808839+0000","last_fullsized":"2024-04-27T03:19:34.808839+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:12.887800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.348234+0000","last_change":"2024-04-27T03:17:57.066613+0000","last_active":"2024-04-27T03:21:00.348234+0000","last_peered":"2024-04-27T03:21:00.348234+0000","last_clean":"2024-04-27T03:21:00.348234+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:21:00.348234+0000","last_undegraded":"2024-04-27T03:21:00.348234+0000","last_fullsized":"2024-04-27T03:21:00.348234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949296+0000","last_change":"2024-04-27T03:17:49.017718+0000","last_active":"2024-04-27T03:21:00.949296+0000","last_peered":"2024-04-27T03:21:00.949296+0000","last_clean":"2024-04-27T03:21:00.949296+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:00.949296+0000","last_undegraded":"2024-04-27T03:21:00.949296+0000","last_fullsized":"2024-04-27T03:21:00.949296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:40.378270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819810+0000","last_change":"2024-04-27T03:17:47.022941+0000","last_active":"2024-04-27T03:19:34.819810+0000","last_peered":"2024-04-27T03:19:34.819810+0000","last_clean":"2024-04-27T03:19:34.819810+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:19:34.819810+0000","last_undegraded":"2024-04-27T03:19:34.819810+0000","last_fullsized":"2024-04-27T03:19:34.819810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:09.787441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816507+0000","last_change":"2024-04-27T03:17:51.033578+0000","last_active":"2024-04-27T03:19:34.816507+0000","last_peered":"2024-04-27T03:19:34.816507+0000","last_clean":"2024-04-27T03:19:34.816507+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:19:34.816507+0000","last_undegraded":"2024-04-27T03:19:34.816507+0000","last_fullsized":"2024-04-27T03:19:34.816507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'38","reported_seq":90,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.244009+0000","last_change":"2024-04-27T03:17:53.040115+0000","last_active":"2024-04-27T03:19:58.244009+0000","last_peered":"2024-04-27T03:19:58.244009+0000","last_clean":"2024-04-27T03:19:58.244009+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:19:58.244009+0000","last_undegraded":"2024-04-27T03:19:58.244009+0000","last_fullsized":"2024-04-27T03:19:58.244009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:28.843466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815617+0000","last_change":"2024-04-27T03:17:55.058615+0000","last_active":"2024-04-27T03:19:34.815617+0000","last_peered":"2024-04-27T03:19:34.815617+0000","last_clean":"2024-04-27T03:19:34.815617+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:19:34.815617+0000","last_undegraded":"2024-04-27T03:19:34.815617+0000","last_fullsized":"2024-04-27T03:19:34.815617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.401928+0000","last_change":"2024-04-27T03:17:57.087706+0000","last_active":"2024-04-27T03:21:01.401928+0000","last_peered":"2024-04-27T03:21:01.401928+0000","last_clean":"2024-04-27T03:21:01.401928+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:21:01.401928+0000","last_undegraded":"2024-04-27T03:21:01.401928+0000","last_fullsized":"2024-04-27T03:21:01.401928+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:57.341894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950567+0000","last_change":"2024-04-27T03:17:49.011902+0000","last_active":"2024-04-27T03:21:00.950567+0000","last_peered":"2024-04-27T03:21:00.950567+0000","last_clean":"2024-04-27T03:21:00.950567+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:21:00.950567+0000","last_undegraded":"2024-04-27T03:21:00.950567+0000","last_fullsized":"2024-04-27T03:21:00.950567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.735456+0000","last_change":"2024-04-27T03:17:47.016344+0000","last_active":"2024-04-27T03:19:47.735456+0000","last_peered":"2024-04-27T03:19:47.735456+0000","last_clean":"2024-04-27T03:19:47.735456+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:19:47.735456+0000","last_undegraded":"2024-04-27T03:19:47.735456+0000","last_fullsized":"2024-04-27T03:19:47.735456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377191+0000","last_change":"2024-04-27T03:17:51.029804+0000","last_active":"2024-04-27T03:19:35.377191+0000","last_peered":"2024-04-27T03:19:35.377191+0000","last_clean":"2024-04-27T03:19:35.377191+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:19:35.377191+0000","last_undegraded":"2024-04-27T03:19:35.377191+0000","last_fullsized":"2024-04-27T03:19:35.377191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:13.111551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'32","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.286319+0000","last_change":"2024-04-27T03:17:53.041146+0000","last_active":"2024-04-27T03:19:58.286319+0000","last_peered":"2024-04-27T03:19:58.286319+0000","last_clean":"2024-04-27T03:19:58.286319+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:19:58.286319+0000","last_undegraded":"2024-04-27T03:19:58.286319+0000","last_fullsized":"2024-04-27T03:19:58.286319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:15.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816372+0000","last_change":"2024-04-27T03:17:55.071034+0000","last_active":"2024-04-27T03:19:34.816372+0000","last_peered":"2024-04-27T03:19:34.816372+0000","last_clean":"2024-04-27T03:19:34.816372+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:19:34.816372+0000","last_undegraded":"2024-04-27T03:19:34.816372+0000","last_fullsized":"2024-04-27T03:19:34.816372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.302058+0000","last_change":"2024-04-27T03:17:57.086785+0000","last_active":"2024-04-27T03:19:58.302058+0000","last_peered":"2024-04-27T03:19:58.302058+0000","last_clean":"2024-04-27T03:19:58.302058+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:19:58.302058+0000","last_undegraded":"2024-04-27T03:19:58.302058+0000","last_fullsized":"2024-04-27T03:19:58.302058+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:25:08.779674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397096+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.397096+0000","last_peered":"2024-04-27T03:21:01.397096+0000","last_clean":"2024-04-27T03:21:01.397096+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:01.397096+0000","last_undegraded":"2024-04-27T03:21:01.397096+0000","last_fullsized":"2024-04-27T03:21:01.397096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.019895+0000","last_change":"2024-04-27T03:17:47.002631+0000","last_active":"2024-04-27T03:20:00.019895+0000","last_peered":"2024-04-27T03:20:00.019895+0000","last_clean":"2024-04-27T03:20:00.019895+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:20:00.019895+0000","last_undegraded":"2024-04-27T03:20:00.019895+0000","last_fullsized":"2024-04-27T03:20:00.019895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'2","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.673843+0000","last_change":"2024-04-27T03:18:34.794992+0000","last_active":"2024-04-27T03:19:56.673843+0000","last_peered":"2024-04-27T03:19:56.673843+0000","last_clean":"2024-04-27T03:19:56.673843+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:19:56.673843+0000","last_undegraded":"2024-04-27T03:19:56.673843+0000","last_fullsized":"2024-04-27T03:19:56.673843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:56:59.713686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'44","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.246215+0000","last_change":"2024-04-27T03:17:53.039095+0000","last_active":"2024-04-27T03:21:00.246215+0000","last_peered":"2024-04-27T03:21:00.246215+0000","last_clean":"2024-04-27T03:21:00.246215+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:00.246215+0000","last_undegraded":"2024-04-27T03:21:00.246215+0000","last_fullsized":"2024-04-27T03:21:00.246215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":30,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438638+0000","last_change":"2024-04-27T03:17:55.063864+0000","last_active":"2024-04-27T03:19:35.438638+0000","last_peered":"2024-04-27T03:19:35.438638+0000","last_clean":"2024-04-27T03:19:35.438638+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:19:35.438638+0000","last_undegraded":"2024-04-27T03:19:35.438638+0000","last_fullsized":"2024-04-27T03:19:35.438638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:46.421450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.307196+0000","last_change":"2024-04-27T03:17:57.061982+0000","last_active":"2024-04-27T03:19:58.307196+0000","last_peered":"2024-04-27T03:19:58.307196+0000","last_clean":"2024-04-27T03:19:58.307196+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:19:58.307196+0000","last_undegraded":"2024-04-27T03:19:58.307196+0000","last_fullsized":"2024-04-27T03:19:58.307196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386099+0000","last_change":"2024-04-27T03:17:49.001808+0000","last_active":"2024-04-27T03:19:35.386099+0000","last_peered":"2024-04-27T03:19:35.386099+0000","last_clean":"2024-04-27T03:19:35.386099+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:19:35.386099+0000","last_undegraded":"2024-04-27T03:19:35.386099+0000","last_fullsized":"2024-04-27T03:19:35.386099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:33.807122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436337+0000","last_change":"2024-04-27T03:17:47.032632+0000","last_active":"2024-04-27T03:19:35.436337+0000","last_peered":"2024-04-27T03:19:35.436337+0000","last_clean":"2024-04-27T03:19:35.436337+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:19:35.436337+0000","last_undegraded":"2024-04-27T03:19:35.436337+0000","last_fullsized":"2024-04-27T03:19:35.436337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:53:43.154787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385434+0000","last_change":"2024-04-27T03:17:49.011651+0000","last_active":"2024-04-27T03:19:35.385434+0000","last_peered":"2024-04-27T03:19:35.385434+0000","last_clean":"2024-04-27T03:19:35.385434+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:19:35.385434+0000","last_undegraded":"2024-04-27T03:19:35.385434+0000","last_fullsized":"2024-04-27T03:19:35.385434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'1","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.732801+0000","last_change":"2024-04-27T03:18:34.773740+0000","last_active":"2024-04-27T03:19:52.732801+0000","last_peered":"2024-04-27T03:19:52.732801+0000","last_clean":"2024-04-27T03:19:52.732801+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:19:52.732801+0000","last_undegraded":"2024-04-27T03:19:52.732801+0000","last_fullsized":"2024-04-27T03:19:52.732801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:04.248651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639872,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111155+0000","last_change":"2024-04-27T03:18:34.784905+0000","last_active":"2024-04-27T03:19:56.111155+0000","last_peered":"2024-04-27T03:19:56.111155+0000","last_clean":"2024-04-27T03:19:56.111155+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:19:56.111155+0000","last_undegraded":"2024-04-27T03:19:56.111155+0000","last_fullsized":"2024-04-27T03:19:56.111155+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:28.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810481+0000","last_change":"2024-04-27T03:17:47.036148+0000","last_active":"2024-04-27T03:19:34.810481+0000","last_peered":"2024-04-27T03:19:34.810481+0000","last_clean":"2024-04-27T03:19:34.810481+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:19:34.810481+0000","last_undegraded":"2024-04-27T03:19:34.810481+0000","last_fullsized":"2024-04-27T03:19:34.810481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382422+0000","last_change":"2024-04-27T03:17:49.017582+0000","last_active":"2024-04-27T03:19:35.382422+0000","last_peered":"2024-04-27T03:19:35.382422+0000","last_clean":"2024-04-27T03:19:35.382422+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:19:35.382422+0000","last_undegraded":"2024-04-27T03:19:35.382422+0000","last_fullsized":"2024-04-27T03:19:35.382422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:20.541354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438828+0000","last_change":"2024-04-27T03:17:47.016123+0000","last_active":"2024-04-27T03:19:35.438828+0000","last_peered":"2024-04-27T03:19:35.438828+0000","last_clean":"2024-04-27T03:19:35.438828+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:19:35.438828+0000","last_undegraded":"2024-04-27T03:19:35.438828+0000","last_fullsized":"2024-04-27T03:19:35.438828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:59.109823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300171+0000","last_change":"2024-04-27T03:17:49.021184+0000","last_active":"2024-04-27T03:21:01.300171+0000","last_peered":"2024-04-27T03:21:01.300171+0000","last_clean":"2024-04-27T03:21:01.300171+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:01.300171+0000","last_undegraded":"2024-04-27T03:21:01.300171+0000","last_fullsized":"2024-04-27T03:21:01.300171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811874+0000","last_change":"2024-04-27T03:17:47.039423+0000","last_active":"2024-04-27T03:19:34.811874+0000","last_peered":"2024-04-27T03:19:34.811874+0000","last_clean":"2024-04-27T03:19:34.811874+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:19:34.811874+0000","last_undegraded":"2024-04-27T03:19:34.811874+0000","last_fullsized":"2024-04-27T03:19:34.811874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381685+0000","last_change":"2024-04-27T03:17:49.006801+0000","last_active":"2024-04-27T03:19:35.381685+0000","last_peered":"2024-04-27T03:19:35.381685+0000","last_clean":"2024-04-27T03:19:35.381685+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:19:35.381685+0000","last_undegraded":"2024-04-27T03:19:35.381685+0000","last_fullsized":"2024-04-27T03:19:35.381685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:39:13.312552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.947064+0000","last_change":"2024-04-27T03:17:47.030288+0000","last_active":"2024-04-27T03:19:59.947064+0000","last_peered":"2024-04-27T03:19:59.947064+0000","last_clean":"2024-04-27T03:19:59.947064+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:19:59.947064+0000","last_undegraded":"2024-04-27T03:19:59.947064+0000","last_fullsized":"2024-04-27T03:19:59.947064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:52:43.758948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300194+0000","last_change":"2024-04-27T03:17:49.015796+0000","last_active":"2024-04-27T03:21:01.300194+0000","last_peered":"2024-04-27T03:21:01.300194+0000","last_clean":"2024-04-27T03:21:01.300194+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:01.300194+0000","last_undegraded":"2024-04-27T03:21:01.300194+0000","last_fullsized":"2024-04-27T03:21:01.300194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.756327+0000","last_change":"2024-04-27T03:17:47.031857+0000","last_active":"2024-04-27T03:21:00.756327+0000","last_peered":"2024-04-27T03:21:00.756327+0000","last_clean":"2024-04-27T03:21:00.756327+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:00.756327+0000","last_undegraded":"2024-04-27T03:21:00.756327+0000","last_fullsized":"2024-04-27T03:21:00.756327+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":78,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949139+0000","last_change":"2024-04-27T03:17:49.017220+0000","last_active":"2024-04-27T03:21:00.949139+0000","last_peered":"2024-04-27T03:21:00.949139+0000","last_clean":"2024-04-27T03:21:00.949139+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:00.949139+0000","last_undegraded":"2024-04-27T03:21:00.949139+0000","last_fullsized":"2024-04-27T03:21:00.949139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815261+0000","last_change":"2024-04-27T03:17:47.039005+0000","last_active":"2024-04-27T03:19:34.815261+0000","last_peered":"2024-04-27T03:19:34.815261+0000","last_clean":"2024-04-27T03:19:34.815261+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:19:34.815261+0000","last_undegraded":"2024-04-27T03:19:34.815261+0000","last_fullsized":"2024-04-27T03:19:34.815261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299434+0000","last_change":"2024-04-27T03:17:49.011708+0000","last_active":"2024-04-27T03:21:01.299434+0000","last_peered":"2024-04-27T03:21:01.299434+0000","last_clean":"2024-04-27T03:21:01.299434+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:01.299434+0000","last_undegraded":"2024-04-27T03:21:01.299434+0000","last_fullsized":"2024-04-27T03:21:01.299434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:52.139159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809399+0000","last_change":"2024-04-27T03:17:47.016024+0000","last_active":"2024-04-27T03:19:34.809399+0000","last_peered":"2024-04-27T03:19:34.809399+0000","last_clean":"2024-04-27T03:19:34.809399+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:19:34.809399+0000","last_undegraded":"2024-04-27T03:19:34.809399+0000","last_fullsized":"2024-04-27T03:19:34.809399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300134+0000","last_change":"2024-04-27T03:17:49.010865+0000","last_active":"2024-04-27T03:21:01.300134+0000","last_peered":"2024-04-27T03:21:01.300134+0000","last_clean":"2024-04-27T03:21:01.300134+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:21:01.300134+0000","last_undegraded":"2024-04-27T03:21:01.300134+0000","last_fullsized":"2024-04-27T03:21:01.300134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810923+0000","last_change":"2024-04-27T03:17:47.019986+0000","last_active":"2024-04-27T03:19:34.810923+0000","last_peered":"2024-04-27T03:19:34.810923+0000","last_clean":"2024-04-27T03:19:34.810923+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:19:34.810923+0000","last_undegraded":"2024-04-27T03:19:34.810923+0000","last_fullsized":"2024-04-27T03:19:34.810923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949386+0000","last_change":"2024-04-27T03:17:49.005667+0000","last_active":"2024-04-27T03:21:00.949386+0000","last_peered":"2024-04-27T03:21:00.949386+0000","last_clean":"2024-04-27T03:21:00.949386+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:00.949386+0000","last_undegraded":"2024-04-27T03:21:00.949386+0000","last_fullsized":"2024-04-27T03:21:00.949386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:07.432944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817074+0000","last_change":"2024-04-27T03:17:47.023505+0000","last_active":"2024-04-27T03:19:34.817074+0000","last_peered":"2024-04-27T03:19:34.817074+0000","last_clean":"2024-04-27T03:19:34.817074+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:19:34.817074+0000","last_undegraded":"2024-04-27T03:19:34.817074+0000","last_fullsized":"2024-04-27T03:19:34.817074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382540+0000","last_change":"2024-04-27T03:17:49.009462+0000","last_active":"2024-04-27T03:19:35.382540+0000","last_peered":"2024-04-27T03:19:35.382540+0000","last_clean":"2024-04-27T03:19:35.382540+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:19:35.382540+0000","last_undegraded":"2024-04-27T03:19:35.382540+0000","last_fullsized":"2024-04-27T03:19:35.382540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:32.201166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382267+0000","last_change":"2024-04-27T03:17:47.015754+0000","last_active":"2024-04-27T03:19:35.382267+0000","last_peered":"2024-04-27T03:19:35.382267+0000","last_clean":"2024-04-27T03:19:35.382267+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:19:35.382267+0000","last_undegraded":"2024-04-27T03:19:35.382267+0000","last_fullsized":"2024-04-27T03:19:35.382267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'7","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744638+0000","last_change":"2024-04-27T03:17:49.017194+0000","last_active":"2024-04-27T03:20:07.744638+0000","last_peered":"2024-04-27T03:20:07.744638+0000","last_clean":"2024-04-27T03:20:07.744638+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:20:07.744638+0000","last_undegraded":"2024-04-27T03:20:07.744638+0000","last_fullsized":"2024-04-27T03:20:07.744638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812147+0000","last_change":"2024-04-27T03:17:47.038806+0000","last_active":"2024-04-27T03:19:34.812147+0000","last_peered":"2024-04-27T03:19:34.812147+0000","last_clean":"2024-04-27T03:19:34.812147+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:19:34.812147+0000","last_undegraded":"2024-04-27T03:19:34.812147+0000","last_fullsized":"2024-04-27T03:19:34.812147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.808067+0000","last_change":"2024-04-27T03:17:49.006535+0000","last_active":"2024-04-27T03:21:00.808067+0000","last_peered":"2024-04-27T03:21:00.808067+0000","last_clean":"2024-04-27T03:21:00.808067+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:00.808067+0000","last_undegraded":"2024-04-27T03:21:00.808067+0000","last_fullsized":"2024-04-27T03:21:00.808067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:21.664514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442247+0000","last_change":"2024-04-27T03:19:35.441020+0000","last_active":"2024-04-27T03:19:35.442247+0000","last_peered":"2024-04-27T03:19:35.442247+0000","last_clean":"2024-04-27T03:19:35.442247+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:19:35.442247+0000","last_undegraded":"2024-04-27T03:19:35.442247+0000","last_fullsized":"2024-04-27T03:19:35.442247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:39:26.708815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814170+0000","last_change":"2024-04-27T03:17:49.016545+0000","last_active":"2024-04-27T03:19:34.814170+0000","last_peered":"2024-04-27T03:19:34.814170+0000","last_clean":"2024-04-27T03:19:34.814170+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:19:34.814170+0000","last_undegraded":"2024-04-27T03:19:34.814170+0000","last_fullsized":"2024-04-27T03:19:34.814170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:52.886758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442831+0000","last_change":"2024-04-27T03:17:47.022441+0000","last_active":"2024-04-27T03:19:35.442831+0000","last_peered":"2024-04-27T03:19:35.442831+0000","last_clean":"2024-04-27T03:19:35.442831+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:19:35.442831+0000","last_undegraded":"2024-04-27T03:19:35.442831+0000","last_fullsized":"2024-04-27T03:19:35.442831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146608+0000","last_change":"2024-04-27T03:17:49.018017+0000","last_active":"2024-04-27T03:21:01.146608+0000","last_peered":"2024-04-27T03:21:01.146608+0000","last_clean":"2024-04-27T03:21:01.146608+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:01.146608+0000","last_undegraded":"2024-04-27T03:21:01.146608+0000","last_fullsized":"2024-04-27T03:21:01.146608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815679+0000","last_change":"2024-04-27T03:17:47.026063+0000","last_active":"2024-04-27T03:19:34.815679+0000","last_peered":"2024-04-27T03:19:34.815679+0000","last_clean":"2024-04-27T03:19:34.815679+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:19:34.815679+0000","last_undegraded":"2024-04-27T03:19:34.815679+0000","last_fullsized":"2024-04-27T03:19:34.815679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814107+0000","last_change":"2024-04-27T03:17:47.002843+0000","last_active":"2024-04-27T03:19:34.814107+0000","last_peered":"2024-04-27T03:19:34.814107+0000","last_clean":"2024-04-27T03:19:34.814107+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:19:34.814107+0000","last_undegraded":"2024-04-27T03:19:34.814107+0000","last_fullsized":"2024-04-27T03:19:34.814107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'11","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164859+0000","last_change":"2024-04-27T03:17:49.014133+0000","last_active":"2024-04-27T03:21:01.164859+0000","last_peered":"2024-04-27T03:21:01.164859+0000","last_clean":"2024-04-27T03:21:01.164859+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:01.164859+0000","last_undegraded":"2024-04-27T03:21:01.164859+0000","last_fullsized":"2024-04-27T03:21:01.164859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:38.383961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379782+0000","last_change":"2024-04-27T03:17:47.022513+0000","last_active":"2024-04-27T03:19:35.379782+0000","last_peered":"2024-04-27T03:19:35.379782+0000","last_clean":"2024-04-27T03:19:35.379782+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:19:35.379782+0000","last_undegraded":"2024-04-27T03:19:35.379782+0000","last_fullsized":"2024-04-27T03:19:35.379782+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:24.322205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385155+0000","last_change":"2024-04-27T03:17:49.012955+0000","last_active":"2024-04-27T03:19:35.385155+0000","last_peered":"2024-04-27T03:19:35.385155+0000","last_clean":"2024-04-27T03:19:35.385155+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:19:35.385155+0000","last_undegraded":"2024-04-27T03:19:35.385155+0000","last_fullsized":"2024-04-27T03:19:35.385155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2256,"ondisk_log_size":2256,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":13384,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1690,"num_read_kb":1444,"num_write":723,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":10824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1470,"ondisk_log_size":1470,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":961,"num_read_kb":961,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":473,"ondisk_log_size":473,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":25805696,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":309,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38719488,"data_stored":38719488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46056,"kb_used_data":6560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94325784,"statfs":{"total":96636764160,"available":96589602816,"internally_reserved":0,"allocated":6717440,"data_stored":6440144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57224,"kb_used_data":13568,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94314616,"statfs":{"total":96636764160,"available":96578166784,"internally_reserved":0,"allocated":13893632,"data_stored":13644798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":9524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":9752576,"data_stored":9451527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50504,"kb_used_data":11008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94321336,"statfs":{"total":96636764160,"available":96585048064,"internally_reserved":0,"allocated":11272192,"data_stored":10958989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10809344,"data_stored":10809344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8712192,"data_stored":8712192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":12906496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":4422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:21:09.213 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-27T03:21:09.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:21:09.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.354+0000 7f2549296640 1 Processor -- start 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.354+0000 7f2549296640 1 -- start start 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.354+0000 7f2549296640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f254412f2a0 0x7f2544131690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.354+0000 7f2549296640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f254412bfd0 con 0x7f254412c400 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.354+0000 7f2549296640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f254412cef0 con 0x7f254412f2a0 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f25427fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f254412f2a0 0x7f2544131690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2542ffd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f254412c400 0x7f254412c7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46706/0 (socket says 172.21.15.77:46706) 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f25427fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f254412f2a0 0x7f2544131690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56782/0 (socket says 172.21.15.77:56782) 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2542ffd640 1 -- 172.21.15.77:0/3723426630 learned_addr learned my addr 172.21.15.77:0/3723426630 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f25427fc640 1 -- 172.21.15.77:0/3723426630 >> v1:172.21.15.77:6789/0 conn(0x7f254412c400 legacy=0x7f254412c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f25427fc640 1 -- 172.21.15.77:0/3723426630 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254409ac80 con 0x7f254412f2a0 2024-04-27T03:21:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f25427fc640 1 --2- 172.21.15.77:0/3723426630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412f2a0 0x7f2544131690 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2538009740 tx=0x7f253801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b76401dd334cf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2541ffb640 1 -- 172.21.15.77:0/3723426630 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25380b0070 con 0x7f254412f2a0 2024-04-27T03:21:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2541ffb640 1 -- 172.21.15.77:0/3723426630 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25380ab530 con 0x7f254412f2a0 2024-04-27T03:21:09.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2541ffb640 1 -- 172.21.15.77:0/3723426630 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25380ab830 con 0x7f254412f2a0 2024-04-27T03:21:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2549296640 1 -- 172.21.15.77:0/3723426630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412f2a0 msgr2=0x7f2544131690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2549296640 1 --2- 172.21.15.77:0/3723426630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412f2a0 0x7f2544131690 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2538009740 tx=0x7f253801c370 comp rx=0 tx=0).stop 2024-04-27T03:21:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2549296640 1 -- 172.21.15.77:0/3723426630 shutdown_connections 2024-04-27T03:21:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2549296640 1 --2- 172.21.15.77:0/3723426630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412f2a0 0x7f2544131690 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.371 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2549296640 1 -- 172.21.15.77:0/3723426630 >> 172.21.15.77:0/3723426630 conn(0x7f254409c320 msgr2=0x7f254409c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:09.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2549296640 1 -- 172.21.15.77:0/3723426630 shutdown_connections 2024-04-27T03:21:09.372 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.358+0000 7f2549296640 1 -- 172.21.15.77:0/3723426630 wait complete. 2024-04-27T03:21:09.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2549296640 1 Processor -- start 2024-04-27T03:21:09.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2549296640 1 -- start start 2024-04-27T03:21:09.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2549296640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412c400 0x7f25441277d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2549296640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f2544134780 con 0x7f254412c400 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2542ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412c400 0x7f25441277d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2542ffd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412c400 0x7f25441277d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56792/0 (socket says 172.21.15.77:56792) 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2542ffd640 1 -- 172.21.15.77:0/2927992821 learned_addr learned my addr 172.21.15.77:0/2927992821 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2542ffd640 1 -- 172.21.15.77:0/2927992821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2544129180 con 0x7f254412c400 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2542ffd640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412c400 0x7f25441277d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f253400c480 tx=0x7f253400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f252b7fe640 1 -- 172.21.15.77:0/2927992821 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2534017070 con 0x7f254412c400 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f252b7fe640 1 -- 172.21.15.77:0/2927992821 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2534005c20 con 0x7f254412c400 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f252b7fe640 1 -- 172.21.15.77:0/2927992821 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2534007070 con 0x7f254412c400 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25441150c0 con 0x7f254412c400 2024-04-27T03:21:09.374 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2544114820 con 0x7f254412c400 2024-04-27T03:21:09.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f252b7fe640 1 -- 172.21.15.77:0/2927992821 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f253401a070 con 0x7f254412c400 2024-04-27T03:21:09.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2544062e50 con 0x7f254412c400 2024-04-27T03:21:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f252b7fe640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2530035db0 0x7f2530038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f252b7fe640 1 -- 172.21.15.77:0/2927992821 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f2534048510 con 0x7f254412c400 2024-04-27T03:21:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.362+0000 7f25427fc640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2530035db0 0x7f2530038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:09.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.366+0000 7f25427fc640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2530035db0 0x7f2530038270 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f25380ad000 tx=0x7f25380a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:09.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.370+0000 7f252b7fe640 1 -- 172.21.15.77:0/2927992821 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2534007210 con 0x7f254412c400 2024-04-27T03:21:09.538 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.526+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2544114e50 con 0x7f2530035db0 2024-04-27T03:21:09.583 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.570+0000 7f252b7fe640 1 -- 172.21.15.77:0/2927992821 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+665756 (secure 0 0 0) 0x7f2544114e50 con 0x7f2530035db0 2024-04-27T03:21:09.589 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:21:09.595 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2530035db0 msgr2=0x7f2530038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2530035db0 0x7f2530038270 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f25380ad000 tx=0x7f25380a6000 comp rx=0 tx=0).stop 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412c400 msgr2=0x7f25441277d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412c400 0x7f25441277d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f253400c480 tx=0x7f253400c940 comp rx=0 tx=0).stop 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 shutdown_connections 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f2530035db0 0x7f2530038270 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 --2- 172.21.15.77:0/2927992821 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f254412c400 0x7f25441277d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 >> 172.21.15.77:0/2927992821 conn(0x7f254409c320 msgr2=0x7f2544098a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 shutdown_connections 2024-04-27T03:21:09.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.578+0000 7f2549296640 1 -- 172.21.15.77:0/2927992821 wait complete. 2024-04-27T03:21:09.606 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-04-27T03:21:07.732296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26412789,"num_objects":259,"num_object_clones":0,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":5177,"num_read_kb":4722,"num_write":2298,"num_write_kb":61373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4672,"ondisk_log_size":4672,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":190324,"kb_used_data":40660,"kb_used_omap":6,"kb_used_meta":149625,"kb_avail":377297036,"statfs":{"total":386547056640,"available":386352164864,"internally_reserved":0,"allocated":41635840,"data_stored":40495458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":153216790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25811687,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":510,"num_read_kb":465,"num_write":386,"num_write_kb":29372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005707"},"pg_stats":[{"pgid":"3.3f","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741206+0000","last_change":"2024-04-27T03:17:47.025622+0000","last_active":"2024-04-27T03:19:52.741206+0000","last_peered":"2024-04-27T03:19:52.741206+0000","last_clean":"2024-04-27T03:19:52.741206+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:19:52.741206+0000","last_undegraded":"2024-04-27T03:19:52.741206+0000","last_fullsized":"2024-04-27T03:19:52.741206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:56.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'5","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246474+0000","last_change":"2024-04-27T03:17:49.019025+0000","last_active":"2024-04-27T03:21:01.246474+0000","last_peered":"2024-04-27T03:21:01.246474+0000","last_clean":"2024-04-27T03:21:01.246474+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:01.246474+0000","last_undegraded":"2024-04-27T03:21:01.246474+0000","last_fullsized":"2024-04-27T03:21:01.246474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385062+0000","last_change":"2024-04-27T03:17:47.023458+0000","last_active":"2024-04-27T03:19:35.385062+0000","last_peered":"2024-04-27T03:19:35.385062+0000","last_clean":"2024-04-27T03:19:35.385062+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:19:35.385062+0000","last_undegraded":"2024-04-27T03:19:35.385062+0000","last_fullsized":"2024-04-27T03:19:35.385062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810083+0000","last_change":"2024-04-27T03:17:49.020940+0000","last_active":"2024-04-27T03:19:34.810083+0000","last_peered":"2024-04-27T03:19:34.810083+0000","last_clean":"2024-04-27T03:19:34.810083+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:19:34.810083+0000","last_undegraded":"2024-04-27T03:19:34.810083+0000","last_fullsized":"2024-04-27T03:19:34.810083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811946+0000","last_change":"2024-04-27T03:17:47.039594+0000","last_active":"2024-04-27T03:19:34.811946+0000","last_peered":"2024-04-27T03:19:34.811946+0000","last_clean":"2024-04-27T03:19:34.811946+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:19:34.811946+0000","last_undegraded":"2024-04-27T03:19:34.811946+0000","last_fullsized":"2024-04-27T03:19:34.811946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111615+0000","last_change":"2024-04-27T03:17:49.016480+0000","last_active":"2024-04-27T03:19:56.111615+0000","last_peered":"2024-04-27T03:19:56.111615+0000","last_clean":"2024-04-27T03:19:56.111615+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:19:56.111615+0000","last_undegraded":"2024-04-27T03:19:56.111615+0000","last_fullsized":"2024-04-27T03:19:56.111615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439532+0000","last_change":"2024-04-27T03:17:47.021409+0000","last_active":"2024-04-27T03:19:35.439532+0000","last_peered":"2024-04-27T03:19:35.439532+0000","last_clean":"2024-04-27T03:19:35.439532+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:19:35.439532+0000","last_undegraded":"2024-04-27T03:19:35.439532+0000","last_fullsized":"2024-04-27T03:19:35.439532+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:24.826641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948899+0000","last_change":"2024-04-27T03:17:49.021774+0000","last_active":"2024-04-27T03:21:00.948899+0000","last_peered":"2024-04-27T03:21:00.948899+0000","last_clean":"2024-04-27T03:21:00.948899+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:00.948899+0000","last_undegraded":"2024-04-27T03:21:00.948899+0000","last_fullsized":"2024-04-27T03:21:00.948899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817333+0000","last_change":"2024-04-27T03:17:47.015865+0000","last_active":"2024-04-27T03:19:34.817333+0000","last_peered":"2024-04-27T03:19:34.817333+0000","last_clean":"2024-04-27T03:19:34.817333+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:19:34.817333+0000","last_undegraded":"2024-04-27T03:19:34.817333+0000","last_fullsized":"2024-04-27T03:19:34.817333+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439003+0000","last_change":"2024-04-27T03:18:34.776780+0000","last_active":"2024-04-27T03:19:35.439003+0000","last_peered":"2024-04-27T03:19:35.439003+0000","last_clean":"2024-04-27T03:19:35.439003+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:19:35.439003+0000","last_undegraded":"2024-04-27T03:19:35.439003+0000","last_fullsized":"2024-04-27T03:19:35.439003+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:30.737857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.111562+0000","last_change":"2024-04-27T03:17:47.026206+0000","last_active":"2024-04-27T03:20:00.111562+0000","last_peered":"2024-04-27T03:20:00.111562+0000","last_clean":"2024-04-27T03:20:00.111562+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:20:00.111562+0000","last_undegraded":"2024-04-27T03:20:00.111562+0000","last_fullsized":"2024-04-27T03:20:00.111562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'11","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742400+0000","last_change":"2024-04-27T03:17:49.015210+0000","last_active":"2024-04-27T03:19:57.742400+0000","last_peered":"2024-04-27T03:19:57.742400+0000","last_clean":"2024-04-27T03:19:57.742400+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:19:57.742400+0000","last_undegraded":"2024-04-27T03:19:57.742400+0000","last_fullsized":"2024-04-27T03:19:57.742400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811270+0000","last_change":"2024-04-27T03:17:47.036460+0000","last_active":"2024-04-27T03:19:34.811270+0000","last_peered":"2024-04-27T03:19:34.811270+0000","last_clean":"2024-04-27T03:19:34.811270+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:19:34.811270+0000","last_undegraded":"2024-04-27T03:19:34.811270+0000","last_fullsized":"2024-04-27T03:19:34.811270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:56.284851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298837+0000","last_change":"2024-04-27T03:17:49.024190+0000","last_active":"2024-04-27T03:21:01.298837+0000","last_peered":"2024-04-27T03:21:01.298837+0000","last_clean":"2024-04-27T03:21:01.298837+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:01.298837+0000","last_undegraded":"2024-04-27T03:21:01.298837+0000","last_fullsized":"2024-04-27T03:21:01.298837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810250+0000","last_change":"2024-04-27T03:17:47.013877+0000","last_active":"2024-04-27T03:19:34.810250+0000","last_peered":"2024-04-27T03:19:34.810250+0000","last_clean":"2024-04-27T03:19:34.810250+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:19:34.810250+0000","last_undegraded":"2024-04-27T03:19:34.810250+0000","last_fullsized":"2024-04-27T03:19:34.810250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.444094+0000","last_change":"2024-04-27T03:17:49.020265+0000","last_active":"2024-04-27T03:19:35.444094+0000","last_peered":"2024-04-27T03:19:35.444094+0000","last_clean":"2024-04-27T03:19:35.444094+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:19:35.444094+0000","last_undegraded":"2024-04-27T03:19:35.444094+0000","last_fullsized":"2024-04-27T03:19:35.444094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:23.713924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298707+0000","last_change":"2024-04-27T03:17:49.019484+0000","last_active":"2024-04-27T03:21:01.298707+0000","last_peered":"2024-04-27T03:21:01.298707+0000","last_clean":"2024-04-27T03:21:01.298707+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:01.298707+0000","last_undegraded":"2024-04-27T03:21:01.298707+0000","last_fullsized":"2024-04-27T03:21:01.298707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385569+0000","last_change":"2024-04-27T03:17:47.027209+0000","last_active":"2024-04-27T03:19:35.385569+0000","last_peered":"2024-04-27T03:19:35.385569+0000","last_clean":"2024-04-27T03:19:35.385569+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:19:35.385569+0000","last_undegraded":"2024-04-27T03:19:35.385569+0000","last_fullsized":"2024-04-27T03:19:35.385569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814961+0000","last_change":"2024-04-27T03:17:49.013701+0000","last_active":"2024-04-27T03:19:34.814961+0000","last_peered":"2024-04-27T03:19:34.814961+0000","last_clean":"2024-04-27T03:19:34.814961+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:19:34.814961+0000","last_undegraded":"2024-04-27T03:19:34.814961+0000","last_fullsized":"2024-04-27T03:19:34.814961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.443009+0000","last_change":"2024-04-27T03:17:47.015536+0000","last_active":"2024-04-27T03:19:35.443009+0000","last_peered":"2024-04-27T03:19:35.443009+0000","last_clean":"2024-04-27T03:19:35.443009+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:19:35.443009+0000","last_undegraded":"2024-04-27T03:19:35.443009+0000","last_fullsized":"2024-04-27T03:19:35.443009+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:51:44.696219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.809825+0000","last_change":"2024-04-27T03:17:49.011747+0000","last_active":"2024-04-27T03:21:00.809825+0000","last_peered":"2024-04-27T03:21:00.809825+0000","last_clean":"2024-04-27T03:21:00.809825+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:21:00.809825+0000","last_undegraded":"2024-04-27T03:21:00.809825+0000","last_fullsized":"2024-04-27T03:21:00.809825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820563+0000","last_change":"2024-04-27T03:17:47.023848+0000","last_active":"2024-04-27T03:19:34.820563+0000","last_peered":"2024-04-27T03:19:34.820563+0000","last_clean":"2024-04-27T03:19:34.820563+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:19:34.820563+0000","last_undegraded":"2024-04-27T03:19:34.820563+0000","last_fullsized":"2024-04-27T03:19:34.820563+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.231670+0000","last_change":"2024-04-27T03:17:49.019463+0000","last_active":"2024-04-27T03:21:01.231670+0000","last_peered":"2024-04-27T03:21:01.231670+0000","last_clean":"2024-04-27T03:21:01.231670+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:01.231670+0000","last_undegraded":"2024-04-27T03:21:01.231670+0000","last_fullsized":"2024-04-27T03:21:01.231670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:53.154789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816701+0000","last_change":"2024-04-27T03:17:47.012402+0000","last_active":"2024-04-27T03:19:34.816701+0000","last_peered":"2024-04-27T03:19:34.816701+0000","last_clean":"2024-04-27T03:19:34.816701+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:19:34.816701+0000","last_undegraded":"2024-04-27T03:19:34.816701+0000","last_fullsized":"2024-04-27T03:19:34.816701+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146748+0000","last_change":"2024-04-27T03:17:49.008483+0000","last_active":"2024-04-27T03:21:01.146748+0000","last_peered":"2024-04-27T03:21:01.146748+0000","last_clean":"2024-04-27T03:21:01.146748+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:01.146748+0000","last_undegraded":"2024-04-27T03:21:01.146748+0000","last_fullsized":"2024-04-27T03:21:01.146748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:11.803924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812204+0000","last_change":"2024-04-27T03:17:47.036082+0000","last_active":"2024-04-27T03:19:34.812204+0000","last_peered":"2024-04-27T03:19:34.812204+0000","last_clean":"2024-04-27T03:19:34.812204+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:19:34.812204+0000","last_undegraded":"2024-04-27T03:19:34.812204+0000","last_fullsized":"2024-04-27T03:19:34.812204+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164153+0000","last_change":"2024-04-27T03:17:49.005822+0000","last_active":"2024-04-27T03:21:01.164153+0000","last_peered":"2024-04-27T03:21:01.164153+0000","last_clean":"2024-04-27T03:21:01.164153+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:01.164153+0000","last_undegraded":"2024-04-27T03:21:01.164153+0000","last_fullsized":"2024-04-27T03:21:01.164153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:40.042792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811583+0000","last_change":"2024-04-27T03:17:47.033466+0000","last_active":"2024-04-27T03:19:34.811583+0000","last_peered":"2024-04-27T03:19:34.811583+0000","last_clean":"2024-04-27T03:19:34.811583+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:19:34.811583+0000","last_undegraded":"2024-04-27T03:19:34.811583+0000","last_fullsized":"2024-04-27T03:19:34.811583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'37","reported_seq":105,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.190758+0000","last_change":"2024-04-27T03:17:57.088698+0000","last_active":"2024-04-27T03:21:01.190758+0000","last_peered":"2024-04-27T03:21:01.190758+0000","last_clean":"2024-04-27T03:21:01.190758+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:21:01.190758+0000","last_undegraded":"2024-04-27T03:21:01.190758+0000","last_fullsized":"2024-04-27T03:21:01.190758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:57.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233462+0000","last_change":"2024-04-27T03:17:49.015482+0000","last_active":"2024-04-27T03:21:01.233462+0000","last_peered":"2024-04-27T03:21:01.233462+0000","last_clean":"2024-04-27T03:21:01.233462+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:01.233462+0000","last_undegraded":"2024-04-27T03:21:01.233462+0000","last_fullsized":"2024-04-27T03:21:01.233462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811777+0000","last_change":"2024-04-27T03:17:47.018824+0000","last_active":"2024-04-27T03:19:34.811777+0000","last_peered":"2024-04-27T03:19:34.811777+0000","last_clean":"2024-04-27T03:19:34.811777+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:19:34.811777+0000","last_undegraded":"2024-04-27T03:19:34.811777+0000","last_fullsized":"2024-04-27T03:19:34.811777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815532+0000","last_change":"2024-04-27T03:17:51.032891+0000","last_active":"2024-04-27T03:19:34.815532+0000","last_peered":"2024-04-27T03:19:34.815532+0000","last_clean":"2024-04-27T03:19:34.815532+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:19:34.815532+0000","last_undegraded":"2024-04-27T03:19:34.815532+0000","last_fullsized":"2024-04-27T03:19:34.815532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'43","reported_seq":101,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.274412+0000","last_change":"2024-04-27T03:17:53.040688+0000","last_active":"2024-04-27T03:19:58.274412+0000","last_peered":"2024-04-27T03:19:58.274412+0000","last_clean":"2024-04-27T03:19:58.274412+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:19:58.274412+0000","last_undegraded":"2024-04-27T03:19:58.274412+0000","last_fullsized":"2024-04-27T03:19:58.274412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819799+0000","last_change":"2024-04-27T03:17:55.072849+0000","last_active":"2024-04-27T03:19:34.819799+0000","last_peered":"2024-04-27T03:19:34.819799+0000","last_clean":"2024-04-27T03:19:34.819799+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:19:34.819799+0000","last_undegraded":"2024-04-27T03:19:34.819799+0000","last_fullsized":"2024-04-27T03:19:34.819799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.848140+0000","last_change":"2024-04-27T03:17:57.061550+0000","last_active":"2024-04-27T03:21:00.848140+0000","last_peered":"2024-04-27T03:21:00.848140+0000","last_clean":"2024-04-27T03:21:00.848140+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:00.848140+0000","last_undegraded":"2024-04-27T03:21:00.848140+0000","last_fullsized":"2024-04-27T03:21:00.848140+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:54:11.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814870+0000","last_change":"2024-04-27T03:17:49.017246+0000","last_active":"2024-04-27T03:19:34.814870+0000","last_peered":"2024-04-27T03:19:34.814870+0000","last_clean":"2024-04-27T03:19:34.814870+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:19:34.814870+0000","last_undegraded":"2024-04-27T03:19:34.814870+0000","last_fullsized":"2024-04-27T03:19:34.814870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:15.229307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436921+0000","last_change":"2024-04-27T03:17:47.032159+0000","last_active":"2024-04-27T03:19:35.436921+0000","last_peered":"2024-04-27T03:19:35.436921+0000","last_clean":"2024-04-27T03:19:35.436921+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:19:35.436921+0000","last_undegraded":"2024-04-27T03:19:35.436921+0000","last_fullsized":"2024-04-27T03:19:35.436921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:43.992214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815036+0000","last_change":"2024-04-27T03:17:51.012211+0000","last_active":"2024-04-27T03:19:34.815036+0000","last_peered":"2024-04-27T03:19:34.815036+0000","last_clean":"2024-04-27T03:19:34.815036+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:19:34.815036+0000","last_undegraded":"2024-04-27T03:19:34.815036+0000","last_fullsized":"2024-04-27T03:19:34.815036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"27'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384958+0000","last_change":"2024-04-27T03:17:53.049040+0000","last_active":"2024-04-27T03:19:35.384958+0000","last_peered":"2024-04-27T03:19:35.384958+0000","last_clean":"2024-04-27T03:19:35.384958+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:19:35.384958+0000","last_undegraded":"2024-04-27T03:19:35.384958+0000","last_fullsized":"2024-04-27T03:19:35.384958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:34.768461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819788+0000","last_change":"2024-04-27T03:17:55.058791+0000","last_active":"2024-04-27T03:19:34.819788+0000","last_peered":"2024-04-27T03:19:34.819788+0000","last_clean":"2024-04-27T03:19:34.819788+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:19:34.819788+0000","last_undegraded":"2024-04-27T03:19:34.819788+0000","last_fullsized":"2024-04-27T03:19:34.819788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809843+0000","last_change":"2024-04-27T03:17:57.074831+0000","last_active":"2024-04-27T03:19:34.809843+0000","last_peered":"2024-04-27T03:19:34.809843+0000","last_clean":"2024-04-27T03:19:34.809843+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:19:34.809843+0000","last_undegraded":"2024-04-27T03:19:34.809843+0000","last_fullsized":"2024-04-27T03:19:34.809843+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'15","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.165698+0000","last_change":"2024-04-27T03:17:49.022969+0000","last_active":"2024-04-27T03:21:01.165698+0000","last_peered":"2024-04-27T03:21:01.165698+0000","last_clean":"2024-04-27T03:21:01.165698+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:01.165698+0000","last_undegraded":"2024-04-27T03:21:01.165698+0000","last_fullsized":"2024-04-27T03:21:01.165698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814358+0000","last_change":"2024-04-27T03:17:47.009721+0000","last_active":"2024-04-27T03:19:34.814358+0000","last_peered":"2024-04-27T03:19:34.814358+0000","last_clean":"2024-04-27T03:19:34.814358+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:19:34.814358+0000","last_undegraded":"2024-04-27T03:19:34.814358+0000","last_fullsized":"2024-04-27T03:19:34.814358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814852+0000","last_change":"2024-04-27T03:17:51.034093+0000","last_active":"2024-04-27T03:19:34.814852+0000","last_peered":"2024-04-27T03:19:34.814852+0000","last_clean":"2024-04-27T03:19:34.814852+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:19:34.814852+0000","last_undegraded":"2024-04-27T03:19:34.814852+0000","last_fullsized":"2024-04-27T03:19:34.814852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"27'22","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440526+0000","last_change":"2024-04-27T03:17:53.032443+0000","last_active":"2024-04-27T03:19:35.440526+0000","last_peered":"2024-04-27T03:19:35.440526+0000","last_clean":"2024-04-27T03:19:35.440526+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:19:35.440526+0000","last_undegraded":"2024-04-27T03:19:35.440526+0000","last_fullsized":"2024-04-27T03:19:35.440526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:35.232648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'282","reported_seq":426,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277953+0000","last_change":"2024-04-27T03:17:55.069049+0000","last_active":"2024-04-27T03:21:01.277953+0000","last_peered":"2024-04-27T03:21:01.277953+0000","last_clean":"2024-04-27T03:21:01.277953+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:01.277953+0000","last_undegraded":"2024-04-27T03:21:01.277953+0000","last_fullsized":"2024-04-27T03:21:01.277953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438345+0000","last_change":"2024-04-27T03:17:57.072058+0000","last_active":"2024-04-27T03:19:35.438345+0000","last_peered":"2024-04-27T03:19:35.438345+0000","last_clean":"2024-04-27T03:19:35.438345+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:19:35.438345+0000","last_undegraded":"2024-04-27T03:19:35.438345+0000","last_fullsized":"2024-04-27T03:19:35.438345+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:56.749229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950160+0000","last_change":"2024-04-27T03:17:49.017573+0000","last_active":"2024-04-27T03:21:00.950160+0000","last_peered":"2024-04-27T03:21:00.950160+0000","last_clean":"2024-04-27T03:21:00.950160+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:00.950160+0000","last_undegraded":"2024-04-27T03:21:00.950160+0000","last_fullsized":"2024-04-27T03:21:00.950160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:04.132727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813820+0000","last_change":"2024-04-27T03:17:47.003169+0000","last_active":"2024-04-27T03:19:34.813820+0000","last_peered":"2024-04-27T03:19:34.813820+0000","last_clean":"2024-04-27T03:19:34.813820+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:19:34.813820+0000","last_undegraded":"2024-04-27T03:19:34.813820+0000","last_fullsized":"2024-04-27T03:19:34.813820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822747+0000","last_change":"2024-04-27T03:17:51.020242+0000","last_active":"2024-04-27T03:19:34.822747+0000","last_peered":"2024-04-27T03:19:34.822747+0000","last_clean":"2024-04-27T03:19:34.822747+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:19:34.822747+0000","last_undegraded":"2024-04-27T03:19:34.822747+0000","last_fullsized":"2024-04-27T03:19:34.822747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'36","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.265097+0000","last_change":"2024-04-27T03:17:53.038866+0000","last_active":"2024-04-27T03:19:58.265097+0000","last_peered":"2024-04-27T03:19:58.265097+0000","last_clean":"2024-04-27T03:19:58.265097+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:19:58.265097+0000","last_undegraded":"2024-04-27T03:19:58.265097+0000","last_fullsized":"2024-04-27T03:19:58.265097+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:23.026213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'282","reported_seq":406,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.306917+0000","last_change":"2024-04-27T03:17:55.074657+0000","last_active":"2024-04-27T03:21:01.306917+0000","last_peered":"2024-04-27T03:21:01.306917+0000","last_clean":"2024-04-27T03:21:01.306917+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:01.306917+0000","last_undegraded":"2024-04-27T03:21:01.306917+0000","last_fullsized":"2024-04-27T03:21:01.306917+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'31","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246782+0000","last_change":"2024-04-27T03:18:34.790257+0000","last_active":"2024-04-27T03:21:01.246782+0000","last_peered":"2024-04-27T03:21:01.246782+0000","last_clean":"2024-04-27T03:21:01.246782+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:21:01.246782+0000","last_undegraded":"2024-04-27T03:21:01.246782+0000","last_fullsized":"2024-04-27T03:21:01.246782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:11.176662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.716875+0000","last_change":"2024-04-27T03:17:47.013403+0000","last_active":"2024-04-27T03:21:00.716875+0000","last_peered":"2024-04-27T03:21:00.716875+0000","last_clean":"2024-04-27T03:21:00.716875+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:21:00.716875+0000","last_undegraded":"2024-04-27T03:21:00.716875+0000","last_fullsized":"2024-04-27T03:21:00.716875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:47.849076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377293+0000","last_change":"2024-04-27T03:17:51.027190+0000","last_active":"2024-04-27T03:19:35.377293+0000","last_peered":"2024-04-27T03:19:35.377293+0000","last_clean":"2024-04-27T03:19:35.377293+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:19:35.377293+0000","last_undegraded":"2024-04-27T03:19:35.377293+0000","last_fullsized":"2024-04-27T03:19:35.377293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:50.708783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'47","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.221245+0000","last_change":"2024-04-27T03:17:53.060562+0000","last_active":"2024-04-27T03:20:00.221245+0000","last_peered":"2024-04-27T03:20:00.221245+0000","last_clean":"2024-04-27T03:20:00.221245+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:20:00.221245+0000","last_undegraded":"2024-04-27T03:20:00.221245+0000","last_fullsized":"2024-04-27T03:20:00.221245+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":41,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380552+0000","last_change":"2024-04-27T03:17:55.063723+0000","last_active":"2024-04-27T03:19:35.380552+0000","last_peered":"2024-04-27T03:19:35.380552+0000","last_clean":"2024-04-27T03:19:35.380552+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:19:35.380552+0000","last_undegraded":"2024-04-27T03:19:35.380552+0000","last_fullsized":"2024-04-27T03:19:35.380552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:59.616012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439014+0000","last_change":"2024-04-27T03:17:57.086011+0000","last_active":"2024-04-27T03:19:35.439014+0000","last_peered":"2024-04-27T03:19:35.439014+0000","last_clean":"2024-04-27T03:19:35.439014+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:19:35.439014+0000","last_undegraded":"2024-04-27T03:19:35.439014+0000","last_fullsized":"2024-04-27T03:19:35.439014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:11.219539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299288+0000","last_change":"2024-04-27T03:17:49.003206+0000","last_active":"2024-04-27T03:21:01.299288+0000","last_peered":"2024-04-27T03:21:01.299288+0000","last_clean":"2024-04-27T03:21:01.299288+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:01.299288+0000","last_undegraded":"2024-04-27T03:21:01.299288+0000","last_fullsized":"2024-04-27T03:21:01.299288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:42.242631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820229+0000","last_change":"2024-04-27T03:17:46.995080+0000","last_active":"2024-04-27T03:19:34.820229+0000","last_peered":"2024-04-27T03:19:34.820229+0000","last_clean":"2024-04-27T03:19:34.820229+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:19:34.820229+0000","last_undegraded":"2024-04-27T03:19:34.820229+0000","last_fullsized":"2024-04-27T03:19:34.820229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437771+0000","last_change":"2024-04-27T03:17:51.036969+0000","last_active":"2024-04-27T03:19:35.437771+0000","last_peered":"2024-04-27T03:19:35.437771+0000","last_clean":"2024-04-27T03:19:35.437771+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:19:35.437771+0000","last_undegraded":"2024-04-27T03:19:35.437771+0000","last_fullsized":"2024-04-27T03:19:35.437771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:48:00.532358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'37","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.245893+0000","last_change":"2024-04-27T03:17:53.041431+0000","last_active":"2024-04-27T03:19:58.245893+0000","last_peered":"2024-04-27T03:19:58.245893+0000","last_clean":"2024-04-27T03:19:58.245893+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:19:58.245893+0000","last_undegraded":"2024-04-27T03:19:58.245893+0000","last_fullsized":"2024-04-27T03:19:58.245893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809982+0000","last_change":"2024-04-27T03:17:55.075008+0000","last_active":"2024-04-27T03:19:34.809982+0000","last_peered":"2024-04-27T03:19:34.809982+0000","last_clean":"2024-04-27T03:19:34.809982+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:19:34.809982+0000","last_undegraded":"2024-04-27T03:19:34.809982+0000","last_fullsized":"2024-04-27T03:19:34.809982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:01.297073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"29'5","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382753+0000","last_change":"2024-04-27T03:17:57.084457+0000","last_active":"2024-04-27T03:19:35.382753+0000","last_peered":"2024-04-27T03:19:35.382753+0000","last_clean":"2024-04-27T03:19:35.382753+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:19:35.382753+0000","last_undegraded":"2024-04-27T03:19:35.382753+0000","last_fullsized":"2024-04-27T03:19:35.382753+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:34.604396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.827907+0000","last_change":"2024-04-27T03:18:34.767220+0000","last_active":"2024-04-27T03:21:00.827907+0000","last_peered":"2024-04-27T03:21:00.827907+0000","last_clean":"2024-04-27T03:21:00.827907+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:00.827907+0000","last_undegraded":"2024-04-27T03:21:00.827907+0000","last_fullsized":"2024-04-27T03:21:00.827907+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:38.334837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820841+0000","last_change":"2024-04-27T03:17:47.028376+0000","last_active":"2024-04-27T03:19:34.820841+0000","last_peered":"2024-04-27T03:19:34.820841+0000","last_clean":"2024-04-27T03:19:34.820841+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:19:34.820841+0000","last_undegraded":"2024-04-27T03:19:34.820841+0000","last_fullsized":"2024-04-27T03:19:34.820841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:18.893441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438808+0000","last_change":"2024-04-27T03:17:51.037525+0000","last_active":"2024-04-27T03:19:35.438808+0000","last_peered":"2024-04-27T03:19:35.438808+0000","last_clean":"2024-04-27T03:19:35.438808+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:19:35.438808+0000","last_undegraded":"2024-04-27T03:19:35.438808+0000","last_fullsized":"2024-04-27T03:19:35.438808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:38.895777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'48","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.284523+0000","last_change":"2024-04-27T03:18:34.800954+0000","last_active":"2024-04-27T03:19:58.284523+0000","last_peered":"2024-04-27T03:19:58.284523+0000","last_clean":"2024-04-27T03:19:58.284523+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:19:58.284523+0000","last_undegraded":"2024-04-27T03:19:58.284523+0000","last_fullsized":"2024-04-27T03:19:58.284523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:59.816977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'282","reported_seq":381,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.495106+0000","last_change":"2024-04-27T03:17:55.063480+0000","last_active":"2024-04-27T03:21:01.495106+0000","last_peered":"2024-04-27T03:21:01.495106+0000","last_clean":"2024-04-27T03:21:01.495106+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:01.495106+0000","last_undegraded":"2024-04-27T03:21:01.495106+0000","last_fullsized":"2024-04-27T03:21:01.495106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:25.995264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382973+0000","last_change":"2024-04-27T03:17:57.077044+0000","last_active":"2024-04-27T03:19:35.382973+0000","last_peered":"2024-04-27T03:19:35.382973+0000","last_clean":"2024-04-27T03:19:35.382973+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:19:35.382973+0000","last_undegraded":"2024-04-27T03:19:35.382973+0000","last_fullsized":"2024-04-27T03:19:35.382973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:03.311426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299661+0000","last_change":"2024-04-27T03:17:49.007911+0000","last_active":"2024-04-27T03:21:01.299661+0000","last_peered":"2024-04-27T03:21:01.299661+0000","last_clean":"2024-04-27T03:21:01.299661+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:01.299661+0000","last_undegraded":"2024-04-27T03:21:01.299661+0000","last_fullsized":"2024-04-27T03:21:01.299661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:07.206167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816213+0000","last_change":"2024-04-27T03:18:34.772611+0000","last_active":"2024-04-27T03:19:34.816213+0000","last_peered":"2024-04-27T03:19:34.816213+0000","last_clean":"2024-04-27T03:19:34.816213+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:19:34.816213+0000","last_undegraded":"2024-04-27T03:19:34.816213+0000","last_fullsized":"2024-04-27T03:19:34.816213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:47.500325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437638+0000","last_change":"2024-04-27T03:17:51.038976+0000","last_active":"2024-04-27T03:19:35.437638+0000","last_peered":"2024-04-27T03:19:35.437638+0000","last_clean":"2024-04-27T03:19:35.437638+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:19:35.437638+0000","last_undegraded":"2024-04-27T03:19:35.437638+0000","last_fullsized":"2024-04-27T03:19:35.437638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:17:48.170469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'66","reported_seq":138,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.287412+0000","last_change":"2024-04-27T03:17:53.050762+0000","last_active":"2024-04-27T03:19:58.287412+0000","last_peered":"2024-04-27T03:19:58.287412+0000","last_clean":"2024-04-27T03:19:58.287412+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:19:58.287412+0000","last_undegraded":"2024-04-27T03:19:58.287412+0000","last_fullsized":"2024-04-27T03:19:58.287412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":67,"num_read_kb":56,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436043+0000","last_change":"2024-04-27T03:17:55.057576+0000","last_active":"2024-04-27T03:19:35.436043+0000","last_peered":"2024-04-27T03:19:35.436043+0000","last_clean":"2024-04-27T03:19:35.436043+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:19:35.436043+0000","last_undegraded":"2024-04-27T03:19:35.436043+0000","last_fullsized":"2024-04-27T03:19:35.436043+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:41.241891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'18","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.456884+0000","last_change":"2024-04-27T03:17:57.062348+0000","last_active":"2024-04-27T03:21:01.456884+0000","last_peered":"2024-04-27T03:21:01.456884+0000","last_clean":"2024-04-27T03:21:01.456884+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:01.456884+0000","last_undegraded":"2024-04-27T03:21:01.456884+0000","last_fullsized":"2024-04-27T03:21:01.456884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:14.902153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232625+0000","last_change":"2024-04-27T03:17:49.009833+0000","last_active":"2024-04-27T03:21:01.232625+0000","last_peered":"2024-04-27T03:21:01.232625+0000","last_clean":"2024-04-27T03:21:01.232625+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:01.232625+0000","last_undegraded":"2024-04-27T03:21:01.232625+0000","last_fullsized":"2024-04-27T03:21:01.232625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:36:59.530315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437653+0000","last_change":"2024-04-27T03:18:34.767518+0000","last_active":"2024-04-27T03:19:35.437653+0000","last_peered":"2024-04-27T03:19:35.437653+0000","last_clean":"2024-04-27T03:19:35.437653+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:19:35.437653+0000","last_undegraded":"2024-04-27T03:19:35.437653+0000","last_fullsized":"2024-04-27T03:19:35.437653+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:02.579718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435824+0000","last_change":"2024-04-27T03:17:51.029195+0000","last_active":"2024-04-27T03:19:35.435824+0000","last_peered":"2024-04-27T03:19:35.435824+0000","last_clean":"2024-04-27T03:19:35.435824+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:19:35.435824+0000","last_undegraded":"2024-04-27T03:19:35.435824+0000","last_fullsized":"2024-04-27T03:19:35.435824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:42.059879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'40","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.253858+0000","last_change":"2024-04-27T03:17:53.049067+0000","last_active":"2024-04-27T03:19:58.253858+0000","last_peered":"2024-04-27T03:19:58.253858+0000","last_clean":"2024-04-27T03:19:58.253858+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:19:58.253858+0000","last_undegraded":"2024-04-27T03:19:58.253858+0000","last_fullsized":"2024-04-27T03:19:58.253858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:55:37.400674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":54,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815163+0000","last_change":"2024-04-27T03:17:55.072520+0000","last_active":"2024-04-27T03:19:34.815163+0000","last_peered":"2024-04-27T03:19:34.815163+0000","last_clean":"2024-04-27T03:19:34.815163+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:19:34.815163+0000","last_undegraded":"2024-04-27T03:19:34.815163+0000","last_fullsized":"2024-04-27T03:19:34.815163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'3","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.883363+0000","last_change":"2024-04-27T03:17:57.080943+0000","last_active":"2024-04-27T03:19:57.883363+0000","last_peered":"2024-04-27T03:19:57.883363+0000","last_clean":"2024-04-27T03:19:57.883363+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:19:57.883363+0000","last_undegraded":"2024-04-27T03:19:57.883363+0000","last_fullsized":"2024-04-27T03:19:57.883363+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822407+0000","last_change":"2024-04-27T03:17:49.016254+0000","last_active":"2024-04-27T03:19:34.822407+0000","last_peered":"2024-04-27T03:19:34.822407+0000","last_clean":"2024-04-27T03:19:34.822407+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:19:34.822407+0000","last_undegraded":"2024-04-27T03:19:34.822407+0000","last_fullsized":"2024-04-27T03:19:34.822407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817143+0000","last_change":"2024-04-27T03:17:47.026419+0000","last_active":"2024-04-27T03:19:34.817143+0000","last_peered":"2024-04-27T03:19:34.817143+0000","last_clean":"2024-04-27T03:19:34.817143+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:19:34.817143+0000","last_undegraded":"2024-04-27T03:19:34.817143+0000","last_fullsized":"2024-04-27T03:19:34.817143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:25.121818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377466+0000","last_change":"2024-04-27T03:17:51.036820+0000","last_active":"2024-04-27T03:19:35.377466+0000","last_peered":"2024-04-27T03:19:35.377466+0000","last_clean":"2024-04-27T03:19:35.377466+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:19:35.377466+0000","last_undegraded":"2024-04-27T03:19:35.377466+0000","last_fullsized":"2024-04-27T03:19:35.377466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:24.012671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.298340+0000","last_change":"2024-04-27T03:17:53.043412+0000","last_active":"2024-04-27T03:19:58.298340+0000","last_peered":"2024-04-27T03:19:58.298340+0000","last_clean":"2024-04-27T03:19:58.298340+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:19:58.298340+0000","last_undegraded":"2024-04-27T03:19:58.298340+0000","last_fullsized":"2024-04-27T03:19:58.298340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:37:03.560918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'282","reported_seq":408,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.467183+0000","last_change":"2024-04-27T03:17:55.070428+0000","last_active":"2024-04-27T03:21:01.467183+0000","last_peered":"2024-04-27T03:21:01.467183+0000","last_clean":"2024-04-27T03:21:01.467183+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:01.467183+0000","last_undegraded":"2024-04-27T03:21:01.467183+0000","last_fullsized":"2024-04-27T03:21:01.467183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:20.977192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'12","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.727848+0000","last_change":"2024-04-27T03:17:57.077926+0000","last_active":"2024-04-27T03:20:02.727848+0000","last_peered":"2024-04-27T03:20:02.727848+0000","last_clean":"2024-04-27T03:20:02.727848+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:20:02.727848+0000","last_undegraded":"2024-04-27T03:20:02.727848+0000","last_fullsized":"2024-04-27T03:20:02.727848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:14:58.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.441233+0000","last_change":"2024-04-27T03:17:49.008951+0000","last_active":"2024-04-27T03:19:35.441233+0000","last_peered":"2024-04-27T03:19:35.441233+0000","last_clean":"2024-04-27T03:19:35.441233+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:19:35.441233+0000","last_undegraded":"2024-04-27T03:19:35.441233+0000","last_fullsized":"2024-04-27T03:19:35.441233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:15.953778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810391+0000","last_change":"2024-04-27T03:17:47.036712+0000","last_active":"2024-04-27T03:19:34.810391+0000","last_peered":"2024-04-27T03:19:34.810391+0000","last_clean":"2024-04-27T03:19:34.810391+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:19:34.810391+0000","last_undegraded":"2024-04-27T03:19:34.810391+0000","last_fullsized":"2024-04-27T03:19:34.810391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'3","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724954+0000","last_change":"2024-04-27T03:17:51.037543+0000","last_active":"2024-04-27T03:19:59.724954+0000","last_peered":"2024-04-27T03:19:59.724954+0000","last_clean":"2024-04-27T03:19:59.724954+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:19:59.724954+0000","last_undegraded":"2024-04-27T03:19:59.724954+0000","last_fullsized":"2024-04-27T03:19:59.724954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":381,"num_read_kb":381,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"27'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814593+0000","last_change":"2024-04-27T03:17:53.042236+0000","last_active":"2024-04-27T03:19:34.814593+0000","last_peered":"2024-04-27T03:19:34.814593+0000","last_clean":"2024-04-27T03:19:34.814593+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:19:34.814593+0000","last_undegraded":"2024-04-27T03:19:34.814593+0000","last_fullsized":"2024-04-27T03:19:34.814593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'282","reported_seq":464,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.408184+0000","last_change":"2024-04-27T03:17:55.068988+0000","last_active":"2024-04-27T03:21:01.408184+0000","last_peered":"2024-04-27T03:21:01.408184+0000","last_clean":"2024-04-27T03:21:01.408184+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:01.408184+0000","last_undegraded":"2024-04-27T03:21:01.408184+0000","last_fullsized":"2024-04-27T03:21:01.408184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.880007+0000","last_change":"2024-04-27T03:17:57.074965+0000","last_active":"2024-04-27T03:19:59.880007+0000","last_peered":"2024-04-27T03:19:59.880007+0000","last_clean":"2024-04-27T03:19:59.880007+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:19:59.880007+0000","last_undegraded":"2024-04-27T03:19:59.880007+0000","last_fullsized":"2024-04-27T03:19:59.880007+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:30.053117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386155+0000","last_change":"2024-04-27T03:17:49.018519+0000","last_active":"2024-04-27T03:19:35.386155+0000","last_peered":"2024-04-27T03:19:35.386155+0000","last_clean":"2024-04-27T03:19:35.386155+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:19:35.386155+0000","last_undegraded":"2024-04-27T03:19:35.386155+0000","last_fullsized":"2024-04-27T03:19:35.386155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:46.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809539+0000","last_change":"2024-04-27T03:17:47.031891+0000","last_active":"2024-04-27T03:19:34.809539+0000","last_peered":"2024-04-27T03:19:34.809539+0000","last_clean":"2024-04-27T03:19:34.809539+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:19:34.809539+0000","last_undegraded":"2024-04-27T03:19:34.809539+0000","last_fullsized":"2024-04-27T03:19:34.809539+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815512+0000","last_change":"2024-04-27T03:17:51.033741+0000","last_active":"2024-04-27T03:19:34.815512+0000","last_peered":"2024-04-27T03:19:34.815512+0000","last_clean":"2024-04-27T03:19:34.815512+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:19:34.815512+0000","last_undegraded":"2024-04-27T03:19:34.815512+0000","last_fullsized":"2024-04-27T03:19:34.815512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'38","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.276705+0000","last_change":"2024-04-27T03:17:53.048795+0000","last_active":"2024-04-27T03:19:58.276705+0000","last_peered":"2024-04-27T03:19:58.276705+0000","last_clean":"2024-04-27T03:19:58.276705+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:19:58.276705+0000","last_undegraded":"2024-04-27T03:19:58.276705+0000","last_fullsized":"2024-04-27T03:19:58.276705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817470+0000","last_change":"2024-04-27T03:17:55.071142+0000","last_active":"2024-04-27T03:19:34.817470+0000","last_peered":"2024-04-27T03:19:34.817470+0000","last_clean":"2024-04-27T03:19:34.817470+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:19:34.817470+0000","last_undegraded":"2024-04-27T03:19:34.817470+0000","last_fullsized":"2024-04-27T03:19:34.817470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'8","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.368117+0000","last_change":"2024-04-27T03:17:57.071496+0000","last_active":"2024-04-27T03:21:01.368117+0000","last_peered":"2024-04-27T03:21:01.368117+0000","last_clean":"2024-04-27T03:21:01.368117+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:01.368117+0000","last_undegraded":"2024-04-27T03:21:01.368117+0000","last_fullsized":"2024-04-27T03:21:01.368117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:32:20.548838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.811995+0000","last_change":"2024-04-27T03:17:49.013855+0000","last_active":"2024-04-27T03:21:00.811995+0000","last_peered":"2024-04-27T03:21:00.811995+0000","last_clean":"2024-04-27T03:21:00.811995+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:00.811995+0000","last_undegraded":"2024-04-27T03:21:00.811995+0000","last_fullsized":"2024-04-27T03:21:00.811995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":201,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379585+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:19:35.379585+0000","last_peered":"2024-04-27T03:19:35.379585+0000","last_clean":"2024-04-27T03:19:35.379585+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:19:35.379585+0000","last_undegraded":"2024-04-27T03:19:35.379585+0000","last_fullsized":"2024-04-27T03:19:35.379585+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816708+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:19:34.816708+0000","last_peered":"2024-04-27T03:19:34.816708+0000","last_clean":"2024-04-27T03:19:34.816708+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:19:34.816708+0000","last_undegraded":"2024-04-27T03:19:34.816708+0000","last_fullsized":"2024-04-27T03:19:34.816708+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:25.777031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.100108+0000","last_change":"2024-04-27T03:17:47.039911+0000","last_active":"2024-04-27T03:21:01.100108+0000","last_peered":"2024-04-27T03:21:01.100108+0000","last_clean":"2024-04-27T03:21:01.100108+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:21:01.100108+0000","last_undegraded":"2024-04-27T03:21:01.100108+0000","last_fullsized":"2024-04-27T03:21:01.100108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382198+0000","last_change":"2024-04-27T03:17:51.036961+0000","last_active":"2024-04-27T03:19:35.382198+0000","last_peered":"2024-04-27T03:19:35.382198+0000","last_clean":"2024-04-27T03:19:35.382198+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:19:35.382198+0000","last_undegraded":"2024-04-27T03:19:35.382198+0000","last_fullsized":"2024-04-27T03:19:35.382198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:16.046086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'82","reported_seq":161,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.296556+0000","last_change":"2024-04-27T03:17:53.051108+0000","last_active":"2024-04-27T03:19:58.296556+0000","last_peered":"2024-04-27T03:19:58.296556+0000","last_clean":"2024-04-27T03:19:58.296556+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:19:58.296556+0000","last_undegraded":"2024-04-27T03:19:58.296556+0000","last_fullsized":"2024-04-27T03:19:58.296556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:32:18.585035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812224+0000","last_change":"2024-04-27T03:17:55.071697+0000","last_active":"2024-04-27T03:19:34.812224+0000","last_peered":"2024-04-27T03:19:34.812224+0000","last_clean":"2024-04-27T03:19:34.812224+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:19:34.812224+0000","last_undegraded":"2024-04-27T03:19:34.812224+0000","last_fullsized":"2024-04-27T03:19:34.812224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":167,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.324346+0000","last_change":"2024-04-27T03:17:57.075459+0000","last_active":"2024-04-27T03:21:01.324346+0000","last_peered":"2024-04-27T03:21:01.324346+0000","last_clean":"2024-04-27T03:21:01.324346+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:01.324346+0000","last_undegraded":"2024-04-27T03:21:01.324346+0000","last_fullsized":"2024-04-27T03:21:01.324346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:22.086532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233419+0000","last_change":"2024-04-27T03:17:49.007601+0000","last_active":"2024-04-27T03:21:01.233419+0000","last_peered":"2024-04-27T03:21:01.233419+0000","last_clean":"2024-04-27T03:21:01.233419+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:21:01.233419+0000","last_undegraded":"2024-04-27T03:21:01.233419+0000","last_fullsized":"2024-04-27T03:21:01.233419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815344+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:19:34.815344+0000","last_peered":"2024-04-27T03:19:34.815344+0000","last_clean":"2024-04-27T03:19:34.815344+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:19:34.815344+0000","last_undegraded":"2024-04-27T03:19:34.815344+0000","last_fullsized":"2024-04-27T03:19:34.815344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:47:23.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380638+0000","last_change":"2024-04-27T03:17:47.015528+0000","last_active":"2024-04-27T03:19:35.380638+0000","last_peered":"2024-04-27T03:19:35.380638+0000","last_clean":"2024-04-27T03:19:35.380638+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:19:35.380638+0000","last_undegraded":"2024-04-27T03:19:35.380638+0000","last_fullsized":"2024-04-27T03:19:35.380638+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822099+0000","last_change":"2024-04-27T03:17:51.034048+0000","last_active":"2024-04-27T03:19:34.822099+0000","last_peered":"2024-04-27T03:19:34.822099+0000","last_clean":"2024-04-27T03:19:34.822099+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:19:34.822099+0000","last_undegraded":"2024-04-27T03:19:34.822099+0000","last_fullsized":"2024-04-27T03:19:34.822099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'279","reported_seq":493,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.279126+0000","last_change":"2024-04-27T03:17:53.058256+0000","last_active":"2024-04-27T03:21:01.279126+0000","last_peered":"2024-04-27T03:21:01.279126+0000","last_clean":"2024-04-27T03:21:01.279126+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:01.279126+0000","last_undegraded":"2024-04-27T03:21:01.279126+0000","last_fullsized":"2024-04-27T03:21:01.279126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:08.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":366,"num_read_kb":350,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811204+0000","last_change":"2024-04-27T03:17:55.074606+0000","last_active":"2024-04-27T03:19:34.811204+0000","last_peered":"2024-04-27T03:19:34.811204+0000","last_clean":"2024-04-27T03:19:34.811204+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:19:34.811204+0000","last_undegraded":"2024-04-27T03:19:34.811204+0000","last_fullsized":"2024-04-27T03:19:34.811204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379830+0000","last_change":"2024-04-27T03:17:57.086888+0000","last_active":"2024-04-27T03:19:35.379830+0000","last_peered":"2024-04-27T03:19:35.379830+0000","last_clean":"2024-04-27T03:19:35.379830+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:19:35.379830+0000","last_undegraded":"2024-04-27T03:19:35.379830+0000","last_fullsized":"2024-04-27T03:19:35.379830+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:30.189251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232768+0000","last_change":"2024-04-27T03:17:49.014111+0000","last_active":"2024-04-27T03:21:01.232768+0000","last_peered":"2024-04-27T03:21:01.232768+0000","last_clean":"2024-04-27T03:21:01.232768+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:01.232768+0000","last_undegraded":"2024-04-27T03:21:01.232768+0000","last_fullsized":"2024-04-27T03:21:01.232768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:15.967813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436752+0000","last_change":"2024-04-27T03:17:39.633841+0000","last_active":"2024-04-27T03:19:35.436752+0000","last_peered":"2024-04-27T03:19:35.436752+0000","last_clean":"2024-04-27T03:19:35.436752+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:19:35.436752+0000","last_undegraded":"2024-04-27T03:19:35.436752+0000","last_fullsized":"2024-04-27T03:19:35.436752+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.559603+0000","last_change":"2024-04-27T03:17:47.011695+0000","last_active":"2024-04-27T03:21:00.559603+0000","last_peered":"2024-04-27T03:21:00.559603+0000","last_clean":"2024-04-27T03:21:00.559603+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:21:00.559603+0000","last_undegraded":"2024-04-27T03:21:00.559603+0000","last_fullsized":"2024-04-27T03:21:00.559603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"20'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435807+0000","last_change":"2024-04-27T03:17:51.031676+0000","last_active":"2024-04-27T03:19:35.435807+0000","last_peered":"2024-04-27T03:19:35.435807+0000","last_clean":"2024-04-27T03:19:35.435807+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:19:35.435807+0000","last_undegraded":"2024-04-27T03:19:35.435807+0000","last_fullsized":"2024-04-27T03:19:35.435807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:23.681601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'16","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.278287+0000","last_change":"2024-04-27T03:17:53.051185+0000","last_active":"2024-04-27T03:21:01.278287+0000","last_peered":"2024-04-27T03:21:01.278287+0000","last_clean":"2024-04-27T03:21:01.278287+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:01.278287+0000","last_undegraded":"2024-04-27T03:21:01.278287+0000","last_fullsized":"2024-04-27T03:21:01.278287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382273+0000","last_change":"2024-04-27T03:17:55.070180+0000","last_active":"2024-04-27T03:19:35.382273+0000","last_peered":"2024-04-27T03:19:35.382273+0000","last_clean":"2024-04-27T03:19:35.382273+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:19:35.382273+0000","last_undegraded":"2024-04-27T03:19:35.382273+0000","last_fullsized":"2024-04-27T03:19:35.382273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:15.975975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.206940+0000","last_change":"2024-04-27T03:17:57.071109+0000","last_active":"2024-04-27T03:21:01.206940+0000","last_peered":"2024-04-27T03:21:01.206940+0000","last_clean":"2024-04-27T03:21:01.206940+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:01.206940+0000","last_undegraded":"2024-04-27T03:21:01.206940+0000","last_fullsized":"2024-04-27T03:21:01.206940+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802933+0000","last_change":"2024-04-27T03:17:49.009764+0000","last_active":"2024-04-27T03:21:00.802933+0000","last_peered":"2024-04-27T03:21:00.802933+0000","last_clean":"2024-04-27T03:21:00.802933+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:00.802933+0000","last_undegraded":"2024-04-27T03:21:00.802933+0000","last_fullsized":"2024-04-27T03:21:00.802933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:31.202044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816459+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:19:34.816459+0000","last_peered":"2024-04-27T03:19:34.816459+0000","last_clean":"2024-04-27T03:19:34.816459+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:19:34.816459+0000","last_undegraded":"2024-04-27T03:19:34.816459+0000","last_fullsized":"2024-04-27T03:19:34.816459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.133613+0000","last_change":"2024-04-27T03:17:47.015018+0000","last_active":"2024-04-27T03:21:01.133613+0000","last_peered":"2024-04-27T03:21:01.133613+0000","last_clean":"2024-04-27T03:21:01.133613+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:21:01.133613+0000","last_undegraded":"2024-04-27T03:21:01.133613+0000","last_fullsized":"2024-04-27T03:21:01.133613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.378112+0000","last_change":"2024-04-27T03:17:51.029533+0000","last_active":"2024-04-27T03:19:35.378112+0000","last_peered":"2024-04-27T03:19:35.378112+0000","last_clean":"2024-04-27T03:19:35.378112+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:19:35.378112+0000","last_undegraded":"2024-04-27T03:19:35.378112+0000","last_fullsized":"2024-04-27T03:19:35.378112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:52.883679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"29'14","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.236793+0000","last_change":"2024-04-27T03:17:53.045958+0000","last_active":"2024-04-27T03:21:00.236793+0000","last_peered":"2024-04-27T03:21:00.236793+0000","last_clean":"2024-04-27T03:21:00.236793+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:00.236793+0000","last_undegraded":"2024-04-27T03:21:00.236793+0000","last_fullsized":"2024-04-27T03:21:00.236793+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815623+0000","last_change":"2024-04-27T03:17:55.078514+0000","last_active":"2024-04-27T03:19:34.815623+0000","last_peered":"2024-04-27T03:19:34.815623+0000","last_clean":"2024-04-27T03:19:34.815623+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:19:34.815623+0000","last_undegraded":"2024-04-27T03:19:34.815623+0000","last_fullsized":"2024-04-27T03:19:34.815623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.953374+0000","last_change":"2024-04-27T03:17:57.089242+0000","last_active":"2024-04-27T03:21:00.953374+0000","last_peered":"2024-04-27T03:21:00.953374+0000","last_clean":"2024-04-27T03:21:00.953374+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:21:00.953374+0000","last_undegraded":"2024-04-27T03:21:00.953374+0000","last_fullsized":"2024-04-27T03:21:00.953374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:12.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299342+0000","last_change":"2024-04-27T03:17:49.006116+0000","last_active":"2024-04-27T03:21:01.299342+0000","last_peered":"2024-04-27T03:21:01.299342+0000","last_clean":"2024-04-27T03:21:01.299342+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:01.299342+0000","last_undegraded":"2024-04-27T03:21:01.299342+0000","last_fullsized":"2024-04-27T03:21:01.299342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:05.838106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822302+0000","last_change":"2024-04-27T03:17:47.024776+0000","last_active":"2024-04-27T03:19:34.822302+0000","last_peered":"2024-04-27T03:19:34.822302+0000","last_clean":"2024-04-27T03:19:34.822302+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:19:34.822302+0000","last_undegraded":"2024-04-27T03:19:34.822302+0000","last_fullsized":"2024-04-27T03:19:34.822302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382818+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:19:35.382818+0000","last_peered":"2024-04-27T03:19:35.382818+0000","last_clean":"2024-04-27T03:19:35.382818+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:19:35.382818+0000","last_undegraded":"2024-04-27T03:19:35.382818+0000","last_fullsized":"2024-04-27T03:19:35.382818+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:01:48.896188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.733547+0000","last_change":"2024-04-27T03:17:51.037696+0000","last_active":"2024-04-27T03:19:57.733547+0000","last_peered":"2024-04-27T03:19:57.733547+0000","last_clean":"2024-04-27T03:19:57.733547+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:19:57.733547+0000","last_undegraded":"2024-04-27T03:19:57.733547+0000","last_fullsized":"2024-04-27T03:19:57.733547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:44.259718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'73","reported_seq":148,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.297267+0000","last_change":"2024-04-27T03:17:53.054938+0000","last_active":"2024-04-27T03:19:58.297267+0000","last_peered":"2024-04-27T03:19:58.297267+0000","last_clean":"2024-04-27T03:19:58.297267+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:19:58.297267+0000","last_undegraded":"2024-04-27T03:19:58.297267+0000","last_fullsized":"2024-04-27T03:19:58.297267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:38.318924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":63,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815814+0000","last_change":"2024-04-27T03:17:55.068676+0000","last_active":"2024-04-27T03:19:34.815814+0000","last_peered":"2024-04-27T03:19:34.815814+0000","last_clean":"2024-04-27T03:19:34.815814+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:19:34.815814+0000","last_undegraded":"2024-04-27T03:19:34.815814+0000","last_fullsized":"2024-04-27T03:19:34.815814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:04:55.186362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.082757+0000","last_change":"2024-04-27T03:17:57.068043+0000","last_active":"2024-04-27T03:20:00.082757+0000","last_peered":"2024-04-27T03:20:00.082757+0000","last_clean":"2024-04-27T03:20:00.082757+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:20:00.082757+0000","last_undegraded":"2024-04-27T03:20:00.082757+0000","last_fullsized":"2024-04-27T03:20:00.082757+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:14.279033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382103+0000","last_change":"2024-04-27T03:17:49.009882+0000","last_active":"2024-04-27T03:19:35.382103+0000","last_peered":"2024-04-27T03:19:35.382103+0000","last_clean":"2024-04-27T03:19:35.382103+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:19:35.382103+0000","last_undegraded":"2024-04-27T03:19:35.382103+0000","last_fullsized":"2024-04-27T03:19:35.382103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:36.878689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.286999+0000","last_change":"2024-04-27T03:17:47.030065+0000","last_active":"2024-04-27T03:21:01.286999+0000","last_peered":"2024-04-27T03:21:01.286999+0000","last_clean":"2024-04-27T03:21:01.286999+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:01.286999+0000","last_undegraded":"2024-04-27T03:21:01.286999+0000","last_fullsized":"2024-04-27T03:21:01.286999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:45:21.469900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":21,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820336+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:19:34.820336+0000","last_peered":"2024-04-27T03:19:34.820336+0000","last_clean":"2024-04-27T03:19:34.820336+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:19:34.820336+0000","last_undegraded":"2024-04-27T03:19:34.820336+0000","last_fullsized":"2024-04-27T03:19:34.820336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:48:41.511344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816118+0000","last_change":"2024-04-27T03:17:51.035808+0000","last_active":"2024-04-27T03:19:34.816118+0000","last_peered":"2024-04-27T03:19:34.816118+0000","last_clean":"2024-04-27T03:19:34.816118+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:19:34.816118+0000","last_undegraded":"2024-04-27T03:19:34.816118+0000","last_fullsized":"2024-04-27T03:19:34.816118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"27'20","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381588+0000","last_change":"2024-04-27T03:17:53.049471+0000","last_active":"2024-04-27T03:19:35.381588+0000","last_peered":"2024-04-27T03:19:35.381588+0000","last_clean":"2024-04-27T03:19:35.381588+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:19:35.381588+0000","last_undegraded":"2024-04-27T03:19:35.381588+0000","last_fullsized":"2024-04-27T03:19:35.381588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:42.365441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'282","reported_seq":361,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.481977+0000","last_change":"2024-04-27T03:17:55.077154+0000","last_active":"2024-04-27T03:21:01.481977+0000","last_peered":"2024-04-27T03:21:01.481977+0000","last_clean":"2024-04-27T03:21:01.481977+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:01.481977+0000","last_undegraded":"2024-04-27T03:21:01.481977+0000","last_fullsized":"2024-04-27T03:21:01.481977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:55:52.856782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.236855+0000","last_change":"2024-04-27T03:17:57.091210+0000","last_active":"2024-04-27T03:21:01.236855+0000","last_peered":"2024-04-27T03:21:01.236855+0000","last_clean":"2024-04-27T03:21:01.236855+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:21:01.236855+0000","last_undegraded":"2024-04-27T03:21:01.236855+0000","last_fullsized":"2024-04-27T03:21:01.236855+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:12.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948922+0000","last_change":"2024-04-27T03:17:49.007942+0000","last_active":"2024-04-27T03:21:00.948922+0000","last_peered":"2024-04-27T03:21:00.948922+0000","last_clean":"2024-04-27T03:21:00.948922+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:21:00.948922+0000","last_undegraded":"2024-04-27T03:21:00.948922+0000","last_fullsized":"2024-04-27T03:21:00.948922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815379+0000","last_change":"2024-04-27T03:17:47.039532+0000","last_active":"2024-04-27T03:19:34.815379+0000","last_peered":"2024-04-27T03:19:34.815379+0000","last_clean":"2024-04-27T03:19:34.815379+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:19:34.815379+0000","last_undegraded":"2024-04-27T03:19:34.815379+0000","last_fullsized":"2024-04-27T03:19:34.815379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384884+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:19:35.384884+0000","last_peered":"2024-04-27T03:19:35.384884+0000","last_clean":"2024-04-27T03:19:35.384884+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:19:35.384884+0000","last_undegraded":"2024-04-27T03:19:35.384884+0000","last_fullsized":"2024-04-27T03:19:35.384884+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:45.567442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815568+0000","last_change":"2024-04-27T03:17:51.020247+0000","last_active":"2024-04-27T03:19:34.815568+0000","last_peered":"2024-04-27T03:19:34.815568+0000","last_clean":"2024-04-27T03:19:34.815568+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:19:34.815568+0000","last_undegraded":"2024-04-27T03:19:34.815568+0000","last_fullsized":"2024-04-27T03:19:34.815568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812598+0000","last_change":"2024-04-27T03:17:53.048021+0000","last_active":"2024-04-27T03:19:34.812598+0000","last_peered":"2024-04-27T03:19:34.812598+0000","last_clean":"2024-04-27T03:19:34.812598+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:19:34.812598+0000","last_undegraded":"2024-04-27T03:19:34.812598+0000","last_fullsized":"2024-04-27T03:19:34.812598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:13.963398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'282","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.458427+0000","last_change":"2024-04-27T03:17:55.075368+0000","last_active":"2024-04-27T03:21:01.458427+0000","last_peered":"2024-04-27T03:21:01.458427+0000","last_clean":"2024-04-27T03:21:01.458427+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:01.458427+0000","last_undegraded":"2024-04-27T03:21:01.458427+0000","last_fullsized":"2024-04-27T03:21:01.458427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381777+0000","last_change":"2024-04-27T03:17:57.087004+0000","last_active":"2024-04-27T03:19:35.381777+0000","last_peered":"2024-04-27T03:19:35.381777+0000","last_clean":"2024-04-27T03:19:35.381777+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:19:35.381777+0000","last_undegraded":"2024-04-27T03:19:35.381777+0000","last_fullsized":"2024-04-27T03:19:35.381777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:17.983425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.193216+0000","last_change":"2024-04-27T03:17:49.004339+0000","last_active":"2024-04-27T03:20:00.193216+0000","last_peered":"2024-04-27T03:20:00.193216+0000","last_clean":"2024-04-27T03:20:00.193216+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:20:00.193216+0000","last_undegraded":"2024-04-27T03:20:00.193216+0000","last_fullsized":"2024-04-27T03:20:00.193216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:08.912344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.153066+0000","last_change":"2024-04-27T03:17:47.000780+0000","last_active":"2024-04-27T03:20:00.153066+0000","last_peered":"2024-04-27T03:20:00.153066+0000","last_clean":"2024-04-27T03:20:00.153066+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:20:00.153066+0000","last_undegraded":"2024-04-27T03:20:00.153066+0000","last_fullsized":"2024-04-27T03:20:00.153066+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819785+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:19:34.819785+0000","last_peered":"2024-04-27T03:19:34.819785+0000","last_clean":"2024-04-27T03:19:34.819785+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:19:34.819785+0000","last_undegraded":"2024-04-27T03:19:34.819785+0000","last_fullsized":"2024-04-27T03:19:34.819785+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:34.021988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810326+0000","last_change":"2024-04-27T03:17:51.033805+0000","last_active":"2024-04-27T03:19:34.810326+0000","last_peered":"2024-04-27T03:19:34.810326+0000","last_clean":"2024-04-27T03:19:34.810326+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:19:34.810326+0000","last_undegraded":"2024-04-27T03:19:34.810326+0000","last_fullsized":"2024-04-27T03:19:34.810326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'92","reported_seq":174,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.728448+0000","last_change":"2024-04-27T03:17:53.038345+0000","last_active":"2024-04-27T03:21:00.728448+0000","last_peered":"2024-04-27T03:21:00.728448+0000","last_clean":"2024-04-27T03:21:00.728448+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:00.728448+0000","last_undegraded":"2024-04-27T03:21:00.728448+0000","last_fullsized":"2024-04-27T03:21:00.728448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6026,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":86,"num_read_kb":71,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814813+0000","last_change":"2024-04-27T03:17:55.081924+0000","last_active":"2024-04-27T03:19:34.814813+0000","last_peered":"2024-04-27T03:19:34.814813+0000","last_clean":"2024-04-27T03:19:34.814813+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:19:34.814813+0000","last_undegraded":"2024-04-27T03:19:34.814813+0000","last_fullsized":"2024-04-27T03:19:34.814813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439932+0000","last_change":"2024-04-27T03:17:57.068568+0000","last_active":"2024-04-27T03:19:35.439932+0000","last_peered":"2024-04-27T03:19:35.439932+0000","last_clean":"2024-04-27T03:19:35.439932+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:19:35.439932+0000","last_undegraded":"2024-04-27T03:19:35.439932+0000","last_fullsized":"2024-04-27T03:19:35.439932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:43.785644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442918+0000","last_change":"2024-04-27T03:17:57.068516+0000","last_active":"2024-04-27T03:19:35.442918+0000","last_peered":"2024-04-27T03:19:35.442918+0000","last_clean":"2024-04-27T03:19:35.442918+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:19:35.442918+0000","last_undegraded":"2024-04-27T03:19:35.442918+0000","last_fullsized":"2024-04-27T03:19:35.442918+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:58.924125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802035+0000","last_change":"2024-04-27T03:17:49.009366+0000","last_active":"2024-04-27T03:21:00.802035+0000","last_peered":"2024-04-27T03:21:00.802035+0000","last_clean":"2024-04-27T03:21:00.802035+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:00.802035+0000","last_undegraded":"2024-04-27T03:21:00.802035+0000","last_fullsized":"2024-04-27T03:21:00.802035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:58.301334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.124874+0000","last_change":"2024-04-27T03:17:47.023352+0000","last_active":"2024-04-27T03:21:01.124874+0000","last_peered":"2024-04-27T03:21:01.124874+0000","last_clean":"2024-04-27T03:21:01.124874+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:01.124874+0000","last_undegraded":"2024-04-27T03:21:01.124874+0000","last_fullsized":"2024-04-27T03:21:01.124874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"20'1","reported_seq":222,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724858+0000","last_change":"2024-04-27T03:17:51.031849+0000","last_active":"2024-04-27T03:19:59.724858+0000","last_peered":"2024-04-27T03:19:59.724858+0000","last_clean":"2024-04-27T03:19:59.724858+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:19:59.724858+0000","last_undegraded":"2024-04-27T03:19:59.724858+0000","last_fullsized":"2024-04-27T03:19:59.724858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:05.179093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":282,"num_read_kb":282,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'38","reported_seq":91,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.283287+0000","last_change":"2024-04-27T03:17:53.041357+0000","last_active":"2024-04-27T03:19:58.283287+0000","last_peered":"2024-04-27T03:19:58.283287+0000","last_clean":"2024-04-27T03:19:58.283287+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:19:58.283287+0000","last_undegraded":"2024-04-27T03:19:58.283287+0000","last_fullsized":"2024-04-27T03:19:58.283287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:57.603435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809115+0000","last_change":"2024-04-27T03:17:55.074798+0000","last_active":"2024-04-27T03:19:34.809115+0000","last_peered":"2024-04-27T03:19:34.809115+0000","last_clean":"2024-04-27T03:19:34.809115+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:19:34.809115+0000","last_undegraded":"2024-04-27T03:19:34.809115+0000","last_fullsized":"2024-04-27T03:19:34.809115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:15.275458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.885734+0000","last_change":"2024-04-27T03:17:57.087180+0000","last_active":"2024-04-27T03:19:57.885734+0000","last_peered":"2024-04-27T03:19:57.885734+0000","last_clean":"2024-04-27T03:19:57.885734+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:19:57.885734+0000","last_undegraded":"2024-04-27T03:19:57.885734+0000","last_fullsized":"2024-04-27T03:19:57.885734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:50.266677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146716+0000","last_change":"2024-04-27T03:17:49.005188+0000","last_active":"2024-04-27T03:21:01.146716+0000","last_peered":"2024-04-27T03:21:01.146716+0000","last_clean":"2024-04-27T03:21:01.146716+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:01.146716+0000","last_undegraded":"2024-04-27T03:21:01.146716+0000","last_fullsized":"2024-04-27T03:21:01.146716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:24.812863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440214+0000","last_change":"2024-04-27T03:17:47.022474+0000","last_active":"2024-04-27T03:19:35.440214+0000","last_peered":"2024-04-27T03:19:35.440214+0000","last_clean":"2024-04-27T03:19:35.440214+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:19:35.440214+0000","last_undegraded":"2024-04-27T03:19:35.440214+0000","last_fullsized":"2024-04-27T03:19:35.440214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:13.293355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811181+0000","last_change":"2024-04-27T03:17:51.024948+0000","last_active":"2024-04-27T03:19:34.811181+0000","last_peered":"2024-04-27T03:19:34.811181+0000","last_clean":"2024-04-27T03:19:34.811181+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:19:34.811181+0000","last_undegraded":"2024-04-27T03:19:34.811181+0000","last_fullsized":"2024-04-27T03:19:34.811181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.271365+0000","last_change":"2024-04-27T03:17:53.053658+0000","last_active":"2024-04-27T03:21:00.271365+0000","last_peered":"2024-04-27T03:21:00.271365+0000","last_clean":"2024-04-27T03:21:00.271365+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:00.271365+0000","last_undegraded":"2024-04-27T03:21:00.271365+0000","last_fullsized":"2024-04-27T03:21:00.271365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:12.928546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438320+0000","last_change":"2024-04-27T03:17:55.051277+0000","last_active":"2024-04-27T03:19:35.438320+0000","last_peered":"2024-04-27T03:19:35.438320+0000","last_clean":"2024-04-27T03:19:35.438320+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:19:35.438320+0000","last_undegraded":"2024-04-27T03:19:35.438320+0000","last_fullsized":"2024-04-27T03:19:35.438320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:49.716708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382470+0000","last_change":"2024-04-27T03:17:57.089220+0000","last_active":"2024-04-27T03:19:35.382470+0000","last_peered":"2024-04-27T03:19:35.382470+0000","last_clean":"2024-04-27T03:19:35.382470+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:19:35.382470+0000","last_undegraded":"2024-04-27T03:19:35.382470+0000","last_fullsized":"2024-04-27T03:19:35.382470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:17.289812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949086+0000","last_change":"2024-04-27T03:18:34.799245+0000","last_active":"2024-04-27T03:21:00.949086+0000","last_peered":"2024-04-27T03:21:00.949086+0000","last_clean":"2024-04-27T03:21:00.949086+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:21:00.949086+0000","last_undegraded":"2024-04-27T03:21:00.949086+0000","last_fullsized":"2024-04-27T03:21:00.949086+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:17.961788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.107590+0000","last_change":"2024-04-27T03:17:47.012224+0000","last_active":"2024-04-27T03:21:01.107590+0000","last_peered":"2024-04-27T03:21:01.107590+0000","last_clean":"2024-04-27T03:21:01.107590+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:21:01.107590+0000","last_undegraded":"2024-04-27T03:21:01.107590+0000","last_fullsized":"2024-04-27T03:21:01.107590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813595+0000","last_change":"2024-04-27T03:17:51.037763+0000","last_active":"2024-04-27T03:19:34.813595+0000","last_peered":"2024-04-27T03:19:34.813595+0000","last_clean":"2024-04-27T03:19:34.813595+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:19:34.813595+0000","last_undegraded":"2024-04-27T03:19:34.813595+0000","last_fullsized":"2024-04-27T03:19:34.813595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:19.793632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'40","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.321256+0000","last_change":"2024-04-27T03:17:53.047021+0000","last_active":"2024-04-27T03:21:00.321256+0000","last_peered":"2024-04-27T03:21:00.321256+0000","last_clean":"2024-04-27T03:21:00.321256+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:00.321256+0000","last_undegraded":"2024-04-27T03:21:00.321256+0000","last_fullsized":"2024-04-27T03:21:00.321256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":61,"num_read_kb":64,"num_write":46,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379567+0000","last_change":"2024-04-27T03:17:55.075762+0000","last_active":"2024-04-27T03:19:35.379567+0000","last_peered":"2024-04-27T03:19:35.379567+0000","last_clean":"2024-04-27T03:19:35.379567+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:19:35.379567+0000","last_undegraded":"2024-04-27T03:19:35.379567+0000","last_fullsized":"2024-04-27T03:19:35.379567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:49:01.187937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812651+0000","last_change":"2024-04-27T03:17:57.086967+0000","last_active":"2024-04-27T03:19:34.812651+0000","last_peered":"2024-04-27T03:19:34.812651+0000","last_clean":"2024-04-27T03:19:34.812651+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:19:34.812651+0000","last_undegraded":"2024-04-27T03:19:34.812651+0000","last_fullsized":"2024-04-27T03:19:34.812651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:54.374770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.234017+0000","last_change":"2024-04-27T03:17:49.014835+0000","last_active":"2024-04-27T03:21:01.234017+0000","last_peered":"2024-04-27T03:21:01.234017+0000","last_clean":"2024-04-27T03:21:01.234017+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:01.234017+0000","last_undegraded":"2024-04-27T03:21:01.234017+0000","last_fullsized":"2024-04-27T03:21:01.234017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385250+0000","last_change":"2024-04-27T03:17:47.026622+0000","last_active":"2024-04-27T03:19:35.385250+0000","last_peered":"2024-04-27T03:19:35.385250+0000","last_clean":"2024-04-27T03:19:35.385250+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:19:35.385250+0000","last_undegraded":"2024-04-27T03:19:35.385250+0000","last_fullsized":"2024-04-27T03:19:35.385250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809701+0000","last_change":"2024-04-27T03:17:51.014069+0000","last_active":"2024-04-27T03:19:34.809701+0000","last_peered":"2024-04-27T03:19:34.809701+0000","last_clean":"2024-04-27T03:19:34.809701+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:19:34.809701+0000","last_undegraded":"2024-04-27T03:19:34.809701+0000","last_fullsized":"2024-04-27T03:19:34.809701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'31","reported_seq":77,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745864+0000","last_change":"2024-04-27T03:17:53.038792+0000","last_active":"2024-04-27T03:20:12.745864+0000","last_peered":"2024-04-27T03:20:12.745864+0000","last_clean":"2024-04-27T03:20:12.745864+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:20:12.745864+0000","last_undegraded":"2024-04-27T03:20:12.745864+0000","last_fullsized":"2024-04-27T03:20:12.745864+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438653+0000","last_change":"2024-04-27T03:17:55.055483+0000","last_active":"2024-04-27T03:19:35.438653+0000","last_peered":"2024-04-27T03:19:35.438653+0000","last_clean":"2024-04-27T03:19:35.438653+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:19:35.438653+0000","last_undegraded":"2024-04-27T03:19:35.438653+0000","last_fullsized":"2024-04-27T03:19:35.438653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:17.695216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'1","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742486+0000","last_change":"2024-04-27T03:17:49.022141+0000","last_active":"2024-04-27T03:19:57.742486+0000","last_peered":"2024-04-27T03:19:57.742486+0000","last_clean":"2024-04-27T03:19:57.742486+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:19:57.742486+0000","last_undegraded":"2024-04-27T03:19:57.742486+0000","last_fullsized":"2024-04-27T03:19:57.742486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.117502+0000","last_change":"2024-04-27T03:17:47.032082+0000","last_active":"2024-04-27T03:21:01.117502+0000","last_peered":"2024-04-27T03:21:01.117502+0000","last_clean":"2024-04-27T03:21:01.117502+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:21:01.117502+0000","last_undegraded":"2024-04-27T03:21:01.117502+0000","last_fullsized":"2024-04-27T03:21:01.117502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'4","reported_seq":230,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.696608+0000","last_change":"2024-04-27T03:17:51.024084+0000","last_active":"2024-04-27T03:19:59.696608+0000","last_peered":"2024-04-27T03:19:59.696608+0000","last_clean":"2024-04-27T03:19:59.696608+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:19:59.696608+0000","last_undegraded":"2024-04-27T03:19:59.696608+0000","last_fullsized":"2024-04-27T03:19:59.696608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:37.745198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":298,"num_read_kb":298,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'62","reported_seq":128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.293689+0000","last_change":"2024-04-27T03:17:53.035870+0000","last_active":"2024-04-27T03:19:58.293689+0000","last_peered":"2024-04-27T03:19:58.293689+0000","last_clean":"2024-04-27T03:19:58.293689+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:19:58.293689+0000","last_undegraded":"2024-04-27T03:19:58.293689+0000","last_fullsized":"2024-04-27T03:19:58.293689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:10.295737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":24,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439324+0000","last_change":"2024-04-27T03:17:55.075718+0000","last_active":"2024-04-27T03:19:35.439324+0000","last_peered":"2024-04-27T03:19:35.439324+0000","last_clean":"2024-04-27T03:19:35.439324+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:19:35.439324+0000","last_undegraded":"2024-04-27T03:19:35.439324+0000","last_fullsized":"2024-04-27T03:19:35.439324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:58.929938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.074643+0000","last_change":"2024-04-27T03:17:57.070510+0000","last_active":"2024-04-27T03:20:00.074643+0000","last_peered":"2024-04-27T03:20:00.074643+0000","last_clean":"2024-04-27T03:20:00.074643+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:20:00.074643+0000","last_undegraded":"2024-04-27T03:20:00.074643+0000","last_fullsized":"2024-04-27T03:20:00.074643+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111625+0000","last_change":"2024-04-27T03:17:49.000295+0000","last_active":"2024-04-27T03:19:56.111625+0000","last_peered":"2024-04-27T03:19:56.111625+0000","last_clean":"2024-04-27T03:19:56.111625+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:19:56.111625+0000","last_undegraded":"2024-04-27T03:19:56.111625+0000","last_fullsized":"2024-04-27T03:19:56.111625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379919+0000","last_change":"2024-04-27T03:17:47.021864+0000","last_active":"2024-04-27T03:19:35.379919+0000","last_peered":"2024-04-27T03:19:35.379919+0000","last_clean":"2024-04-27T03:19:35.379919+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:19:35.379919+0000","last_undegraded":"2024-04-27T03:19:35.379919+0000","last_fullsized":"2024-04-27T03:19:35.379919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:14.918204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820383+0000","last_change":"2024-04-27T03:17:51.033166+0000","last_active":"2024-04-27T03:19:34.820383+0000","last_peered":"2024-04-27T03:19:34.820383+0000","last_clean":"2024-04-27T03:19:34.820383+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:19:34.820383+0000","last_undegraded":"2024-04-27T03:19:34.820383+0000","last_fullsized":"2024-04-27T03:19:34.820383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:40.351925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.260042+0000","last_change":"2024-04-27T03:17:53.038085+0000","last_active":"2024-04-27T03:19:58.260042+0000","last_peered":"2024-04-27T03:19:58.260042+0000","last_clean":"2024-04-27T03:19:58.260042+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:19:58.260042+0000","last_undegraded":"2024-04-27T03:19:58.260042+0000","last_fullsized":"2024-04-27T03:19:58.260042+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.383103+0000","last_change":"2024-04-27T03:17:55.069433+0000","last_active":"2024-04-27T03:19:35.383103+0000","last_peered":"2024-04-27T03:19:35.383103+0000","last_clean":"2024-04-27T03:19:35.383103+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:19:35.383103+0000","last_undegraded":"2024-04-27T03:19:35.383103+0000","last_fullsized":"2024-04-27T03:19:35.383103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:17.345394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814942+0000","last_change":"2024-04-27T03:17:57.070067+0000","last_active":"2024-04-27T03:19:34.814942+0000","last_peered":"2024-04-27T03:19:34.814942+0000","last_clean":"2024-04-27T03:19:34.814942+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:19:34.814942+0000","last_undegraded":"2024-04-27T03:19:34.814942+0000","last_fullsized":"2024-04-27T03:19:34.814942+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814892+0000","last_change":"2024-04-27T03:17:49.004464+0000","last_active":"2024-04-27T03:19:34.814892+0000","last_peered":"2024-04-27T03:19:34.814892+0000","last_clean":"2024-04-27T03:19:34.814892+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:19:34.814892+0000","last_undegraded":"2024-04-27T03:19:34.814892+0000","last_fullsized":"2024-04-27T03:19:34.814892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.095190+0000","last_change":"2024-04-27T03:17:47.027156+0000","last_active":"2024-04-27T03:19:56.095190+0000","last_peered":"2024-04-27T03:19:56.095190+0000","last_clean":"2024-04-27T03:19:56.095190+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:19:56.095190+0000","last_undegraded":"2024-04-27T03:19:56.095190+0000","last_fullsized":"2024-04-27T03:19:56.095190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816982+0000","last_change":"2024-04-27T03:17:51.021222+0000","last_active":"2024-04-27T03:19:34.816982+0000","last_peered":"2024-04-27T03:19:34.816982+0000","last_clean":"2024-04-27T03:19:34.816982+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:19:34.816982+0000","last_undegraded":"2024-04-27T03:19:34.816982+0000","last_fullsized":"2024-04-27T03:19:34.816982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'39","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.872099+0000","last_change":"2024-04-27T03:17:53.041372+0000","last_active":"2024-04-27T03:19:57.872099+0000","last_peered":"2024-04-27T03:19:57.872099+0000","last_clean":"2024-04-27T03:19:57.872099+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:19:57.872099+0000","last_undegraded":"2024-04-27T03:19:57.872099+0000","last_fullsized":"2024-04-27T03:19:57.872099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384616+0000","last_change":"2024-04-27T03:17:55.077381+0000","last_active":"2024-04-27T03:19:35.384616+0000","last_peered":"2024-04-27T03:19:35.384616+0000","last_clean":"2024-04-27T03:19:35.384616+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:19:35.384616+0000","last_undegraded":"2024-04-27T03:19:35.384616+0000","last_fullsized":"2024-04-27T03:19:35.384616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:08:00.332438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.205156+0000","last_change":"2024-04-27T03:17:57.066218+0000","last_active":"2024-04-27T03:20:00.205156+0000","last_peered":"2024-04-27T03:20:00.205156+0000","last_clean":"2024-04-27T03:20:00.205156+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:20:00.205156+0000","last_undegraded":"2024-04-27T03:20:00.205156+0000","last_fullsized":"2024-04-27T03:20:00.205156+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.739365+0000","last_change":"2024-04-27T03:17:49.019967+0000","last_active":"2024-04-27T03:20:22.739365+0000","last_peered":"2024-04-27T03:20:22.739365+0000","last_clean":"2024-04-27T03:20:22.739365+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:20:22.739365+0000","last_undegraded":"2024-04-27T03:20:22.739365+0000","last_fullsized":"2024-04-27T03:20:22.739365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:45.458581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812087+0000","last_change":"2024-04-27T03:17:47.018817+0000","last_active":"2024-04-27T03:19:34.812087+0000","last_peered":"2024-04-27T03:19:34.812087+0000","last_clean":"2024-04-27T03:19:34.812087+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:19:34.812087+0000","last_undegraded":"2024-04-27T03:19:34.812087+0000","last_fullsized":"2024-04-27T03:19:34.812087+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377658+0000","last_change":"2024-04-27T03:17:51.030031+0000","last_active":"2024-04-27T03:19:35.377658+0000","last_peered":"2024-04-27T03:19:35.377658+0000","last_clean":"2024-04-27T03:19:35.377658+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:19:35.377658+0000","last_undegraded":"2024-04-27T03:19:35.377658+0000","last_fullsized":"2024-04-27T03:19:35.377658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:08.238587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814172+0000","last_change":"2024-04-27T03:17:53.046967+0000","last_active":"2024-04-27T03:19:34.814172+0000","last_peered":"2024-04-27T03:19:34.814172+0000","last_clean":"2024-04-27T03:19:34.814172+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:19:34.814172+0000","last_undegraded":"2024-04-27T03:19:34.814172+0000","last_fullsized":"2024-04-27T03:19:34.814172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'282","reported_seq":380,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277975+0000","last_change":"2024-04-27T03:17:55.060259+0000","last_active":"2024-04-27T03:21:01.277975+0000","last_peered":"2024-04-27T03:21:01.277975+0000","last_clean":"2024-04-27T03:21:01.277975+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:01.277975+0000","last_undegraded":"2024-04-27T03:21:01.277975+0000","last_fullsized":"2024-04-27T03:21:01.277975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:04:03.670215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813560+0000","last_change":"2024-04-27T03:17:57.084816+0000","last_active":"2024-04-27T03:19:34.813560+0000","last_peered":"2024-04-27T03:19:34.813560+0000","last_clean":"2024-04-27T03:19:34.813560+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:19:34.813560+0000","last_undegraded":"2024-04-27T03:19:34.813560+0000","last_fullsized":"2024-04-27T03:19:34.813560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:03.843907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248654+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.248654+0000","last_peered":"2024-04-27T03:21:01.248654+0000","last_clean":"2024-04-27T03:21:01.248654+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:01.248654+0000","last_undegraded":"2024-04-27T03:21:01.248654+0000","last_fullsized":"2024-04-27T03:21:01.248654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:22.252077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382608+0000","last_change":"2024-04-27T03:17:47.027621+0000","last_active":"2024-04-27T03:19:35.382608+0000","last_peered":"2024-04-27T03:19:35.382608+0000","last_clean":"2024-04-27T03:19:35.382608+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:19:35.382608+0000","last_undegraded":"2024-04-27T03:19:35.382608+0000","last_fullsized":"2024-04-27T03:19:35.382608+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377762+0000","last_change":"2024-04-27T03:17:51.032091+0000","last_active":"2024-04-27T03:19:35.377762+0000","last_peered":"2024-04-27T03:19:35.377762+0000","last_clean":"2024-04-27T03:19:35.377762+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:19:35.377762+0000","last_undegraded":"2024-04-27T03:19:35.377762+0000","last_fullsized":"2024-04-27T03:19:35.377762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:06.149814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'60","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.243349+0000","last_change":"2024-04-27T03:17:53.039949+0000","last_active":"2024-04-27T03:19:58.243349+0000","last_peered":"2024-04-27T03:19:58.243349+0000","last_clean":"2024-04-27T03:19:58.243349+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:19:58.243349+0000","last_undegraded":"2024-04-27T03:19:58.243349+0000","last_fullsized":"2024-04-27T03:19:58.243349+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.808839+0000","last_change":"2024-04-27T03:17:55.071558+0000","last_active":"2024-04-27T03:19:34.808839+0000","last_peered":"2024-04-27T03:19:34.808839+0000","last_clean":"2024-04-27T03:19:34.808839+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:19:34.808839+0000","last_undegraded":"2024-04-27T03:19:34.808839+0000","last_fullsized":"2024-04-27T03:19:34.808839+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:12.887800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.348234+0000","last_change":"2024-04-27T03:17:57.066613+0000","last_active":"2024-04-27T03:21:00.348234+0000","last_peered":"2024-04-27T03:21:00.348234+0000","last_clean":"2024-04-27T03:21:00.348234+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:21:00.348234+0000","last_undegraded":"2024-04-27T03:21:00.348234+0000","last_fullsized":"2024-04-27T03:21:00.348234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949296+0000","last_change":"2024-04-27T03:17:49.017718+0000","last_active":"2024-04-27T03:21:00.949296+0000","last_peered":"2024-04-27T03:21:00.949296+0000","last_clean":"2024-04-27T03:21:00.949296+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:00.949296+0000","last_undegraded":"2024-04-27T03:21:00.949296+0000","last_fullsized":"2024-04-27T03:21:00.949296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:40.378270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819810+0000","last_change":"2024-04-27T03:17:47.022941+0000","last_active":"2024-04-27T03:19:34.819810+0000","last_peered":"2024-04-27T03:19:34.819810+0000","last_clean":"2024-04-27T03:19:34.819810+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:19:34.819810+0000","last_undegraded":"2024-04-27T03:19:34.819810+0000","last_fullsized":"2024-04-27T03:19:34.819810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:09.787441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816507+0000","last_change":"2024-04-27T03:17:51.033578+0000","last_active":"2024-04-27T03:19:34.816507+0000","last_peered":"2024-04-27T03:19:34.816507+0000","last_clean":"2024-04-27T03:19:34.816507+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:19:34.816507+0000","last_undegraded":"2024-04-27T03:19:34.816507+0000","last_fullsized":"2024-04-27T03:19:34.816507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'38","reported_seq":90,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.244009+0000","last_change":"2024-04-27T03:17:53.040115+0000","last_active":"2024-04-27T03:19:58.244009+0000","last_peered":"2024-04-27T03:19:58.244009+0000","last_clean":"2024-04-27T03:19:58.244009+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:19:58.244009+0000","last_undegraded":"2024-04-27T03:19:58.244009+0000","last_fullsized":"2024-04-27T03:19:58.244009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:28.843466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815617+0000","last_change":"2024-04-27T03:17:55.058615+0000","last_active":"2024-04-27T03:19:34.815617+0000","last_peered":"2024-04-27T03:19:34.815617+0000","last_clean":"2024-04-27T03:19:34.815617+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:19:34.815617+0000","last_undegraded":"2024-04-27T03:19:34.815617+0000","last_fullsized":"2024-04-27T03:19:34.815617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.401928+0000","last_change":"2024-04-27T03:17:57.087706+0000","last_active":"2024-04-27T03:21:01.401928+0000","last_peered":"2024-04-27T03:21:01.401928+0000","last_clean":"2024-04-27T03:21:01.401928+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:21:01.401928+0000","last_undegraded":"2024-04-27T03:21:01.401928+0000","last_fullsized":"2024-04-27T03:21:01.401928+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:57.341894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950567+0000","last_change":"2024-04-27T03:17:49.011902+0000","last_active":"2024-04-27T03:21:00.950567+0000","last_peered":"2024-04-27T03:21:00.950567+0000","last_clean":"2024-04-27T03:21:00.950567+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:21:00.950567+0000","last_undegraded":"2024-04-27T03:21:00.950567+0000","last_fullsized":"2024-04-27T03:21:00.950567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.735456+0000","last_change":"2024-04-27T03:17:47.016344+0000","last_active":"2024-04-27T03:19:47.735456+0000","last_peered":"2024-04-27T03:19:47.735456+0000","last_clean":"2024-04-27T03:19:47.735456+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:19:47.735456+0000","last_undegraded":"2024-04-27T03:19:47.735456+0000","last_fullsized":"2024-04-27T03:19:47.735456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377191+0000","last_change":"2024-04-27T03:17:51.029804+0000","last_active":"2024-04-27T03:19:35.377191+0000","last_peered":"2024-04-27T03:19:35.377191+0000","last_clean":"2024-04-27T03:19:35.377191+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:19:35.377191+0000","last_undegraded":"2024-04-27T03:19:35.377191+0000","last_fullsized":"2024-04-27T03:19:35.377191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:13.111551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'32","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.286319+0000","last_change":"2024-04-27T03:17:53.041146+0000","last_active":"2024-04-27T03:19:58.286319+0000","last_peered":"2024-04-27T03:19:58.286319+0000","last_clean":"2024-04-27T03:19:58.286319+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:19:58.286319+0000","last_undegraded":"2024-04-27T03:19:58.286319+0000","last_fullsized":"2024-04-27T03:19:58.286319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:15.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816372+0000","last_change":"2024-04-27T03:17:55.071034+0000","last_active":"2024-04-27T03:19:34.816372+0000","last_peered":"2024-04-27T03:19:34.816372+0000","last_clean":"2024-04-27T03:19:34.816372+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:19:34.816372+0000","last_undegraded":"2024-04-27T03:19:34.816372+0000","last_fullsized":"2024-04-27T03:19:34.816372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.302058+0000","last_change":"2024-04-27T03:17:57.086785+0000","last_active":"2024-04-27T03:19:58.302058+0000","last_peered":"2024-04-27T03:19:58.302058+0000","last_clean":"2024-04-27T03:19:58.302058+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:19:58.302058+0000","last_undegraded":"2024-04-27T03:19:58.302058+0000","last_fullsized":"2024-04-27T03:19:58.302058+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:25:08.779674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397096+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.397096+0000","last_peered":"2024-04-27T03:21:01.397096+0000","last_clean":"2024-04-27T03:21:01.397096+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:01.397096+0000","last_undegraded":"2024-04-27T03:21:01.397096+0000","last_fullsized":"2024-04-27T03:21:01.397096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.019895+0000","last_change":"2024-04-27T03:17:47.002631+0000","last_active":"2024-04-27T03:20:00.019895+0000","last_peered":"2024-04-27T03:20:00.019895+0000","last_clean":"2024-04-27T03:20:00.019895+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:20:00.019895+0000","last_undegraded":"2024-04-27T03:20:00.019895+0000","last_fullsized":"2024-04-27T03:20:00.019895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'2","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.673843+0000","last_change":"2024-04-27T03:18:34.794992+0000","last_active":"2024-04-27T03:19:56.673843+0000","last_peered":"2024-04-27T03:19:56.673843+0000","last_clean":"2024-04-27T03:19:56.673843+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:19:56.673843+0000","last_undegraded":"2024-04-27T03:19:56.673843+0000","last_fullsized":"2024-04-27T03:19:56.673843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:56:59.713686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'44","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.246215+0000","last_change":"2024-04-27T03:17:53.039095+0000","last_active":"2024-04-27T03:21:00.246215+0000","last_peered":"2024-04-27T03:21:00.246215+0000","last_clean":"2024-04-27T03:21:00.246215+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:00.246215+0000","last_undegraded":"2024-04-27T03:21:00.246215+0000","last_fullsized":"2024-04-27T03:21:00.246215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":30,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438638+0000","last_change":"2024-04-27T03:17:55.063864+0000","last_active":"2024-04-27T03:19:35.438638+0000","last_peered":"2024-04-27T03:19:35.438638+0000","last_clean":"2024-04-27T03:19:35.438638+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:19:35.438638+0000","last_undegraded":"2024-04-27T03:19:35.438638+0000","last_fullsized":"2024-04-27T03:19:35.438638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:46.421450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.307196+0000","last_change":"2024-04-27T03:17:57.061982+0000","last_active":"2024-04-27T03:19:58.307196+0000","last_peered":"2024-04-27T03:19:58.307196+0000","last_clean":"2024-04-27T03:19:58.307196+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:19:58.307196+0000","last_undegraded":"2024-04-27T03:19:58.307196+0000","last_fullsized":"2024-04-27T03:19:58.307196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386099+0000","last_change":"2024-04-27T03:17:49.001808+0000","last_active":"2024-04-27T03:19:35.386099+0000","last_peered":"2024-04-27T03:19:35.386099+0000","last_clean":"2024-04-27T03:19:35.386099+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:19:35.386099+0000","last_undegraded":"2024-04-27T03:19:35.386099+0000","last_fullsized":"2024-04-27T03:19:35.386099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:33.807122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436337+0000","last_change":"2024-04-27T03:17:47.032632+0000","last_active":"2024-04-27T03:19:35.436337+0000","last_peered":"2024-04-27T03:19:35.436337+0000","last_clean":"2024-04-27T03:19:35.436337+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:19:35.436337+0000","last_undegraded":"2024-04-27T03:19:35.436337+0000","last_fullsized":"2024-04-27T03:19:35.436337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:53:43.154787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385434+0000","last_change":"2024-04-27T03:17:49.011651+0000","last_active":"2024-04-27T03:19:35.385434+0000","last_peered":"2024-04-27T03:19:35.385434+0000","last_clean":"2024-04-27T03:19:35.385434+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:19:35.385434+0000","last_undegraded":"2024-04-27T03:19:35.385434+0000","last_fullsized":"2024-04-27T03:19:35.385434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'1","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.732801+0000","last_change":"2024-04-27T03:18:34.773740+0000","last_active":"2024-04-27T03:19:52.732801+0000","last_peered":"2024-04-27T03:19:52.732801+0000","last_clean":"2024-04-27T03:19:52.732801+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:19:52.732801+0000","last_undegraded":"2024-04-27T03:19:52.732801+0000","last_fullsized":"2024-04-27T03:19:52.732801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:04.248651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639872,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111155+0000","last_change":"2024-04-27T03:18:34.784905+0000","last_active":"2024-04-27T03:19:56.111155+0000","last_peered":"2024-04-27T03:19:56.111155+0000","last_clean":"2024-04-27T03:19:56.111155+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:19:56.111155+0000","last_undegraded":"2024-04-27T03:19:56.111155+0000","last_fullsized":"2024-04-27T03:19:56.111155+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:28.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810481+0000","last_change":"2024-04-27T03:17:47.036148+0000","last_active":"2024-04-27T03:19:34.810481+0000","last_peered":"2024-04-27T03:19:34.810481+0000","last_clean":"2024-04-27T03:19:34.810481+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:19:34.810481+0000","last_undegraded":"2024-04-27T03:19:34.810481+0000","last_fullsized":"2024-04-27T03:19:34.810481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382422+0000","last_change":"2024-04-27T03:17:49.017582+0000","last_active":"2024-04-27T03:19:35.382422+0000","last_peered":"2024-04-27T03:19:35.382422+0000","last_clean":"2024-04-27T03:19:35.382422+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:19:35.382422+0000","last_undegraded":"2024-04-27T03:19:35.382422+0000","last_fullsized":"2024-04-27T03:19:35.382422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:20.541354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438828+0000","last_change":"2024-04-27T03:17:47.016123+0000","last_active":"2024-04-27T03:19:35.438828+0000","last_peered":"2024-04-27T03:19:35.438828+0000","last_clean":"2024-04-27T03:19:35.438828+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:19:35.438828+0000","last_undegraded":"2024-04-27T03:19:35.438828+0000","last_fullsized":"2024-04-27T03:19:35.438828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:59.109823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300171+0000","last_change":"2024-04-27T03:17:49.021184+0000","last_active":"2024-04-27T03:21:01.300171+0000","last_peered":"2024-04-27T03:21:01.300171+0000","last_clean":"2024-04-27T03:21:01.300171+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:01.300171+0000","last_undegraded":"2024-04-27T03:21:01.300171+0000","last_fullsized":"2024-04-27T03:21:01.300171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811874+0000","last_change":"2024-04-27T03:17:47.039423+0000","last_active":"2024-04-27T03:19:34.811874+0000","last_peered":"2024-04-27T03:19:34.811874+0000","last_clean":"2024-04-27T03:19:34.811874+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:19:34.811874+0000","last_undegraded":"2024-04-27T03:19:34.811874+0000","last_fullsized":"2024-04-27T03:19:34.811874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381685+0000","last_change":"2024-04-27T03:17:49.006801+0000","last_active":"2024-04-27T03:19:35.381685+0000","last_peered":"2024-04-27T03:19:35.381685+0000","last_clean":"2024-04-27T03:19:35.381685+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:19:35.381685+0000","last_undegraded":"2024-04-27T03:19:35.381685+0000","last_fullsized":"2024-04-27T03:19:35.381685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:39:13.312552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.947064+0000","last_change":"2024-04-27T03:17:47.030288+0000","last_active":"2024-04-27T03:19:59.947064+0000","last_peered":"2024-04-27T03:19:59.947064+0000","last_clean":"2024-04-27T03:19:59.947064+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:19:59.947064+0000","last_undegraded":"2024-04-27T03:19:59.947064+0000","last_fullsized":"2024-04-27T03:19:59.947064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:52:43.758948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300194+0000","last_change":"2024-04-27T03:17:49.015796+0000","last_active":"2024-04-27T03:21:01.300194+0000","last_peered":"2024-04-27T03:21:01.300194+0000","last_clean":"2024-04-27T03:21:01.300194+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:01.300194+0000","last_undegraded":"2024-04-27T03:21:01.300194+0000","last_fullsized":"2024-04-27T03:21:01.300194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.756327+0000","last_change":"2024-04-27T03:17:47.031857+0000","last_active":"2024-04-27T03:21:00.756327+0000","last_peered":"2024-04-27T03:21:00.756327+0000","last_clean":"2024-04-27T03:21:00.756327+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:00.756327+0000","last_undegraded":"2024-04-27T03:21:00.756327+0000","last_fullsized":"2024-04-27T03:21:00.756327+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":78,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949139+0000","last_change":"2024-04-27T03:17:49.017220+0000","last_active":"2024-04-27T03:21:00.949139+0000","last_peered":"2024-04-27T03:21:00.949139+0000","last_clean":"2024-04-27T03:21:00.949139+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:00.949139+0000","last_undegraded":"2024-04-27T03:21:00.949139+0000","last_fullsized":"2024-04-27T03:21:00.949139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815261+0000","last_change":"2024-04-27T03:17:47.039005+0000","last_active":"2024-04-27T03:19:34.815261+0000","last_peered":"2024-04-27T03:19:34.815261+0000","last_clean":"2024-04-27T03:19:34.815261+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:19:34.815261+0000","last_undegraded":"2024-04-27T03:19:34.815261+0000","last_fullsized":"2024-04-27T03:19:34.815261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299434+0000","last_change":"2024-04-27T03:17:49.011708+0000","last_active":"2024-04-27T03:21:01.299434+0000","last_peered":"2024-04-27T03:21:01.299434+0000","last_clean":"2024-04-27T03:21:01.299434+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:01.299434+0000","last_undegraded":"2024-04-27T03:21:01.299434+0000","last_fullsized":"2024-04-27T03:21:01.299434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:52.139159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809399+0000","last_change":"2024-04-27T03:17:47.016024+0000","last_active":"2024-04-27T03:19:34.809399+0000","last_peered":"2024-04-27T03:19:34.809399+0000","last_clean":"2024-04-27T03:19:34.809399+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:19:34.809399+0000","last_undegraded":"2024-04-27T03:19:34.809399+0000","last_fullsized":"2024-04-27T03:19:34.809399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300134+0000","last_change":"2024-04-27T03:17:49.010865+0000","last_active":"2024-04-27T03:21:01.300134+0000","last_peered":"2024-04-27T03:21:01.300134+0000","last_clean":"2024-04-27T03:21:01.300134+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:21:01.300134+0000","last_undegraded":"2024-04-27T03:21:01.300134+0000","last_fullsized":"2024-04-27T03:21:01.300134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810923+0000","last_change":"2024-04-27T03:17:47.019986+0000","last_active":"2024-04-27T03:19:34.810923+0000","last_peered":"2024-04-27T03:19:34.810923+0000","last_clean":"2024-04-27T03:19:34.810923+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:19:34.810923+0000","last_undegraded":"2024-04-27T03:19:34.810923+0000","last_fullsized":"2024-04-27T03:19:34.810923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949386+0000","last_change":"2024-04-27T03:17:49.005667+0000","last_active":"2024-04-27T03:21:00.949386+0000","last_peered":"2024-04-27T03:21:00.949386+0000","last_clean":"2024-04-27T03:21:00.949386+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:00.949386+0000","last_undegraded":"2024-04-27T03:21:00.949386+0000","last_fullsized":"2024-04-27T03:21:00.949386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:07.432944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817074+0000","last_change":"2024-04-27T03:17:47.023505+0000","last_active":"2024-04-27T03:19:34.817074+0000","last_peered":"2024-04-27T03:19:34.817074+0000","last_clean":"2024-04-27T03:19:34.817074+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:19:34.817074+0000","last_undegraded":"2024-04-27T03:19:34.817074+0000","last_fullsized":"2024-04-27T03:19:34.817074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382540+0000","last_change":"2024-04-27T03:17:49.009462+0000","last_active":"2024-04-27T03:19:35.382540+0000","last_peered":"2024-04-27T03:19:35.382540+0000","last_clean":"2024-04-27T03:19:35.382540+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:19:35.382540+0000","last_undegraded":"2024-04-27T03:19:35.382540+0000","last_fullsized":"2024-04-27T03:19:35.382540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:32.201166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382267+0000","last_change":"2024-04-27T03:17:47.015754+0000","last_active":"2024-04-27T03:19:35.382267+0000","last_peered":"2024-04-27T03:19:35.382267+0000","last_clean":"2024-04-27T03:19:35.382267+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:19:35.382267+0000","last_undegraded":"2024-04-27T03:19:35.382267+0000","last_fullsized":"2024-04-27T03:19:35.382267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'7","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744638+0000","last_change":"2024-04-27T03:17:49.017194+0000","last_active":"2024-04-27T03:20:07.744638+0000","last_peered":"2024-04-27T03:20:07.744638+0000","last_clean":"2024-04-27T03:20:07.744638+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:20:07.744638+0000","last_undegraded":"2024-04-27T03:20:07.744638+0000","last_fullsized":"2024-04-27T03:20:07.744638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812147+0000","last_change":"2024-04-27T03:17:47.038806+0000","last_active":"2024-04-27T03:19:34.812147+0000","last_peered":"2024-04-27T03:19:34.812147+0000","last_clean":"2024-04-27T03:19:34.812147+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:19:34.812147+0000","last_undegraded":"2024-04-27T03:19:34.812147+0000","last_fullsized":"2024-04-27T03:19:34.812147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.808067+0000","last_change":"2024-04-27T03:17:49.006535+0000","last_active":"2024-04-27T03:21:00.808067+0000","last_peered":"2024-04-27T03:21:00.808067+0000","last_clean":"2024-04-27T03:21:00.808067+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:00.808067+0000","last_undegraded":"2024-04-27T03:21:00.808067+0000","last_fullsized":"2024-04-27T03:21:00.808067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:21.664514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442247+0000","last_change":"2024-04-27T03:19:35.441020+0000","last_active":"2024-04-27T03:19:35.442247+0000","last_peered":"2024-04-27T03:19:35.442247+0000","last_clean":"2024-04-27T03:19:35.442247+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:19:35.442247+0000","last_undegraded":"2024-04-27T03:19:35.442247+0000","last_fullsized":"2024-04-27T03:19:35.442247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:39:26.708815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814170+0000","last_change":"2024-04-27T03:17:49.016545+0000","last_active":"2024-04-27T03:19:34.814170+0000","last_peered":"2024-04-27T03:19:34.814170+0000","last_clean":"2024-04-27T03:19:34.814170+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:19:34.814170+0000","last_undegraded":"2024-04-27T03:19:34.814170+0000","last_fullsized":"2024-04-27T03:19:34.814170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:52.886758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442831+0000","last_change":"2024-04-27T03:17:47.022441+0000","last_active":"2024-04-27T03:19:35.442831+0000","last_peered":"2024-04-27T03:19:35.442831+0000","last_clean":"2024-04-27T03:19:35.442831+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:19:35.442831+0000","last_undegraded":"2024-04-27T03:19:35.442831+0000","last_fullsized":"2024-04-27T03:19:35.442831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146608+0000","last_change":"2024-04-27T03:17:49.018017+0000","last_active":"2024-04-27T03:21:01.146608+0000","last_peered":"2024-04-27T03:21:01.146608+0000","last_clean":"2024-04-27T03:21:01.146608+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:01.146608+0000","last_undegraded":"2024-04-27T03:21:01.146608+0000","last_fullsized":"2024-04-27T03:21:01.146608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815679+0000","last_change":"2024-04-27T03:17:47.026063+0000","last_active":"2024-04-27T03:19:34.815679+0000","last_peered":"2024-04-27T03:19:34.815679+0000","last_clean":"2024-04-27T03:19:34.815679+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:19:34.815679+0000","last_undegraded":"2024-04-27T03:19:34.815679+0000","last_fullsized":"2024-04-27T03:19:34.815679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814107+0000","last_change":"2024-04-27T03:17:47.002843+0000","last_active":"2024-04-27T03:19:34.814107+0000","last_peered":"2024-04-27T03:19:34.814107+0000","last_clean":"2024-04-27T03:19:34.814107+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:19:34.814107+0000","last_undegraded":"2024-04-27T03:19:34.814107+0000","last_fullsized":"2024-04-27T03:19:34.814107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'11","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164859+0000","last_change":"2024-04-27T03:17:49.014133+0000","last_active":"2024-04-27T03:21:01.164859+0000","last_peered":"2024-04-27T03:21:01.164859+0000","last_clean":"2024-04-27T03:21:01.164859+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:01.164859+0000","last_undegraded":"2024-04-27T03:21:01.164859+0000","last_fullsized":"2024-04-27T03:21:01.164859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:38.383961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379782+0000","last_change":"2024-04-27T03:17:47.022513+0000","last_active":"2024-04-27T03:19:35.379782+0000","last_peered":"2024-04-27T03:19:35.379782+0000","last_clean":"2024-04-27T03:19:35.379782+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:19:35.379782+0000","last_undegraded":"2024-04-27T03:19:35.379782+0000","last_fullsized":"2024-04-27T03:19:35.379782+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:24.322205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385155+0000","last_change":"2024-04-27T03:17:49.012955+0000","last_active":"2024-04-27T03:19:35.385155+0000","last_peered":"2024-04-27T03:19:35.385155+0000","last_clean":"2024-04-27T03:19:35.385155+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:19:35.385155+0000","last_undegraded":"2024-04-27T03:19:35.385155+0000","last_fullsized":"2024-04-27T03:19:35.385155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2256,"ondisk_log_size":2256,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":13384,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1690,"num_read_kb":1444,"num_write":723,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":10824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1470,"ondisk_log_size":1470,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":961,"num_read_kb":961,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":473,"ondisk_log_size":473,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":25805696,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":309,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38719488,"data_stored":38719488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46056,"kb_used_data":6560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94325784,"statfs":{"total":96636764160,"available":96589602816,"internally_reserved":0,"allocated":6717440,"data_stored":6440144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57224,"kb_used_data":13568,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94314616,"statfs":{"total":96636764160,"available":96578166784,"internally_reserved":0,"allocated":13893632,"data_stored":13644798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":9524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":9752576,"data_stored":9451527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50504,"kb_used_data":11008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94321336,"statfs":{"total":96636764160,"available":96585048064,"internally_reserved":0,"allocated":11272192,"data_stored":10958989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10809344,"data_stored":10809344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8712192,"data_stored":8712192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":12906496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":4422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:21:09.615 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T03:21:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.754+0000 7f643fb39640 1 Processor -- start 2024-04-27T03:21:09.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.754+0000 7f643fb39640 1 -- start start 2024-04-27T03:21:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64380963b0 0x7f6438096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6438098d10 con 0x7f643806d4d0 2024-04-27T03:21:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6438098610 con 0x7f64380963b0 2024-04-27T03:21:09.768 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643d0d4640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f643806d4d0 0x7f643806d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46716/0 (socket says 172.21.15.77:46716) 2024-04-27T03:21:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643d8d5640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f64380963b0 0x7f6438096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643d0d4640 1 -- 172.21.15.77:0/2558946140 learned_addr learned my addr 172.21.15.77:0/2558946140 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643d8d5640 1 -- 172.21.15.77:0/2558946140 >> v1:172.21.15.77:6789/0 conn(0x7f643806d4d0 legacy=0x7f643806d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643d8d5640 1 -- 172.21.15.77:0/2558946140 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6438071f90 con 0x7f64380963b0 2024-04-27T03:21:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643d8d5640 1 --2- 172.21.15.77:0/2558946140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64380963b0 0x7f6438096790 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6434009740 tx=0x7f643401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146c61185d03f12f server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643c8d3640 1 -- 172.21.15.77:0/2558946140 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64340b0070 con 0x7f64380963b0 2024-04-27T03:21:09.770 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643c8d3640 1 -- 172.21.15.77:0/2558946140 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64340ab530 con 0x7f64380963b0 2024-04-27T03:21:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643c8d3640 1 -- 172.21.15.77:0/2558946140 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64340ab830 con 0x7f64380963b0 2024-04-27T03:21:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 -- 172.21.15.77:0/2558946140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64380963b0 msgr2=0x7f6438096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 --2- 172.21.15.77:0/2558946140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64380963b0 0x7f6438096790 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6434009740 tx=0x7f643401c400 comp rx=0 tx=0).stop 2024-04-27T03:21:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 -- 172.21.15.77:0/2558946140 shutdown_connections 2024-04-27T03:21:09.771 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 --2- 172.21.15.77:0/2558946140 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f64380963b0 0x7f6438096790 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 -- 172.21.15.77:0/2558946140 >> 172.21.15.77:0/2558946140 conn(0x7f6438099440 msgr2=0x7f6438099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 -- 172.21.15.77:0/2558946140 shutdown_connections 2024-04-27T03:21:09.772 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.758+0000 7f643fb39640 1 -- 172.21.15.77:0/2558946140 wait complete. 2024-04-27T03:21:09.773 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643fb39640 1 Processor -- start 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643fb39640 1 -- start start 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643fb39640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f643806d4d0 0x7f64381c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643fb39640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6438134d80 con 0x7f643806d4d0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643d8d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f643806d4d0 0x7f64381c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643d8d5640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f643806d4d0 0x7f64381c4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56808/0 (socket says 172.21.15.77:56808) 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643d8d5640 1 -- 172.21.15.77:0/1180244193 learned_addr learned my addr 172.21.15.77:0/1180244193 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643d8d5640 1 -- 172.21.15.77:0/1180244193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6438129770 con 0x7f643806d4d0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643d8d5640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f643806d4d0 0x7f64381c4010 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f64340ae040 tx=0x7f643401ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f642e7fc640 1 -- 172.21.15.77:0/1180244193 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64340b0050 con 0x7f643806d4d0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f642e7fc640 1 -- 172.21.15.77:0/1180244193 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64340b1040 con 0x7f643806d4d0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6438131680 con 0x7f643806d4d0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f642e7fc640 1 -- 172.21.15.77:0/1180244193 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64340b5780 con 0x7f643806d4d0 2024-04-27T03:21:09.774 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6438122960 con 0x7f643806d4d0 2024-04-27T03:21:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f642e7fc640 1 -- 172.21.15.77:0/1180244193 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f64340be030 con 0x7f643806d4d0 2024-04-27T03:21:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f642e7fc640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c035db0 0x7f641c038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6438062e50 con 0x7f643806d4d0 2024-04-27T03:21:09.775 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f642e7fc640 1 -- 172.21.15.77:0/1180244193 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f64340bc070 con 0x7f643806d4d0 2024-04-27T03:21:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.762+0000 7f643d0d4640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c035db0 0x7f641c038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:09.776 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.766+0000 7f643d0d4640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c035db0 0x7f641c038270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6428001570 tx=0x7f6428007ad0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:09.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.766+0000 7f642e7fc640 1 -- 172.21.15.77:0/1180244193 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f64340c5060 con 0x7f643806d4d0 2024-04-27T03:21:09.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.926+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6438124170 con 0x7f643806d4d0 2024-04-27T03:21:09.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.930+0000 7f642e7fc640 1 -- 172.21.15.77:0/1180244193 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+21217 (secure 0 0 0) 0x7f6434015ff0 con 0x7f643806d4d0 2024-04-27T03:21:09.942 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:21:09.943 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":31,"fsid":"3a154d27-fc3d-4e48-9ea2-3a6d919aa8c3","created":"2024-04-27T03:17:28.925654+0000","modified":"2024-04-27T03:19:34.804017+0000","last_up_change":"2024-04-27T03:17:37.618506+0000","last_in_change":"2024-04-27T03:17:30.101677+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-27T03:17:37.706444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-27T03:17:37.988012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2024-04-27T03:17:45.000494+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"client.0","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2024-04-27T03:17:47.360383+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":".rgw.root","create_time":"2024-04-27T03:17:49.270445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2024-04-27T03:17:51.197739+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2024-04-27T03:17:53.093328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2024-04-27T03:17:55.234191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1aa3f085-8c24-4f72-983c-bf2eb9ffc1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6801","nonce":3496324319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6803","nonce":3496324319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6807","nonce":3496324319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3496324319},{"type":"v1","addr":"172.21.15.77:6805","nonce":3496324319}]},"public_addr":"172.21.15.77:6801/3496324319","cluster_addr":"172.21.15.77:6803/3496324319","heartbeat_back_addr":"172.21.15.77:6807/3496324319","heartbeat_front_addr":"172.21.15.77:6805/3496324319","state":["exists","up"]},{"osd":1,"uuid":"74a30006-c87c-4e0e-9e94-9020affcae97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6801","nonce":1860462708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6803","nonce":1860462708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6807","nonce":1860462708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":1860462708},{"type":"v1","addr":"172.21.15.110:6805","nonce":1860462708}]},"public_addr":"172.21.15.110:6801/1860462708","cluster_addr":"172.21.15.110:6803/1860462708","heartbeat_back_addr":"172.21.15.110:6807/1860462708","heartbeat_front_addr":"172.21.15.110:6805/1860462708","state":["exists","up"]},{"osd":2,"uuid":"1bd2dbfb-57cb-4673-9684-d9427036427d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6809","nonce":3575017494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6811","nonce":3575017494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6815","nonce":3575017494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3575017494},{"type":"v1","addr":"172.21.15.110:6813","nonce":3575017494}]},"public_addr":"172.21.15.110:6809/3575017494","cluster_addr":"172.21.15.110:6811/3575017494","heartbeat_back_addr":"172.21.15.110:6815/3575017494","heartbeat_front_addr":"172.21.15.110:6813/3575017494","state":["exists","up"]},{"osd":3,"uuid":"d73614e8-5d01-4f79-8b12-0e08dabad6b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6817","nonce":3916647200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6819","nonce":3916647200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6823","nonce":3916647200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":3916647200},{"type":"v1","addr":"172.21.15.110:6821","nonce":3916647200}]},"public_addr":"172.21.15.110:6817/3916647200","cluster_addr":"172.21.15.110:6819/3916647200","heartbeat_back_addr":"172.21.15.110:6823/3916647200","heartbeat_front_addr":"172.21.15.110:6821/3916647200","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.465777+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.568546+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.593450+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-27T03:17:35.619617+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":2}]},{"pgid":"3.c","mappings":[{"from":3,"to":1}]},{"pgid":"3.21","mappings":[{"from":3,"to":0}]},{"pgid":"3.2d","mappings":[{"from":3,"to":0}]},{"pgid":"4.8","mappings":[{"from":3,"to":2}]},{"pgid":"4.a","mappings":[{"from":0,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":2}]},{"pgid":"4.25","mappings":[{"from":0,"to":2}]},{"pgid":"4.3c","mappings":[{"from":1,"to":2}]},{"pgid":"5.19","mappings":[{"from":1,"to":2}]},{"pgid":"6.8","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"client.0":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T03:21:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c035db0 msgr2=0x7f641c038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c035db0 0x7f641c038270 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6428001570 tx=0x7f6428007ad0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f643806d4d0 msgr2=0x7f64381c4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f643806d4d0 0x7f64381c4010 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f64340ae040 tx=0x7f643401ac40 comp rx=0 tx=0).stop 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 shutdown_connections 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f641c035db0 0x7f641c038270 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 --2- 172.21.15.77:0/1180244193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f643806d4d0 0x7f64381c4010 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 >> 172.21.15.77:0/1180244193 conn(0x7f6438099440 msgr2=0x7f6438095260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 shutdown_connections 2024-04-27T03:21:09.945 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:09.934+0000 7f643fb39640 1 -- 172.21.15.77:0/1180244193 wait complete. 2024-04-27T03:21:10.960 INFO:tasks.ceph:Scrubbing osd.0 2024-04-27T03:21:10.961 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-27T03:21:11.099 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f626565a640 1 Processor -- start 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f626565a640 1 -- start start 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f626565a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f626006e840 0x7f626006ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f626565a640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f626006cfe0 con 0x7f626006f160 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f626565a640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f626006ffa0 con 0x7f626006e840 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f625effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f626006e840 0x7f626006ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f625e7fc640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f626006f160 0x7f62601329c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46718/0 (socket says 172.21.15.77:46718) 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f625e7fc640 1 -- 172.21.15.77:0/3761753192 learned_addr learned my addr 172.21.15.77:0/3761753192 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f625effd640 1 -- 172.21.15.77:0/3761753192 >> v1:172.21.15.77:6789/0 conn(0x7f626006f160 legacy=0x7f62601329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f625dffb640 1 -- 172.21.15.77:0/3761753192 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900301005 0 0) 0x7f626006cfe0 con 0x7f626006f160 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.086+0000 7f625effd640 1 -- 172.21.15.77:0/3761753192 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626006fb60 con 0x7f626006e840 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625effd640 1 --2- 172.21.15.77:0/3761753192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f626006ec20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6258009d90 tx=0x7f625801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ff69b310ec7210 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.100 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625dffb640 1 -- 172.21.15.77:0/3761753192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62580a6640 con 0x7f626006e840 2024-04-27T03:21:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625dffb640 1 -- 172.21.15.77:0/3761753192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62580a67e0 con 0x7f626006e840 2024-04-27T03:21:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625dffb640 1 -- 172.21.15.77:0/3761753192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62580a6ae0 con 0x7f626006e840 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/3761753192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 msgr2=0x7f626006ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 --2- 172.21.15.77:0/3761753192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f626006ec20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6258009d90 tx=0x7f625801c8c0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/3761753192 shutdown_connections 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 --2- 172.21.15.77:0/3761753192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f626006ec20 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/3761753192 >> 172.21.15.77:0/3761753192 conn(0x7f626009d820 msgr2=0x7f626009dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/3761753192 shutdown_connections 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/3761753192 wait complete. 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 Processor -- start 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- start start 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f62601329d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6260136b10 con 0x7f626006e840 2024-04-27T03:21:11.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f62601329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625effd640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f62601329d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56824/0 (socket says 172.21.15.77:56824) 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625effd640 1 -- 172.21.15.77:0/842970488 learned_addr learned my addr 172.21.15.77:0/842970488 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625effd640 1 -- 172.21.15.77:0/842970488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62601249d0 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f625effd640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f62601329d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f62580194d0 tx=0x7f6258019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6258003ad0 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6258003c70 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6260126210 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62580b8640 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6260132f10 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6260062e50 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.090+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6258003f50 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f62577fe640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6240035db0 0x7f6240038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f62580e3e40 con 0x7f626006e840 2024-04-27T03:21:11.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f62577fe640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f62400423b0 0x7f6240044850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62580afe90 con 0x7f62400423b0 2024-04-27T03:21:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f62580e64f0 con 0x7f626006e840 2024-04-27T03:21:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f625f7fe640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f62400423b0 0x7f6240044850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f625e7fc640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6240035db0 0x7f6240038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f625f7fe640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f62400423b0 0x7f6240044850 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== osd.0 v2:172.21.15.77:6800/3496324319 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62580afe90 con 0x7f62400423b0 2024-04-27T03:21:11.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.094+0000 7f625e7fc640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6240035db0 0x7f6240038270 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6248007fd0 tx=0x7f6248007920 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 --> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6260136c90 con 0x7f62400423b0 2024-04-27T03:21:11.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f62577fe640 1 -- 172.21.15.77:0/842970488 <== osd.0 v2:172.21.15.77:6800/3496324319 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6260136c90 con 0x7f62400423b0 2024-04-27T03:21:11.118 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-27T03:21:11.118 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-27T03:21:11.118 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-27T03:21:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f62400423b0 msgr2=0x7f6240044850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f62400423b0 0x7f6240044850 crc :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6240035db0 msgr2=0x7f6240038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6240035db0 0x7f6240038270 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6248007fd0 tx=0x7f6248007920 comp rx=0 tx=0).stop 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 msgr2=0x7f62601329d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f62601329d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f62580194d0 tx=0x7f6258019660 comp rx=0 tx=0).stop 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 shutdown_connections 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:6800/3496324319,v1:172.21.15.77:6801/3496324319] conn(0x7f62400423b0 0x7f6240044850 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6240035db0 0x7f6240038270 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 --2- 172.21.15.77:0/842970488 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f626006e840 0x7f62601329d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 >> 172.21.15.77:0/842970488 conn(0x7f626009d820 msgr2=0x7f62600903c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 shutdown_connections 2024-04-27T03:21:11.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.106+0000 7f626565a640 1 -- 172.21.15.77:0/842970488 wait complete. 2024-04-27T03:21:11.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2024-04-27T03:21:11.134 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f3785fc8640 1 Processor -- start 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f3785fc8640 1 -- start start 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f3785fc8640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3780094ac0 0x7f3780133e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f3785fc8640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3780097040 con 0x7f3780093b60 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f3785fc8640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f3780096af0 con 0x7f3780094ac0 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3780094ac0 0x7f3780133e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377effd640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f3780094ac0 0x7f3780133e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56838/0 (socket says 172.21.15.77:56838) 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377effd640 1 -- 172.21.15.77:0/1491431347 learned_addr learned my addr 172.21.15.77:0/1491431347 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377f7fe640 1 --1- 172.21.15.77:0/1491431347 >> v1:172.21.15.77:6789/0 conn(0x7f3780093b60 0x7f37800943b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46722/0 (socket says 172.21.15.77:46722) 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377effd640 1 -- 172.21.15.77:0/1491431347 >> v1:172.21.15.77:6789/0 conn(0x7f3780093b60 legacy=0x7f37800943b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377effd640 1 -- 172.21.15.77:0/1491431347 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378006b470 con 0x7f3780094ac0 2024-04-27T03:21:11.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377effd640 1 --2- 172.21.15.77:0/1491431347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780094ac0 0x7f3780133e00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f37740097f0 tx=0x7f377401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbf0d273cf6189e server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377e7fc640 1 -- 172.21.15.77:0/1491431347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37740b0070 con 0x7f3780094ac0 2024-04-27T03:21:11.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377e7fc640 1 -- 172.21.15.77:0/1491431347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37740ab530 con 0x7f3780094ac0 2024-04-27T03:21:11.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.234+0000 7f377e7fc640 1 -- 172.21.15.77:0/1491431347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37740ab830 con 0x7f3780094ac0 2024-04-27T03:21:11.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/1491431347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780094ac0 msgr2=0x7f3780133e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 --2- 172.21.15.77:0/1491431347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780094ac0 0x7f3780133e00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f37740097f0 tx=0x7f377401c3d0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/1491431347 shutdown_connections 2024-04-27T03:21:11.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 --2- 172.21.15.77:0/1491431347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780094ac0 0x7f3780133e00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/1491431347 >> 172.21.15.77:0/1491431347 conn(0x7f378009d040 msgr2=0x7f378009d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/1491431347 shutdown_connections 2024-04-27T03:21:11.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/1491431347 wait complete. 2024-04-27T03:21:11.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 Processor -- start 2024-04-27T03:21:11.250 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- start start 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780093b60 0x7f3780121f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f3780136ce0 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780093b60 0x7f3780121f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f377f7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780093b60 0x7f3780121f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56844/0 (socket says 172.21.15.77:56844) 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f377f7fe640 1 -- 172.21.15.77:0/2302873443 learned_addr learned my addr 172.21.15.77:0/2302873443 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f377f7fe640 1 -- 172.21.15.77:0/2302873443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3780072ab0 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f377f7fe640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780093b60 0x7f3780121f60 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f377800c3a0 tx=0x7f377800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3767fff640 1 -- 172.21.15.77:0/2302873443 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3778007410 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3767fff640 1 -- 172.21.15.77:0/2302873443 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37780075b0 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3780124050 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3767fff640 1 -- 172.21.15.77:0/2302873443 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3778004920 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37801246e0 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3780062e50 con 0x7f3780093b60 2024-04-27T03:21:11.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3767fff640 1 -- 172.21.15.77:0/2302873443 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f377801a020 con 0x7f3780093b60 2024-04-27T03:21:11.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3767fff640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f376c035db0 0x7f376c038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f3767fff640 1 -- 172.21.15.77:0/2302873443 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f37780489c0 con 0x7f3780093b60 2024-04-27T03:21:11.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.238+0000 7f377effd640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f376c035db0 0x7f376c038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.252 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.242+0000 7f377effd640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f376c035db0 0x7f376c038270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37740ad000 tx=0x7f37740a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.255 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.246+0000 7f3767fff640 1 -- 172.21.15.77:0/2302873443 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f377801b9f0 con 0x7f3780093b60 2024-04-27T03:21:11.410 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.398+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f37801233e0 con 0x7f376c035db0 2024-04-27T03:21:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3767fff640 1 -- 172.21.15.77:0/2302873443 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f37801233e0 con 0x7f376c035db0 2024-04-27T03:21:11.411 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 0 to deep-scrub 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f376c035db0 msgr2=0x7f376c038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f376c035db0 0x7f376c038270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37740ad000 tx=0x7f37740a6000 comp rx=0 tx=0).stop 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780093b60 msgr2=0x7f3780121f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780093b60 0x7f3780121f60 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f377800c3a0 tx=0x7f377800c860 comp rx=0 tx=0).stop 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 shutdown_connections 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f376c035db0 0x7f376c038270 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 --2- 172.21.15.77:0/2302873443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3780093b60 0x7f3780121f60 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 >> 172.21.15.77:0/2302873443 conn(0x7f378009d040 msgr2=0x7f3780133020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 shutdown_connections 2024-04-27T03:21:11.415 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.402+0000 7f3785fc8640 1 -- 172.21.15.77:0/2302873443 wait complete. 2024-04-27T03:21:11.430 INFO:tasks.ceph:Scrubbing osd.1 2024-04-27T03:21:11.430 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f513e998640 1 Processor -- start 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f513e998640 1 -- start start 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f513e998640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5138094ee0 0x7f51380952c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f513e998640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5138097730 con 0x7f5138095800 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f513e998640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f5138097420 con 0x7f5138094ee0 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f5137fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5138094ee0 0x7f51380952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f5137fff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f5138094ee0 0x7f51380952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56846/0 (socket says 172.21.15.77:56846) 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f5137fff640 1 -- 172.21.15.77:0/1540158673 learned_addr learned my addr 172.21.15.77:0/1540158673 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:11.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f51377fe640 1 --1- 172.21.15.77:0/1540158673 >> v1:172.21.15.77:6789/0 conn(0x7f5138095800 0x7f5138095c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46728/0 (socket says 172.21.15.77:46728) 2024-04-27T03:21:11.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f5137fff640 1 -- 172.21.15.77:0/1540158673 >> v1:172.21.15.77:6789/0 conn(0x7f5138095800 legacy=0x7f5138095c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f5137fff640 1 -- 172.21.15.77:0/1540158673 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5138096b50 con 0x7f5138094ee0 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f5137fff640 1 --2- 172.21.15.77:0/1540158673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f51380952c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f51280097f0 tx=0x7f512801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d0d195df9ebb8d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.534+0000 7f5136ffd640 1 -- 172.21.15.77:0/1540158673 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51280b0070 con 0x7f5138094ee0 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5136ffd640 1 -- 172.21.15.77:0/1540158673 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51280ab530 con 0x7f5138094ee0 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5136ffd640 1 -- 172.21.15.77:0/1540158673 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51280ab830 con 0x7f5138094ee0 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/1540158673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 msgr2=0x7f51380952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 --2- 172.21.15.77:0/1540158673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f51380952c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f51280097f0 tx=0x7f512801c430 comp rx=0 tx=0).stop 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/1540158673 shutdown_connections 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 --2- 172.21.15.77:0/1540158673 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f51380952c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/1540158673 >> 172.21.15.77:0/1540158673 conn(0x7f513809ebc0 msgr2=0x7f513809efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/1540158673 shutdown_connections 2024-04-27T03:21:11.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/1540158673 wait complete. 2024-04-27T03:21:11.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 Processor -- start 2024-04-27T03:21:11.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- start start 2024-04-27T03:21:11.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f5138091410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f51381365f0 con 0x7f5138094ee0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5137fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f5138091410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5137fff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f5138091410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56854/0 (socket says 172.21.15.77:56854) 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5137fff640 1 -- 172.21.15.77:0/2282625067 learned_addr learned my addr 172.21.15.77:0/2282625067 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5137fff640 1 -- 172.21.15.77:0/2282625067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5138125980 con 0x7f5138094ee0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5137fff640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f5138091410 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f51280097f0 tx=0x7f51280191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51280b0040 con 0x7f5138094ee0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51280aa650 con 0x7f5138094ee0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51280aa950 con 0x7f5138094ee0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5138093ed0 con 0x7f5138094ee0 2024-04-27T03:21:11.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5138091950 con 0x7f5138094ee0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5138062e50 con 0x7f5138094ee0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f51280bb030 con 0x7f5138094ee0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5134ff9640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5124035db0 0x7f5124038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.538+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f51280b9070 con 0x7f5138094ee0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f5134ff9640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f5124042410 0x7f51240448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51280b1e50 con 0x7f5124042410 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f51280e72a0 con 0x7f5138094ee0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f51377fe640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5124035db0 0x7f5124038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f513cf35640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f5124042410 0x7f51240448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f51377fe640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5124035db0 0x7f5124038270 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f512c001600 tx=0x7f512c007a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f513cf35640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f5124042410 0x7f51240448b0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.542+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== osd.1 v2:172.21.15.110:6800/1860462708 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f51280b1e50 con 0x7f5124042410 2024-04-27T03:21:11.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.554+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 --> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5138095800 con 0x7f5124042410 2024-04-27T03:21:11.568 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f5134ff9640 1 -- 172.21.15.77:0/2282625067 <== osd.1 v2:172.21.15.110:6800/1860462708 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5138095800 con 0x7f5124042410 2024-04-27T03:21:11.568 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-27T03:21:11.568 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-27T03:21:11.568 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f5124042410 msgr2=0x7f51240448b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f5124042410 0x7f51240448b0 crc :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5124035db0 msgr2=0x7f5124038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5124035db0 0x7f5124038270 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f512c001600 tx=0x7f512c007a50 comp rx=0 tx=0).stop 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 msgr2=0x7f5138091410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f5138091410 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f51280097f0 tx=0x7f51280191d0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 shutdown_connections 2024-04-27T03:21:11.569 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6800/1860462708,v1:172.21.15.110:6801/1860462708] conn(0x7f5124042410 0x7f51240448b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f5124035db0 0x7f5124038270 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 --2- 172.21.15.77:0/2282625067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5138094ee0 0x7f5138091410 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 >> 172.21.15.77:0/2282625067 conn(0x7f513809ebc0 msgr2=0x7f513809b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 shutdown_connections 2024-04-27T03:21:11.570 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.558+0000 7f513e998640 1 -- 172.21.15.77:0/2282625067 wait complete. 2024-04-27T03:21:11.582 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.678+0000 7f9c96d72640 1 Processor -- start 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.678+0000 7f9c96d72640 1 -- start start 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c9012f3c0 0x7f9c901317b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c90070a60 con 0x7f9c900963b0 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f9c90098d10 con 0x7f9c9012f3c0 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c9012f3c0 0x7f9c901317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c94b0e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f9c900963b0 0x7f9c90096790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46744/0 (socket says 172.21.15.77:46744) 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c94b0e640 1 -- 172.21.15.77:0/751236765 learned_addr learned my addr 172.21.15.77:0/751236765 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:11.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8ffff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f9c9012f3c0 0x7f9c901317b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56870/0 (socket says 172.21.15.77:56870) 2024-04-27T03:21:11.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8ffff640 1 -- 172.21.15.77:0/751236765 >> v1:172.21.15.77:6789/0 conn(0x7f9c900963b0 legacy=0x7f9c90096790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8ffff640 1 -- 172.21.15.77:0/751236765 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c90098610 con 0x7f9c9012f3c0 2024-04-27T03:21:11.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8f7fe640 1 -- 172.21.15.77:0/751236765 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4169990043 0 0) 0x7f9c90070a60 con 0x7f9c900963b0 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8ffff640 1 --2- 172.21.15.77:0/751236765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9012f3c0 0x7f9c901317b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009740 tx=0x7f9c7801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73385feb278afc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8f7fe640 1 -- 172.21.15.77:0/751236765 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c780b0070 con 0x7f9c9012f3c0 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8f7fe640 1 -- 172.21.15.77:0/751236765 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c780ab530 con 0x7f9c9012f3c0 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c8f7fe640 1 -- 172.21.15.77:0/751236765 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c780ab830 con 0x7f9c9012f3c0 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 -- 172.21.15.77:0/751236765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9012f3c0 msgr2=0x7f9c901317b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 --2- 172.21.15.77:0/751236765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9012f3c0 0x7f9c901317b0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009740 tx=0x7f9c7801c370 comp rx=0 tx=0).stop 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 -- 172.21.15.77:0/751236765 shutdown_connections 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 --2- 172.21.15.77:0/751236765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c9012f3c0 0x7f9c901317b0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 -- 172.21.15.77:0/751236765 >> 172.21.15.77:0/751236765 conn(0x7f9c90099440 msgr2=0x7f9c90099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 -- 172.21.15.77:0/751236765 shutdown_connections 2024-04-27T03:21:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.682+0000 7f9c96d72640 1 -- 172.21.15.77:0/751236765 wait complete. 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c96d72640 1 Processor -- start 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c96d72640 1 -- start start 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c96d72640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c900963b0 0x7f9c901c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c96d72640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f9c90134690 con 0x7f9c900963b0 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c94b0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c900963b0 0x7f9c901c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c94b0e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c900963b0 0x7f9c901c4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56880/0 (socket says 172.21.15.77:56880) 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c94b0e640 1 -- 172.21.15.77:0/3003766429 learned_addr learned my addr 172.21.15.77:0/3003766429 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c94b0e640 1 -- 172.21.15.77:0/3003766429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c901297c0 con 0x7f9c900963b0 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c94b0e640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c900963b0 0x7f9c901c4c90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c420 tx=0x7f9c8000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.697 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8d7fa640 1 -- 172.21.15.77:0/3003766429 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c80018070 con 0x7f9c900963b0 2024-04-27T03:21:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8d7fa640 1 -- 172.21.15.77:0/3003766429 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c800074d0 con 0x7f9c900963b0 2024-04-27T03:21:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c90128eb0 con 0x7f9c900963b0 2024-04-27T03:21:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8d7fa640 1 -- 172.21.15.77:0/3003766429 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c80004960 con 0x7f9c900963b0 2024-04-27T03:21:11.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9008a630 con 0x7f9c900963b0 2024-04-27T03:21:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c90062e50 con 0x7f9c900963b0 2024-04-27T03:21:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8d7fa640 1 -- 172.21.15.77:0/3003766429 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9c8001b070 con 0x7f9c900963b0 2024-04-27T03:21:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8d7fa640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9c70035db0 0x7f9c70038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8d7fa640 1 -- 172.21.15.77:0/3003766429 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f9c800485d0 con 0x7f9c900963b0 2024-04-27T03:21:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8ffff640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9c70035db0 0x7f9c70038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:11.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.686+0000 7f9c8ffff640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9c70035db0 0x7f9c70038270 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9c780ad000 tx=0x7f9c78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:11.702 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.690+0000 7f9c8d7fa640 1 -- 172.21.15.77:0/3003766429 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9c80004db0 con 0x7f9c900963b0 2024-04-27T03:21:11.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.850+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f9c9006c220 con 0x7f9c70035db0 2024-04-27T03:21:11.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c8d7fa640 1 -- 172.21.15.77:0/3003766429 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9c9006c220 con 0x7f9c70035db0 2024-04-27T03:21:11.863 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 1 to deep-scrub 2024-04-27T03:21:11.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9c70035db0 msgr2=0x7f9c70038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9c70035db0 0x7f9c70038270 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9c780ad000 tx=0x7f9c78008040 comp rx=0 tx=0).stop 2024-04-27T03:21:11.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c900963b0 msgr2=0x7f9c901c4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:11.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c900963b0 0x7f9c901c4c90 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9c8000c420 tx=0x7f9c8000c8e0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 shutdown_connections 2024-04-27T03:21:11.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9c70035db0 0x7f9c70038270 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 --2- 172.21.15.77:0/3003766429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c900963b0 0x7f9c901c4c90 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:11.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 >> 172.21.15.77:0/3003766429 conn(0x7f9c90099440 msgr2=0x7f9c9009a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:11.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 shutdown_connections 2024-04-27T03:21:11.867 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.854+0000 7f9c96d72640 1 -- 172.21.15.77:0/3003766429 wait complete. 2024-04-27T03:21:11.882 INFO:tasks.ceph:Scrubbing osd.2 2024-04-27T03:21:11.882 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f9642a5c640 1 Processor -- start 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f9642a5c640 1 -- start start 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f9642a5c640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f963c072090 0x7f963c12c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f9642a5c640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f963c113000 con 0x7f963c12c9e0 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f9642a5c640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f963c06f8a0 con 0x7f963c072090 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963bfff640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f963c072090 0x7f963c12c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963b7fe640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f963c12c9e0 0x7f963c06e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46758/0 (socket says 172.21.15.77:46758) 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963b7fe640 1 -- 172.21.15.77:0/655695764 learned_addr learned my addr 172.21.15.77:0/655695764 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963bfff640 1 -- 172.21.15.77:0/655695764 >> v1:172.21.15.77:6789/0 conn(0x7f963c12c9e0 legacy=0x7f963c06e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963bfff640 1 -- 172.21.15.77:0/655695764 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963c12c050 con 0x7f963c072090 2024-04-27T03:21:12.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963bfff640 1 --2- 172.21.15.77:0/655695764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c12c4a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f962c009740 tx=0x7f962c01ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e973c777bf1f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963affd640 1 -- 172.21.15.77:0/655695764 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f962c0b5070 con 0x7f963c072090 2024-04-27T03:21:12.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963affd640 1 -- 172.21.15.77:0/655695764 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f962c0a8820 con 0x7f963c072090 2024-04-27T03:21:12.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.994+0000 7f963affd640 1 -- 172.21.15.77:0/655695764 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f962c0a8b20 con 0x7f963c072090 2024-04-27T03:21:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/655695764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 msgr2=0x7f963c12c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 --2- 172.21.15.77:0/655695764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c12c4a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f962c009740 tx=0x7f962c01ebf0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/655695764 shutdown_connections 2024-04-27T03:21:12.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 --2- 172.21.15.77:0/655695764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c12c4a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/655695764 >> 172.21.15.77:0/655695764 conn(0x7f963c0992d0 msgr2=0x7f963c0996e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/655695764 shutdown_connections 2024-04-27T03:21:12.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/655695764 wait complete. 2024-04-27T03:21:12.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 Processor -- start 2024-04-27T03:21:12.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- start start 2024-04-27T03:21:12.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c1c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f963c113760 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f963bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c1c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f963bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c1c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56894/0 (socket says 172.21.15.77:56894) 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f963bfff640 1 -- 172.21.15.77:0/3714934581 learned_addr learned my addr 172.21.15.77:0/3714934581 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f963bfff640 1 -- 172.21.15.77:0/3714934581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963c119640 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f963bfff640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c1c1120 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f962c01c820 tx=0x7f962c01b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f962c0bd020 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f962c0b0630 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f963c1c1660 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f962c0b5040 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f963c1c1b20 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f963c119930 con 0x7f963c072090 2024-04-27T03:21:12.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f962c0c0020 con 0x7f963c072090 2024-04-27T03:21:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9628035db0 0x7f9628038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f962c0e6500 con 0x7f963c072090 2024-04-27T03:21:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9628042410 0x7f96280448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f962c0bae80 con 0x7f9628042410 2024-04-27T03:21:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:11.998+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f962c0ed050 con 0x7f963c072090 2024-04-27T03:21:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.002+0000 7f963b7fe640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9628035db0 0x7f9628038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.002+0000 7f9640ff9640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9628042410 0x7f96280448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.002+0000 7f963b7fe640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9628035db0 0x7f9628038270 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9630001610 tx=0x7f9630007ad0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.002+0000 7f9640ff9640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9628042410 0x7f96280448b0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.002+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== osd.2 v2:172.21.15.110:6808/3575017494 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f962c0bae80 con 0x7f9628042410 2024-04-27T03:21:12.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.014+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 --> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f963c12b390 con 0x7f9628042410 2024-04-27T03:21:12.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9638ff9640 1 -- 172.21.15.77:0/3714934581 <== osd.2 v2:172.21.15.110:6808/3575017494 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f963400ad10 con 0x7f9628042410 2024-04-27T03:21:12.027 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-27T03:21:12.027 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-27T03:21:12.028 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-27T03:21:12.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9628042410 msgr2=0x7f96280448b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9628042410 0x7f96280448b0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9628035db0 msgr2=0x7f9628038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9628035db0 0x7f9628038270 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9630001610 tx=0x7f9630007ad0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 msgr2=0x7f963c1c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c1c1120 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f962c01c820 tx=0x7f962c01b990 comp rx=0 tx=0).stop 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 shutdown_connections 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f9628035db0 0x7f9628038270 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.110:6808/3575017494,v1:172.21.15.110:6809/3575017494] conn(0x7f9628042410 0x7f96280448b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 --2- 172.21.15.77:0/3714934581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f963c072090 0x7f963c1c1120 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 >> 172.21.15.77:0/3714934581 conn(0x7f963c0992d0 msgr2=0x7f963c09b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 shutdown_connections 2024-04-27T03:21:12.029 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.018+0000 7f9642a5c640 1 -- 172.21.15.77:0/3714934581 wait complete. 2024-04-27T03:21:12.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-27T03:21:12.156 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc297a93640 1 Processor -- start 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc297a93640 1 -- start start 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc297a93640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc29006ec60 0x7fc29006f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc297a93640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc290070470 con 0x7fc29006f580 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc297a93640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc2900701a0 con 0x7fc29006ec60 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29582f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc29006ec60 0x7fc29006f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29582f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc29006ec60 0x7fc29006f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56896/0 (socket says 172.21.15.77:56896) 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29502e640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc29006f580 0x7fc29012e2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46770/0 (socket says 172.21.15.77:46770) 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29582f640 1 -- 172.21.15.77:0/1453301576 learned_addr learned my addr 172.21.15.77:0/1453301576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29582f640 1 -- 172.21.15.77:0/1453301576 >> v1:172.21.15.77:6789/0 conn(0x7fc29006f580 legacy=0x7fc29012e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.157 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29582f640 1 -- 172.21.15.77:0/1453301576 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc290071b40 con 0x7fc29006ec60 2024-04-27T03:21:12.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29582f640 1 --2- 172.21.15.77:0/1453301576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29006f040 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0097f0 tx=0x7fc28c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c7ac661af26996 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29482d640 1 -- 172.21.15.77:0/1453301576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc28c0ab580 con 0x7fc29006ec60 2024-04-27T03:21:12.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29482d640 1 -- 172.21.15.77:0/1453301576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc28c0b0070 con 0x7fc29006ec60 2024-04-27T03:21:12.159 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.146+0000 7fc29482d640 1 -- 172.21.15.77:0/1453301576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc28c0ab870 con 0x7fc29006ec60 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/1453301576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 msgr2=0x7fc29006f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 --2- 172.21.15.77:0/1453301576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29006f040 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0097f0 tx=0x7fc28c01c430 comp rx=0 tx=0).stop 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/1453301576 shutdown_connections 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 --2- 172.21.15.77:0/1453301576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29006f040 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/1453301576 >> 172.21.15.77:0/1453301576 conn(0x7fc29009dc80 msgr2=0x7fc29006d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/1453301576 shutdown_connections 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/1453301576 wait complete. 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 Processor -- start 2024-04-27T03:21:12.160 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- start start 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29012a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc2901371b0 con 0x7fc29006ec60 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc29582f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29012a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc29582f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29012a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56900/0 (socket says 172.21.15.77:56900) 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc29582f640 1 -- 172.21.15.77:0/147787516 learned_addr learned my addr 172.21.15.77:0/147787516 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc29582f640 1 -- 172.21.15.77:0/147787516 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29012c080 con 0x7fc29006ec60 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc29582f640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29012a9e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc28c019040 tx=0x7fc28c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc2867fc640 1 -- 172.21.15.77:0/147787516 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc28c0b4600 con 0x7fc29006ec60 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc2867fc640 1 -- 172.21.15.77:0/147787516 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc28c0b47c0 con 0x7fc29006ec60 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc29011d220 con 0x7fc29006ec60 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc2867fc640 1 -- 172.21.15.77:0/147787516 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc28c0b4ae0 con 0x7fc29006ec60 2024-04-27T03:21:12.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc29012af20 con 0x7fc29006ec60 2024-04-27T03:21:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc2867fc640 1 -- 172.21.15.77:0/147787516 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc28c0b4c80 con 0x7fc29006ec60 2024-04-27T03:21:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc290062e50 con 0x7fc29006ec60 2024-04-27T03:21:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc2867fc640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc274035db0 0x7fc274038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc2867fc640 1 -- 172.21.15.77:0/147787516 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc28c0e3e60 con 0x7fc29006ec60 2024-04-27T03:21:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.150+0000 7fc29502e640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc274035db0 0x7fc274038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.154+0000 7fc29502e640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc274035db0 0x7fc274038270 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc280001600 tx=0x7fc280007a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.154+0000 7fc2867fc640 1 -- 172.21.15.77:0/147787516 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc28c0160b0 con 0x7fc29006ec60 2024-04-27T03:21:12.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.298+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fc29006c220 con 0x7fc274035db0 2024-04-27T03:21:12.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc2867fc640 1 -- 172.21.15.77:0/147787516 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc29006c220 con 0x7fc274035db0 2024-04-27T03:21:12.312 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 2 to deep-scrub 2024-04-27T03:21:12.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc274035db0 msgr2=0x7fc274038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc274035db0 0x7fc274038270 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc280001600 tx=0x7fc280007a50 comp rx=0 tx=0).stop 2024-04-27T03:21:12.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 msgr2=0x7fc29012a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29012a9e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc28c019040 tx=0x7fc28c01ac70 comp rx=0 tx=0).stop 2024-04-27T03:21:12.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 shutdown_connections 2024-04-27T03:21:12.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc274035db0 0x7fc274038270 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 --2- 172.21.15.77:0/147787516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc29006ec60 0x7fc29012a9e0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 >> 172.21.15.77:0/147787516 conn(0x7fc29009dc80 msgr2=0x7fc290130e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 shutdown_connections 2024-04-27T03:21:12.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.302+0000 7fc297a93640 1 -- 172.21.15.77:0/147787516 wait complete. 2024-04-27T03:21:12.328 INFO:tasks.ceph:Scrubbing osd.3 2024-04-27T03:21:12.329 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-27T03:21:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fb066e640 1 Processor -- start 2024-04-27T03:21:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fb066e640 1 -- start start 2024-04-27T03:21:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fb066e640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1fa8072090 0x7f1fa812c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fb066e640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fa8113000 con 0x7f1fa812c9e0 2024-04-27T03:21:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fb066e640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f1fa806f8a0 con 0x7f1fa8072090 2024-04-27T03:21:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fae40a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1fa8072090 0x7f1fa812c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fae40a640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f1fa8072090 0x7f1fa812c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56906/0 (socket says 172.21.15.77:56906) 2024-04-27T03:21:12.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.430+0000 7f1fae40a640 1 -- 172.21.15.77:0/2531338059 learned_addr learned my addr 172.21.15.77:0/2531338059 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:12.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fae40a640 1 -- 172.21.15.77:0/2531338059 >> v1:172.21.15.77:6789/0 conn(0x7f1fa812c9e0 legacy=0x7f1fa806e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fae40a640 1 -- 172.21.15.77:0/2531338059 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fa812c050 con 0x7f1fa8072090 2024-04-27T03:21:12.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fad408640 1 -- 172.21.15.77:0/2531338059 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2271548738 0 0) 0x7f1fa8113000 con 0x7f1fa812c9e0 2024-04-27T03:21:12.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fae40a640 1 --2- 172.21.15.77:0/2531338059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa812c4a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009e50 tx=0x7f1fa001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6731eba460a16b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fad408640 1 -- 172.21.15.77:0/2531338059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa00b4070 con 0x7f1fa8072090 2024-04-27T03:21:12.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fad408640 1 -- 172.21.15.77:0/2531338059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fa0009200 con 0x7f1fa8072090 2024-04-27T03:21:12.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fad408640 1 -- 172.21.15.77:0/2531338059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa00af550 con 0x7f1fa8072090 2024-04-27T03:21:12.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 -- 172.21.15.77:0/2531338059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 msgr2=0x7f1fa812c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.446 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2531338059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa812c4a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009e50 tx=0x7f1fa001f1f0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 -- 172.21.15.77:0/2531338059 shutdown_connections 2024-04-27T03:21:12.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2531338059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa812c4a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 -- 172.21.15.77:0/2531338059 >> 172.21.15.77:0/2531338059 conn(0x7f1fa80992d0 msgr2=0x7f1fa80996e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 -- 172.21.15.77:0/2531338059 shutdown_connections 2024-04-27T03:21:12.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 -- 172.21.15.77:0/2531338059 wait complete. 2024-04-27T03:21:12.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 Processor -- start 2024-04-27T03:21:12.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.434+0000 7f1fb066e640 1 -- start start 2024-04-27T03:21:12.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fb066e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fb066e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa8113760 con 0x7f1fa8072090 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fae40a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fae40a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa806fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56908/0 (socket says 172.21.15.77:56908) 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fae40a640 1 -- 172.21.15.77:0/2312347093 learned_addr learned my addr 172.21.15.77:0/2312347093 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fae40a640 1 -- 172.21.15.77:0/2312347093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fa8118af0 con 0x7f1fa8072090 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fae40a640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa806fe80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00b2040 tx=0x7f1fa001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa00be070 con 0x7f1fa8072090 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fa00afb60 con 0x7f1fa8072090 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fa80711d0 con 0x7f1fa8072090 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa00b4040 con 0x7f1fa8072090 2024-04-27T03:21:12.449 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fa8071c20 con 0x7f1fa8072090 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1fa8129610 con 0x7f1fa8072090 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f1fa00c1070 con 0x7f1fa8072090 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1f94035db0 0x7f1f94038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f1fa00e7660 con 0x7f1fa8072090 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7f1f94042410 0x7f1f940448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fa00c1e50 con 0x7f1f94042410 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1fa00ee050 con 0x7f1fa8072090 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fadc09640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1f94035db0 0x7f1f94038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1faec0b640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7f1f94042410 0x7f1f940448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1faec0b640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7f1f94042410 0x7f1f940448b0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.451 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.438+0000 7f1fadc09640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1f94035db0 0x7f1f94038270 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c000f70 tx=0x7f1f9c007910 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.452 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.442+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== osd.3 v2:172.21.15.110:6816/3916647200 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1fa00c1e50 con 0x7f1f94042410 2024-04-27T03:21:12.464 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 --> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1fa8062e50 con 0x7f1f94042410 2024-04-27T03:21:12.465 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fa6ffd640 1 -- 172.21.15.77:0/2312347093 <== osd.3 v2:172.21.15.110:6816/3916647200 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1fa8062e50 con 0x7f1f94042410 2024-04-27T03:21:12.465 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-04-27T03:21:12.465 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-27T03:21:12.465 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-04-27T03:21:12.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7f1f94042410 msgr2=0x7f1f940448b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7f1f94042410 0x7f1f940448b0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1f94035db0 msgr2=0x7f1f94038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.466 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1f94035db0 0x7f1f94038270 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c000f70 tx=0x7f1f9c007910 comp rx=0 tx=0).stop 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 msgr2=0x7f1fa806fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa806fe80 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00b2040 tx=0x7f1fa001bf90 comp rx=0 tx=0).stop 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 shutdown_connections 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6816/3916647200,v1:172.21.15.110:6817/3916647200] conn(0x7f1f94042410 0x7f1f940448b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1f94035db0 0x7f1f94038270 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 --2- 172.21.15.77:0/2312347093 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1fa8072090 0x7f1fa806fe80 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 >> 172.21.15.77:0/2312347093 conn(0x7f1fa80992d0 msgr2=0x7f1fa809a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 shutdown_connections 2024-04-27T03:21:12.467 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.454+0000 7f1fb066e640 1 -- 172.21.15.77:0/2312347093 wait complete. 2024-04-27T03:21:12.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.574+0000 7f01e43fc640 1 Processor -- start 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.574+0000 7f01e43fc640 1 -- start start 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e43fc640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01dc0963b0 0x7f01dc096790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e43fc640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01dc098d10 con 0x7f01dc06d4d0 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e43fc640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f01dc098610 con 0x7f01dc0963b0 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e2198640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f01dc0963b0 0x7f01dc096790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e1997640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f01dc06d4d0 0x7f01dc06d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46784/0 (socket says 172.21.15.77:46784) 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e1997640 1 -- 172.21.15.77:0/193685254 learned_addr learned my addr 172.21.15.77:0/193685254 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:12.588 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e2198640 1 -- 172.21.15.77:0/193685254 >> v1:172.21.15.77:6789/0 conn(0x7f01dc06d4d0 legacy=0x7f01dc06d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e1196640 1 -- 172.21.15.77:0/193685254 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3861914227 0 0) 0x7f01dc098d10 con 0x7f01dc06d4d0 2024-04-27T03:21:12.589 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.578+0000 7f01e2198640 1 -- 172.21.15.77:0/193685254 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc071f90 con 0x7f01dc0963b0 2024-04-27T03:21:12.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e2198640 1 --2- 172.21.15.77:0/193685254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc0963b0 0x7f01dc096790 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009740 tx=0x7f01cc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c59bb759cdffe08 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e1196640 1 -- 172.21.15.77:0/193685254 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01cc0b0070 con 0x7f01dc0963b0 2024-04-27T03:21:12.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e1196640 1 -- 172.21.15.77:0/193685254 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01cc0ab530 con 0x7f01dc0963b0 2024-04-27T03:21:12.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e1196640 1 -- 172.21.15.77:0/193685254 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01cc0ab830 con 0x7f01dc0963b0 2024-04-27T03:21:12.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- 172.21.15.77:0/193685254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc0963b0 msgr2=0x7f01dc096790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 --2- 172.21.15.77:0/193685254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc0963b0 0x7f01dc096790 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009740 tx=0x7f01cc01c400 comp rx=0 tx=0).stop 2024-04-27T03:21:12.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- 172.21.15.77:0/193685254 shutdown_connections 2024-04-27T03:21:12.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 --2- 172.21.15.77:0/193685254 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc0963b0 0x7f01dc096790 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- 172.21.15.77:0/193685254 >> 172.21.15.77:0/193685254 conn(0x7f01dc099440 msgr2=0x7f01dc099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- 172.21.15.77:0/193685254 shutdown_connections 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- 172.21.15.77:0/193685254 wait complete. 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 Processor -- start 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- start start 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc06d4d0 0x7f01dc1c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc134c00 con 0x7f01dc06d4d0 2024-04-27T03:21:12.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e2198640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc06d4d0 0x7f01dc1c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e2198640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc06d4d0 0x7f01dc1c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56926/0 (socket says 172.21.15.77:56926) 2024-04-27T03:21:12.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e2198640 1 -- 172.21.15.77:0/2667846804 learned_addr learned my addr 172.21.15.77:0/2667846804 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:12.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e2198640 1 -- 172.21.15.77:0/2667846804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc06fbe0 con 0x7f01dc06d4d0 2024-04-27T03:21:12.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e2198640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc06d4d0 0x7f01dc1c62f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0ae040 tx=0x7f01cc01ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01d2ffd640 1 -- 172.21.15.77:0/2667846804 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01cc0b0050 con 0x7f01dc06d4d0 2024-04-27T03:21:12.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01d2ffd640 1 -- 172.21.15.77:0/2667846804 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01cc0b1040 con 0x7f01dc06d4d0 2024-04-27T03:21:12.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01dc126390 con 0x7f01dc06d4d0 2024-04-27T03:21:12.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01d2ffd640 1 -- 172.21.15.77:0/2667846804 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01cc0b5780 con 0x7f01dc06d4d0 2024-04-27T03:21:12.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.582+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01dc08a9a0 con 0x7f01dc06d4d0 2024-04-27T03:21:12.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.586+0000 7f01d2ffd640 1 -- 172.21.15.77:0/2667846804 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f01cc0be030 con 0x7f01dc06d4d0 2024-04-27T03:21:12.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.586+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01dc06f9a0 con 0x7f01dc06d4d0 2024-04-27T03:21:12.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.586+0000 7f01d2ffd640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f01c0035db0 0x7f01c0038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.586+0000 7f01d2ffd640 1 -- 172.21.15.77:0/2667846804 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f01cc0bc070 con 0x7f01dc06d4d0 2024-04-27T03:21:12.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.586+0000 7f01e1997640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f01c0035db0 0x7f01c0038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.586+0000 7f01e1997640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f01c0035db0 0x7f01c0038270 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01d80015b0 tx=0x7f01d80079f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.590+0000 7f01d2ffd640 1 -- 172.21.15.77:0/2667846804 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f01cc0c5060 con 0x7f01dc06d4d0 2024-04-27T03:21:12.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.738+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f01dc062e50 con 0x7f01c0035db0 2024-04-27T03:21:12.752 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01d2ffd640 1 -- 172.21.15.77:0/2667846804 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f01dc062e50 con 0x7f01c0035db0 2024-04-27T03:21:12.752 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 3 to deep-scrub 2024-04-27T03:21:12.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f01c0035db0 msgr2=0x7f01c0038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.753 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f01c0035db0 0x7f01c0038270 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01d80015b0 tx=0x7f01d80079f0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc06d4d0 msgr2=0x7f01dc1c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc06d4d0 0x7f01dc1c62f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0ae040 tx=0x7f01cc01ac40 comp rx=0 tx=0).stop 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 shutdown_connections 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f01c0035db0 0x7f01c0038270 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 --2- 172.21.15.77:0/2667846804 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01dc06d4d0 0x7f01dc1c62f0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 >> 172.21.15.77:0/2667846804 conn(0x7f01dc099440 msgr2=0x7f01dc09b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 shutdown_connections 2024-04-27T03:21:12.754 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.742+0000 7f01e43fc640 1 -- 172.21.15.77:0/2667846804 wait complete. 2024-04-27T03:21:12.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:21:12.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f58ae1640 1 Processor -- start 2024-04-27T03:21:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f58ae1640 1 -- start start 2024-04-27T03:21:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f58ae1640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f8f5412cb10 0x7f8f541315c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f58ae1640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f54095fb0 con 0x7f8f5412c020 2024-04-27T03:21:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f58ae1640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f8f540968f0 con 0x7f8f5412cb10 2024-04-27T03:21:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f5259c640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f8f5412c020 0x7f8f5412c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:46788/0 (socket says 172.21.15.77:46788) 2024-04-27T03:21:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f5259c640 1 -- 172.21.15.77:0/3411710284 learned_addr learned my addr 172.21.15.77:0/3411710284 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:12.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f51d9b640 1 --2- 172.21.15.77:0/3411710284 >> v2:172.21.15.77:3300/0 conn(0x7f8f5412cb10 0x7f8f541315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f5159a640 1 -- 172.21.15.77:0/3411710284 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4146605426 0 0) 0x7f8f54095fb0 con 0x7f8f5412c020 2024-04-27T03:21:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f5159a640 1 -- 172.21.15.77:0/3411710284 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f3c003530 con 0x7f8f5412c020 2024-04-27T03:21:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f51d9b640 1 -- 172.21.15.77:0/3411710284 >> v1:172.21.15.77:6789/0 conn(0x7f8f5412c020 legacy=0x7f8f5412c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f51d9b640 1 -- 172.21.15.77:0/3411710284 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5406b470 con 0x7f8f5412cb10 2024-04-27T03:21:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.874+0000 7f8f51d9b640 1 --2- 172.21.15.77:0/3411710284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412cb10 0x7f8f541315c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009740 tx=0x7f8f4801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1ac18d9ccabb33 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5159a640 1 -- 172.21.15.77:0/3411710284 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f480b0070 con 0x7f8f5412cb10 2024-04-27T03:21:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5159a640 1 -- 172.21.15.77:0/3411710284 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f480ab530 con 0x7f8f5412cb10 2024-04-27T03:21:12.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5159a640 1 -- 172.21.15.77:0/3411710284 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f480ab810 con 0x7f8f5412cb10 2024-04-27T03:21:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/3411710284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412cb10 msgr2=0x7f8f541315c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 --2- 172.21.15.77:0/3411710284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412cb10 0x7f8f541315c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009740 tx=0x7f8f4801c370 comp rx=0 tx=0).stop 2024-04-27T03:21:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/3411710284 shutdown_connections 2024-04-27T03:21:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 --2- 172.21.15.77:0/3411710284 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412cb10 0x7f8f541315c0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/3411710284 >> 172.21.15.77:0/3411710284 conn(0x7f8f5409d040 msgr2=0x7f8f5409d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/3411710284 shutdown_connections 2024-04-27T03:21:12.888 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/3411710284 wait complete. 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 Processor -- start 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- start start 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412c020 0x7f8f541c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f8f54134530 con 0x7f8f5412c020 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412c020 0x7f8f541c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5259c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412c020 0x7f8f541c3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56944/0 (socket says 172.21.15.77:56944) 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5259c640 1 -- 172.21.15.77:0/94163908 learned_addr learned my addr 172.21.15.77:0/94163908 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5259c640 1 -- 172.21.15.77:0/94163908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f54124350 con 0x7f8f5412c020 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f5259c640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412c020 0x7f8f541c3b70 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00cb40 tx=0x7f8f4c0021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f42ffd640 1 -- 172.21.15.77:0/94163908 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f4c017070 con 0x7f8f5412c020 2024-04-27T03:21:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f42ffd640 1 -- 172.21.15.77:0/94163908 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f4c00ab80 con 0x7f8f5412c020 2024-04-27T03:21:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f54124fc0 con 0x7f8f5412c020 2024-04-27T03:21:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f42ffd640 1 -- 172.21.15.77:0/94163908 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f4c012400 con 0x7f8f5412c020 2024-04-27T03:21:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f54125680 con 0x7f8f5412c020 2024-04-27T03:21:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f42ffd640 1 -- 172.21.15.77:0/94163908 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8f4c01a070 con 0x7f8f5412c020 2024-04-27T03:21:12.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f54062e50 con 0x7f8f5412c020 2024-04-27T03:21:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f42ffd640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f34035de0 0x7f8f340382a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.878+0000 7f8f42ffd640 1 -- 172.21.15.77:0/94163908 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f8f4c048fe0 con 0x7f8f5412c020 2024-04-27T03:21:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.882+0000 7f8f51d9b640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f34035de0 0x7f8f340382a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:12.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.882+0000 7f8f51d9b640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f34035de0 0x7f8f340382a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8f480ad000 tx=0x7f8f480a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:12.895 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:12.886+0000 7f8f42ffd640 1 -- 172.21.15.77:0/94163908 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8f4c047350 con 0x7f8f5412c020 2024-04-27T03:21:13.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.030+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f541310c0 con 0x7f8f34035de0 2024-04-27T03:21:13.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.082+0000 7f8f42ffd640 1 -- 172.21.15.77:0/94163908 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+665696 (secure 0 0 0) 0x7f8f541310c0 con 0x7f8f34035de0 2024-04-27T03:21:13.096 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:21:13.104 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:21:13.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f34035de0 msgr2=0x7f8f340382a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f34035de0 0x7f8f340382a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8f480ad000 tx=0x7f8f480a6000 comp rx=0 tx=0).stop 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412c020 msgr2=0x7f8f541c3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412c020 0x7f8f541c3b70 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00cb40 tx=0x7f8f4c0021f0 comp rx=0 tx=0).stop 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 shutdown_connections 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f8f34035de0 0x7f8f340382a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 --2- 172.21.15.77:0/94163908 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f5412c020 0x7f8f541c3b70 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 >> 172.21.15.77:0/94163908 conn(0x7f8f5409d040 msgr2=0x7f8f54099650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 shutdown_connections 2024-04-27T03:21:13.106 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:13.090+0000 7f8f58ae1640 1 -- 172.21.15.77:0/94163908 wait complete. 2024-04-27T03:21:13.121 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-04-27T03:21:11.734570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":26412789,"num_objects":259,"num_object_clones":0,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":0,"num_read":5177,"num_read_kb":4722,"num_write":2298,"num_write_kb":61373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4672,"ondisk_log_size":4672,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":190324,"kb_used_data":40660,"kb_used_omap":6,"kb_used_meta":149625,"kb_avail":377297036,"statfs":{"total":386547056640,"available":386352164864,"internally_reserved":0,"allocated":41635840,"data_stored":40495458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":153216790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25811687,"num_objects":9,"num_object_clones":0,"num_object_copies":25,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":510,"num_read_kb":465,"num_write":386,"num_write_kb":29372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006763"},"pg_stats":[{"pgid":"3.3f","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741206+0000","last_change":"2024-04-27T03:17:47.025622+0000","last_active":"2024-04-27T03:19:52.741206+0000","last_peered":"2024-04-27T03:19:52.741206+0000","last_clean":"2024-04-27T03:19:52.741206+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:19:52.741206+0000","last_undegraded":"2024-04-27T03:19:52.741206+0000","last_fullsized":"2024-04-27T03:19:52.741206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:56.498788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'5","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246474+0000","last_change":"2024-04-27T03:17:49.019025+0000","last_active":"2024-04-27T03:21:01.246474+0000","last_peered":"2024-04-27T03:21:01.246474+0000","last_clean":"2024-04-27T03:21:01.246474+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:01.246474+0000","last_undegraded":"2024-04-27T03:21:01.246474+0000","last_fullsized":"2024-04-27T03:21:01.246474+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385062+0000","last_change":"2024-04-27T03:17:47.023458+0000","last_active":"2024-04-27T03:19:35.385062+0000","last_peered":"2024-04-27T03:19:35.385062+0000","last_clean":"2024-04-27T03:19:35.385062+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:19:35.385062+0000","last_undegraded":"2024-04-27T03:19:35.385062+0000","last_fullsized":"2024-04-27T03:19:35.385062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810083+0000","last_change":"2024-04-27T03:17:49.020940+0000","last_active":"2024-04-27T03:19:34.810083+0000","last_peered":"2024-04-27T03:19:34.810083+0000","last_clean":"2024-04-27T03:19:34.810083+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:19:34.810083+0000","last_undegraded":"2024-04-27T03:19:34.810083+0000","last_fullsized":"2024-04-27T03:19:34.810083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811946+0000","last_change":"2024-04-27T03:17:47.039594+0000","last_active":"2024-04-27T03:19:34.811946+0000","last_peered":"2024-04-27T03:19:34.811946+0000","last_clean":"2024-04-27T03:19:34.811946+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:19:34.811946+0000","last_undegraded":"2024-04-27T03:19:34.811946+0000","last_fullsized":"2024-04-27T03:19:34.811946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111615+0000","last_change":"2024-04-27T03:17:49.016480+0000","last_active":"2024-04-27T03:19:56.111615+0000","last_peered":"2024-04-27T03:19:56.111615+0000","last_clean":"2024-04-27T03:19:56.111615+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:19:56.111615+0000","last_undegraded":"2024-04-27T03:19:56.111615+0000","last_fullsized":"2024-04-27T03:19:56.111615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:11.652036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439532+0000","last_change":"2024-04-27T03:17:47.021409+0000","last_active":"2024-04-27T03:19:35.439532+0000","last_peered":"2024-04-27T03:19:35.439532+0000","last_clean":"2024-04-27T03:19:35.439532+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:19:35.439532+0000","last_undegraded":"2024-04-27T03:19:35.439532+0000","last_fullsized":"2024-04-27T03:19:35.439532+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:10:24.826641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948899+0000","last_change":"2024-04-27T03:17:49.021774+0000","last_active":"2024-04-27T03:21:00.948899+0000","last_peered":"2024-04-27T03:21:00.948899+0000","last_clean":"2024-04-27T03:21:00.948899+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:00.948899+0000","last_undegraded":"2024-04-27T03:21:00.948899+0000","last_fullsized":"2024-04-27T03:21:00.948899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817333+0000","last_change":"2024-04-27T03:17:47.015865+0000","last_active":"2024-04-27T03:19:34.817333+0000","last_peered":"2024-04-27T03:19:34.817333+0000","last_clean":"2024-04-27T03:19:34.817333+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:19:34.817333+0000","last_undegraded":"2024-04-27T03:19:34.817333+0000","last_fullsized":"2024-04-27T03:19:34.817333+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439003+0000","last_change":"2024-04-27T03:18:34.776780+0000","last_active":"2024-04-27T03:19:35.439003+0000","last_peered":"2024-04-27T03:19:35.439003+0000","last_clean":"2024-04-27T03:19:35.439003+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:19:35.439003+0000","last_undegraded":"2024-04-27T03:19:35.439003+0000","last_fullsized":"2024-04-27T03:19:35.439003+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:30.737857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.111562+0000","last_change":"2024-04-27T03:17:47.026206+0000","last_active":"2024-04-27T03:20:00.111562+0000","last_peered":"2024-04-27T03:20:00.111562+0000","last_clean":"2024-04-27T03:20:00.111562+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:20:00.111562+0000","last_undegraded":"2024-04-27T03:20:00.111562+0000","last_fullsized":"2024-04-27T03:20:00.111562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:13.937295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'11","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742400+0000","last_change":"2024-04-27T03:17:49.015210+0000","last_active":"2024-04-27T03:19:57.742400+0000","last_peered":"2024-04-27T03:19:57.742400+0000","last_clean":"2024-04-27T03:19:57.742400+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:19:57.742400+0000","last_undegraded":"2024-04-27T03:19:57.742400+0000","last_fullsized":"2024-04-27T03:19:57.742400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811270+0000","last_change":"2024-04-27T03:17:47.036460+0000","last_active":"2024-04-27T03:19:34.811270+0000","last_peered":"2024-04-27T03:19:34.811270+0000","last_clean":"2024-04-27T03:19:34.811270+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:19:34.811270+0000","last_undegraded":"2024-04-27T03:19:34.811270+0000","last_fullsized":"2024-04-27T03:19:34.811270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:56.284851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298837+0000","last_change":"2024-04-27T03:17:49.024190+0000","last_active":"2024-04-27T03:21:01.298837+0000","last_peered":"2024-04-27T03:21:01.298837+0000","last_clean":"2024-04-27T03:21:01.298837+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:01.298837+0000","last_undegraded":"2024-04-27T03:21:01.298837+0000","last_fullsized":"2024-04-27T03:21:01.298837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.094463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810250+0000","last_change":"2024-04-27T03:17:47.013877+0000","last_active":"2024-04-27T03:19:34.810250+0000","last_peered":"2024-04-27T03:19:34.810250+0000","last_clean":"2024-04-27T03:19:34.810250+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:19:34.810250+0000","last_undegraded":"2024-04-27T03:19:34.810250+0000","last_fullsized":"2024-04-27T03:19:34.810250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.444094+0000","last_change":"2024-04-27T03:17:49.020265+0000","last_active":"2024-04-27T03:19:35.444094+0000","last_peered":"2024-04-27T03:19:35.444094+0000","last_clean":"2024-04-27T03:19:35.444094+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:19:35.444094+0000","last_undegraded":"2024-04-27T03:19:35.444094+0000","last_fullsized":"2024-04-27T03:19:35.444094+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:23.713924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.298707+0000","last_change":"2024-04-27T03:17:49.019484+0000","last_active":"2024-04-27T03:21:01.298707+0000","last_peered":"2024-04-27T03:21:01.298707+0000","last_clean":"2024-04-27T03:21:01.298707+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:01.298707+0000","last_undegraded":"2024-04-27T03:21:01.298707+0000","last_fullsized":"2024-04-27T03:21:01.298707+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385569+0000","last_change":"2024-04-27T03:17:47.027209+0000","last_active":"2024-04-27T03:19:35.385569+0000","last_peered":"2024-04-27T03:19:35.385569+0000","last_clean":"2024-04-27T03:19:35.385569+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:19:35.385569+0000","last_undegraded":"2024-04-27T03:19:35.385569+0000","last_fullsized":"2024-04-27T03:19:35.385569+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814961+0000","last_change":"2024-04-27T03:17:49.013701+0000","last_active":"2024-04-27T03:19:34.814961+0000","last_peered":"2024-04-27T03:19:34.814961+0000","last_clean":"2024-04-27T03:19:34.814961+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:19:34.814961+0000","last_undegraded":"2024-04-27T03:19:34.814961+0000","last_fullsized":"2024-04-27T03:19:34.814961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.443009+0000","last_change":"2024-04-27T03:17:47.015536+0000","last_active":"2024-04-27T03:19:35.443009+0000","last_peered":"2024-04-27T03:19:35.443009+0000","last_clean":"2024-04-27T03:19:35.443009+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:19:35.443009+0000","last_undegraded":"2024-04-27T03:19:35.443009+0000","last_fullsized":"2024-04-27T03:19:35.443009+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:51:44.696219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.809825+0000","last_change":"2024-04-27T03:17:49.011747+0000","last_active":"2024-04-27T03:21:00.809825+0000","last_peered":"2024-04-27T03:21:00.809825+0000","last_clean":"2024-04-27T03:21:00.809825+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:21:00.809825+0000","last_undegraded":"2024-04-27T03:21:00.809825+0000","last_fullsized":"2024-04-27T03:21:00.809825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820563+0000","last_change":"2024-04-27T03:17:47.023848+0000","last_active":"2024-04-27T03:19:34.820563+0000","last_peered":"2024-04-27T03:19:34.820563+0000","last_clean":"2024-04-27T03:19:34.820563+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:19:34.820563+0000","last_undegraded":"2024-04-27T03:19:34.820563+0000","last_fullsized":"2024-04-27T03:19:34.820563+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.231670+0000","last_change":"2024-04-27T03:17:49.019463+0000","last_active":"2024-04-27T03:21:01.231670+0000","last_peered":"2024-04-27T03:21:01.231670+0000","last_clean":"2024-04-27T03:21:01.231670+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:01.231670+0000","last_undegraded":"2024-04-27T03:21:01.231670+0000","last_fullsized":"2024-04-27T03:21:01.231670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:53.154789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816701+0000","last_change":"2024-04-27T03:17:47.012402+0000","last_active":"2024-04-27T03:19:34.816701+0000","last_peered":"2024-04-27T03:19:34.816701+0000","last_clean":"2024-04-27T03:19:34.816701+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:19:34.816701+0000","last_undegraded":"2024-04-27T03:19:34.816701+0000","last_fullsized":"2024-04-27T03:19:34.816701+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146748+0000","last_change":"2024-04-27T03:17:49.008483+0000","last_active":"2024-04-27T03:21:01.146748+0000","last_peered":"2024-04-27T03:21:01.146748+0000","last_clean":"2024-04-27T03:21:01.146748+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:01.146748+0000","last_undegraded":"2024-04-27T03:21:01.146748+0000","last_fullsized":"2024-04-27T03:21:01.146748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:11.803924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812204+0000","last_change":"2024-04-27T03:17:47.036082+0000","last_active":"2024-04-27T03:19:34.812204+0000","last_peered":"2024-04-27T03:19:34.812204+0000","last_clean":"2024-04-27T03:19:34.812204+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:19:34.812204+0000","last_undegraded":"2024-04-27T03:19:34.812204+0000","last_fullsized":"2024-04-27T03:19:34.812204+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164153+0000","last_change":"2024-04-27T03:17:49.005822+0000","last_active":"2024-04-27T03:21:01.164153+0000","last_peered":"2024-04-27T03:21:01.164153+0000","last_clean":"2024-04-27T03:21:01.164153+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:01.164153+0000","last_undegraded":"2024-04-27T03:21:01.164153+0000","last_fullsized":"2024-04-27T03:21:01.164153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:40.042792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811583+0000","last_change":"2024-04-27T03:17:47.033466+0000","last_active":"2024-04-27T03:19:34.811583+0000","last_peered":"2024-04-27T03:19:34.811583+0000","last_clean":"2024-04-27T03:19:34.811583+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:19:34.811583+0000","last_undegraded":"2024-04-27T03:19:34.811583+0000","last_fullsized":"2024-04-27T03:19:34.811583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'37","reported_seq":105,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.190758+0000","last_change":"2024-04-27T03:17:57.088698+0000","last_active":"2024-04-27T03:21:01.190758+0000","last_peered":"2024-04-27T03:21:01.190758+0000","last_clean":"2024-04-27T03:21:01.190758+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:21:01.190758+0000","last_undegraded":"2024-04-27T03:21:01.190758+0000","last_fullsized":"2024-04-27T03:21:01.190758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:57.060752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233462+0000","last_change":"2024-04-27T03:17:49.015482+0000","last_active":"2024-04-27T03:21:01.233462+0000","last_peered":"2024-04-27T03:21:01.233462+0000","last_clean":"2024-04-27T03:21:01.233462+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:01.233462+0000","last_undegraded":"2024-04-27T03:21:01.233462+0000","last_fullsized":"2024-04-27T03:21:01.233462+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811777+0000","last_change":"2024-04-27T03:17:47.018824+0000","last_active":"2024-04-27T03:19:34.811777+0000","last_peered":"2024-04-27T03:19:34.811777+0000","last_clean":"2024-04-27T03:19:34.811777+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:19:34.811777+0000","last_undegraded":"2024-04-27T03:19:34.811777+0000","last_fullsized":"2024-04-27T03:19:34.811777+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815532+0000","last_change":"2024-04-27T03:17:51.032891+0000","last_active":"2024-04-27T03:19:34.815532+0000","last_peered":"2024-04-27T03:19:34.815532+0000","last_clean":"2024-04-27T03:19:34.815532+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:19:34.815532+0000","last_undegraded":"2024-04-27T03:19:34.815532+0000","last_fullsized":"2024-04-27T03:19:34.815532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'43","reported_seq":101,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.274412+0000","last_change":"2024-04-27T03:17:53.040688+0000","last_active":"2024-04-27T03:19:58.274412+0000","last_peered":"2024-04-27T03:19:58.274412+0000","last_clean":"2024-04-27T03:19:58.274412+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:19:58.274412+0000","last_undegraded":"2024-04-27T03:19:58.274412+0000","last_fullsized":"2024-04-27T03:19:58.274412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":24,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819799+0000","last_change":"2024-04-27T03:17:55.072849+0000","last_active":"2024-04-27T03:19:34.819799+0000","last_peered":"2024-04-27T03:19:34.819799+0000","last_clean":"2024-04-27T03:19:34.819799+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:19:34.819799+0000","last_undegraded":"2024-04-27T03:19:34.819799+0000","last_fullsized":"2024-04-27T03:19:34.819799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.848140+0000","last_change":"2024-04-27T03:17:57.061550+0000","last_active":"2024-04-27T03:21:00.848140+0000","last_peered":"2024-04-27T03:21:00.848140+0000","last_clean":"2024-04-27T03:21:00.848140+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:00.848140+0000","last_undegraded":"2024-04-27T03:21:00.848140+0000","last_fullsized":"2024-04-27T03:21:00.848140+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:54:11.141536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814870+0000","last_change":"2024-04-27T03:17:49.017246+0000","last_active":"2024-04-27T03:19:34.814870+0000","last_peered":"2024-04-27T03:19:34.814870+0000","last_clean":"2024-04-27T03:19:34.814870+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:19:34.814870+0000","last_undegraded":"2024-04-27T03:19:34.814870+0000","last_fullsized":"2024-04-27T03:19:34.814870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:12:15.229307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436921+0000","last_change":"2024-04-27T03:17:47.032159+0000","last_active":"2024-04-27T03:19:35.436921+0000","last_peered":"2024-04-27T03:19:35.436921+0000","last_clean":"2024-04-27T03:19:35.436921+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:19:35.436921+0000","last_undegraded":"2024-04-27T03:19:35.436921+0000","last_fullsized":"2024-04-27T03:19:35.436921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:43.992214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815036+0000","last_change":"2024-04-27T03:17:51.012211+0000","last_active":"2024-04-27T03:19:34.815036+0000","last_peered":"2024-04-27T03:19:34.815036+0000","last_clean":"2024-04-27T03:19:34.815036+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:19:34.815036+0000","last_undegraded":"2024-04-27T03:19:34.815036+0000","last_fullsized":"2024-04-27T03:19:34.815036+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"27'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384958+0000","last_change":"2024-04-27T03:17:53.049040+0000","last_active":"2024-04-27T03:19:35.384958+0000","last_peered":"2024-04-27T03:19:35.384958+0000","last_clean":"2024-04-27T03:19:35.384958+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:19:35.384958+0000","last_undegraded":"2024-04-27T03:19:35.384958+0000","last_fullsized":"2024-04-27T03:19:35.384958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:34.768461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819788+0000","last_change":"2024-04-27T03:17:55.058791+0000","last_active":"2024-04-27T03:19:34.819788+0000","last_peered":"2024-04-27T03:19:34.819788+0000","last_clean":"2024-04-27T03:19:34.819788+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:19:34.819788+0000","last_undegraded":"2024-04-27T03:19:34.819788+0000","last_fullsized":"2024-04-27T03:19:34.819788+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809843+0000","last_change":"2024-04-27T03:17:57.074831+0000","last_active":"2024-04-27T03:19:34.809843+0000","last_peered":"2024-04-27T03:19:34.809843+0000","last_clean":"2024-04-27T03:19:34.809843+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:19:34.809843+0000","last_undegraded":"2024-04-27T03:19:34.809843+0000","last_fullsized":"2024-04-27T03:19:34.809843+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'15","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.165698+0000","last_change":"2024-04-27T03:17:49.022969+0000","last_active":"2024-04-27T03:21:01.165698+0000","last_peered":"2024-04-27T03:21:01.165698+0000","last_clean":"2024-04-27T03:21:01.165698+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:01.165698+0000","last_undegraded":"2024-04-27T03:21:01.165698+0000","last_fullsized":"2024-04-27T03:21:01.165698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814358+0000","last_change":"2024-04-27T03:17:47.009721+0000","last_active":"2024-04-27T03:19:34.814358+0000","last_peered":"2024-04-27T03:19:34.814358+0000","last_clean":"2024-04-27T03:19:34.814358+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:19:34.814358+0000","last_undegraded":"2024-04-27T03:19:34.814358+0000","last_fullsized":"2024-04-27T03:19:34.814358+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814852+0000","last_change":"2024-04-27T03:17:51.034093+0000","last_active":"2024-04-27T03:19:34.814852+0000","last_peered":"2024-04-27T03:19:34.814852+0000","last_clean":"2024-04-27T03:19:34.814852+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:19:34.814852+0000","last_undegraded":"2024-04-27T03:19:34.814852+0000","last_fullsized":"2024-04-27T03:19:34.814852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"27'22","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440526+0000","last_change":"2024-04-27T03:17:53.032443+0000","last_active":"2024-04-27T03:19:35.440526+0000","last_peered":"2024-04-27T03:19:35.440526+0000","last_clean":"2024-04-27T03:19:35.440526+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:19:35.440526+0000","last_undegraded":"2024-04-27T03:19:35.440526+0000","last_fullsized":"2024-04-27T03:19:35.440526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:35.232648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'282","reported_seq":426,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277953+0000","last_change":"2024-04-27T03:17:55.069049+0000","last_active":"2024-04-27T03:21:01.277953+0000","last_peered":"2024-04-27T03:21:01.277953+0000","last_clean":"2024-04-27T03:21:01.277953+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:01.277953+0000","last_undegraded":"2024-04-27T03:21:01.277953+0000","last_fullsized":"2024-04-27T03:21:01.277953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438345+0000","last_change":"2024-04-27T03:17:57.072058+0000","last_active":"2024-04-27T03:19:35.438345+0000","last_peered":"2024-04-27T03:19:35.438345+0000","last_clean":"2024-04-27T03:19:35.438345+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:19:35.438345+0000","last_undegraded":"2024-04-27T03:19:35.438345+0000","last_fullsized":"2024-04-27T03:19:35.438345+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:56.749229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950160+0000","last_change":"2024-04-27T03:17:49.017573+0000","last_active":"2024-04-27T03:21:00.950160+0000","last_peered":"2024-04-27T03:21:00.950160+0000","last_clean":"2024-04-27T03:21:00.950160+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:00.950160+0000","last_undegraded":"2024-04-27T03:21:00.950160+0000","last_fullsized":"2024-04-27T03:21:00.950160+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:04.132727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813820+0000","last_change":"2024-04-27T03:17:47.003169+0000","last_active":"2024-04-27T03:19:34.813820+0000","last_peered":"2024-04-27T03:19:34.813820+0000","last_clean":"2024-04-27T03:19:34.813820+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:19:34.813820+0000","last_undegraded":"2024-04-27T03:19:34.813820+0000","last_fullsized":"2024-04-27T03:19:34.813820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822747+0000","last_change":"2024-04-27T03:17:51.020242+0000","last_active":"2024-04-27T03:19:34.822747+0000","last_peered":"2024-04-27T03:19:34.822747+0000","last_clean":"2024-04-27T03:19:34.822747+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:19:34.822747+0000","last_undegraded":"2024-04-27T03:19:34.822747+0000","last_fullsized":"2024-04-27T03:19:34.822747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'36","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.265097+0000","last_change":"2024-04-27T03:17:53.038866+0000","last_active":"2024-04-27T03:19:58.265097+0000","last_peered":"2024-04-27T03:19:58.265097+0000","last_clean":"2024-04-27T03:19:58.265097+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:19:58.265097+0000","last_undegraded":"2024-04-27T03:19:58.265097+0000","last_fullsized":"2024-04-27T03:19:58.265097+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:29:23.026213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'282","reported_seq":406,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.306917+0000","last_change":"2024-04-27T03:17:55.074657+0000","last_active":"2024-04-27T03:21:01.306917+0000","last_peered":"2024-04-27T03:21:01.306917+0000","last_clean":"2024-04-27T03:21:01.306917+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:01.306917+0000","last_undegraded":"2024-04-27T03:21:01.306917+0000","last_fullsized":"2024-04-27T03:21:01.306917+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:49:42.874318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'31","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.246782+0000","last_change":"2024-04-27T03:18:34.790257+0000","last_active":"2024-04-27T03:21:01.246782+0000","last_peered":"2024-04-27T03:21:01.246782+0000","last_clean":"2024-04-27T03:21:01.246782+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:21:01.246782+0000","last_undegraded":"2024-04-27T03:21:01.246782+0000","last_fullsized":"2024-04-27T03:21:01.246782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:11.176662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.716875+0000","last_change":"2024-04-27T03:17:47.013403+0000","last_active":"2024-04-27T03:21:00.716875+0000","last_peered":"2024-04-27T03:21:00.716875+0000","last_clean":"2024-04-27T03:21:00.716875+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:21:00.716875+0000","last_undegraded":"2024-04-27T03:21:00.716875+0000","last_fullsized":"2024-04-27T03:21:00.716875+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:47.849076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377293+0000","last_change":"2024-04-27T03:17:51.027190+0000","last_active":"2024-04-27T03:19:35.377293+0000","last_peered":"2024-04-27T03:19:35.377293+0000","last_clean":"2024-04-27T03:19:35.377293+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:19:35.377293+0000","last_undegraded":"2024-04-27T03:19:35.377293+0000","last_fullsized":"2024-04-27T03:19:35.377293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:50.708783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'47","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.221245+0000","last_change":"2024-04-27T03:17:53.060562+0000","last_active":"2024-04-27T03:20:00.221245+0000","last_peered":"2024-04-27T03:20:00.221245+0000","last_clean":"2024-04-27T03:20:00.221245+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:20:00.221245+0000","last_undegraded":"2024-04-27T03:20:00.221245+0000","last_fullsized":"2024-04-27T03:20:00.221245+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":53,"num_read_kb":41,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380552+0000","last_change":"2024-04-27T03:17:55.063723+0000","last_active":"2024-04-27T03:19:35.380552+0000","last_peered":"2024-04-27T03:19:35.380552+0000","last_clean":"2024-04-27T03:19:35.380552+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:19:35.380552+0000","last_undegraded":"2024-04-27T03:19:35.380552+0000","last_fullsized":"2024-04-27T03:19:35.380552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:59.616012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439014+0000","last_change":"2024-04-27T03:17:57.086011+0000","last_active":"2024-04-27T03:19:35.439014+0000","last_peered":"2024-04-27T03:19:35.439014+0000","last_clean":"2024-04-27T03:19:35.439014+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:19:35.439014+0000","last_undegraded":"2024-04-27T03:19:35.439014+0000","last_fullsized":"2024-04-27T03:19:35.439014+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:11.219539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299288+0000","last_change":"2024-04-27T03:17:49.003206+0000","last_active":"2024-04-27T03:21:01.299288+0000","last_peered":"2024-04-27T03:21:01.299288+0000","last_clean":"2024-04-27T03:21:01.299288+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:01.299288+0000","last_undegraded":"2024-04-27T03:21:01.299288+0000","last_fullsized":"2024-04-27T03:21:01.299288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:42.242631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820229+0000","last_change":"2024-04-27T03:17:46.995080+0000","last_active":"2024-04-27T03:19:34.820229+0000","last_peered":"2024-04-27T03:19:34.820229+0000","last_clean":"2024-04-27T03:19:34.820229+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:19:34.820229+0000","last_undegraded":"2024-04-27T03:19:34.820229+0000","last_fullsized":"2024-04-27T03:19:34.820229+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437771+0000","last_change":"2024-04-27T03:17:51.036969+0000","last_active":"2024-04-27T03:19:35.437771+0000","last_peered":"2024-04-27T03:19:35.437771+0000","last_clean":"2024-04-27T03:19:35.437771+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:19:35.437771+0000","last_undegraded":"2024-04-27T03:19:35.437771+0000","last_fullsized":"2024-04-27T03:19:35.437771+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:48:00.532358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'37","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.245893+0000","last_change":"2024-04-27T03:17:53.041431+0000","last_active":"2024-04-27T03:19:58.245893+0000","last_peered":"2024-04-27T03:19:58.245893+0000","last_clean":"2024-04-27T03:19:58.245893+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:19:58.245893+0000","last_undegraded":"2024-04-27T03:19:58.245893+0000","last_fullsized":"2024-04-27T03:19:58.245893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809982+0000","last_change":"2024-04-27T03:17:55.075008+0000","last_active":"2024-04-27T03:19:34.809982+0000","last_peered":"2024-04-27T03:19:34.809982+0000","last_clean":"2024-04-27T03:19:34.809982+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:19:34.809982+0000","last_undegraded":"2024-04-27T03:19:34.809982+0000","last_fullsized":"2024-04-27T03:19:34.809982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:01.297073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"29'5","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382753+0000","last_change":"2024-04-27T03:17:57.084457+0000","last_active":"2024-04-27T03:19:35.382753+0000","last_peered":"2024-04-27T03:19:35.382753+0000","last_clean":"2024-04-27T03:19:35.382753+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:19:35.382753+0000","last_undegraded":"2024-04-27T03:19:35.382753+0000","last_fullsized":"2024-04-27T03:19:35.382753+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:34.604396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.827907+0000","last_change":"2024-04-27T03:18:34.767220+0000","last_active":"2024-04-27T03:21:00.827907+0000","last_peered":"2024-04-27T03:21:00.827907+0000","last_clean":"2024-04-27T03:21:00.827907+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:00.827907+0000","last_undegraded":"2024-04-27T03:21:00.827907+0000","last_fullsized":"2024-04-27T03:21:00.827907+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:38.334837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820841+0000","last_change":"2024-04-27T03:17:47.028376+0000","last_active":"2024-04-27T03:19:34.820841+0000","last_peered":"2024-04-27T03:19:34.820841+0000","last_clean":"2024-04-27T03:19:34.820841+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:19:34.820841+0000","last_undegraded":"2024-04-27T03:19:34.820841+0000","last_fullsized":"2024-04-27T03:19:34.820841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:21:18.893441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438808+0000","last_change":"2024-04-27T03:17:51.037525+0000","last_active":"2024-04-27T03:19:35.438808+0000","last_peered":"2024-04-27T03:19:35.438808+0000","last_clean":"2024-04-27T03:19:35.438808+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:19:35.438808+0000","last_undegraded":"2024-04-27T03:19:35.438808+0000","last_fullsized":"2024-04-27T03:19:35.438808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:38.895777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'48","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.284523+0000","last_change":"2024-04-27T03:18:34.800954+0000","last_active":"2024-04-27T03:19:58.284523+0000","last_peered":"2024-04-27T03:19:58.284523+0000","last_clean":"2024-04-27T03:19:58.284523+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:19:58.284523+0000","last_undegraded":"2024-04-27T03:19:58.284523+0000","last_fullsized":"2024-04-27T03:19:58.284523+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:59.816977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'282","reported_seq":381,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.495106+0000","last_change":"2024-04-27T03:17:55.063480+0000","last_active":"2024-04-27T03:21:01.495106+0000","last_peered":"2024-04-27T03:21:01.495106+0000","last_clean":"2024-04-27T03:21:01.495106+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:01.495106+0000","last_undegraded":"2024-04-27T03:21:01.495106+0000","last_fullsized":"2024-04-27T03:21:01.495106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:25.995264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382973+0000","last_change":"2024-04-27T03:17:57.077044+0000","last_active":"2024-04-27T03:19:35.382973+0000","last_peered":"2024-04-27T03:19:35.382973+0000","last_clean":"2024-04-27T03:19:35.382973+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:19:35.382973+0000","last_undegraded":"2024-04-27T03:19:35.382973+0000","last_fullsized":"2024-04-27T03:19:35.382973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:24:03.311426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299661+0000","last_change":"2024-04-27T03:17:49.007911+0000","last_active":"2024-04-27T03:21:01.299661+0000","last_peered":"2024-04-27T03:21:01.299661+0000","last_clean":"2024-04-27T03:21:01.299661+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:01.299661+0000","last_undegraded":"2024-04-27T03:21:01.299661+0000","last_fullsized":"2024-04-27T03:21:01.299661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:07.206167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816213+0000","last_change":"2024-04-27T03:18:34.772611+0000","last_active":"2024-04-27T03:19:34.816213+0000","last_peered":"2024-04-27T03:19:34.816213+0000","last_clean":"2024-04-27T03:19:34.816213+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:19:34.816213+0000","last_undegraded":"2024-04-27T03:19:34.816213+0000","last_fullsized":"2024-04-27T03:19:34.816213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:30:47.500325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437638+0000","last_change":"2024-04-27T03:17:51.038976+0000","last_active":"2024-04-27T03:19:35.437638+0000","last_peered":"2024-04-27T03:19:35.437638+0000","last_clean":"2024-04-27T03:19:35.437638+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:19:35.437638+0000","last_undegraded":"2024-04-27T03:19:35.437638+0000","last_fullsized":"2024-04-27T03:19:35.437638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:17:48.170469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'66","reported_seq":138,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.287412+0000","last_change":"2024-04-27T03:17:53.050762+0000","last_active":"2024-04-27T03:19:58.287412+0000","last_peered":"2024-04-27T03:19:58.287412+0000","last_clean":"2024-04-27T03:19:58.287412+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:19:58.287412+0000","last_undegraded":"2024-04-27T03:19:58.287412+0000","last_fullsized":"2024-04-27T03:19:58.287412+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":67,"num_read_kb":56,"num_write":30,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436043+0000","last_change":"2024-04-27T03:17:55.057576+0000","last_active":"2024-04-27T03:19:35.436043+0000","last_peered":"2024-04-27T03:19:35.436043+0000","last_clean":"2024-04-27T03:19:35.436043+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:19:35.436043+0000","last_undegraded":"2024-04-27T03:19:35.436043+0000","last_fullsized":"2024-04-27T03:19:35.436043+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:41.241891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'18","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.456884+0000","last_change":"2024-04-27T03:17:57.062348+0000","last_active":"2024-04-27T03:21:01.456884+0000","last_peered":"2024-04-27T03:21:01.456884+0000","last_clean":"2024-04-27T03:21:01.456884+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:01.456884+0000","last_undegraded":"2024-04-27T03:21:01.456884+0000","last_fullsized":"2024-04-27T03:21:01.456884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:14.902153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232625+0000","last_change":"2024-04-27T03:17:49.009833+0000","last_active":"2024-04-27T03:21:01.232625+0000","last_peered":"2024-04-27T03:21:01.232625+0000","last_clean":"2024-04-27T03:21:01.232625+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:01.232625+0000","last_undegraded":"2024-04-27T03:21:01.232625+0000","last_fullsized":"2024-04-27T03:21:01.232625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:36:59.530315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.437653+0000","last_change":"2024-04-27T03:18:34.767518+0000","last_active":"2024-04-27T03:19:35.437653+0000","last_peered":"2024-04-27T03:19:35.437653+0000","last_clean":"2024-04-27T03:19:35.437653+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:19:35.437653+0000","last_undegraded":"2024-04-27T03:19:35.437653+0000","last_fullsized":"2024-04-27T03:19:35.437653+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:02.579718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435824+0000","last_change":"2024-04-27T03:17:51.029195+0000","last_active":"2024-04-27T03:19:35.435824+0000","last_peered":"2024-04-27T03:19:35.435824+0000","last_clean":"2024-04-27T03:19:35.435824+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:19:35.435824+0000","last_undegraded":"2024-04-27T03:19:35.435824+0000","last_fullsized":"2024-04-27T03:19:35.435824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:42.059879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'40","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.253858+0000","last_change":"2024-04-27T03:17:53.049067+0000","last_active":"2024-04-27T03:19:58.253858+0000","last_peered":"2024-04-27T03:19:58.253858+0000","last_clean":"2024-04-27T03:19:58.253858+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:19:58.253858+0000","last_undegraded":"2024-04-27T03:19:58.253858+0000","last_fullsized":"2024-04-27T03:19:58.253858+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:55:37.400674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":54,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815163+0000","last_change":"2024-04-27T03:17:55.072520+0000","last_active":"2024-04-27T03:19:34.815163+0000","last_peered":"2024-04-27T03:19:34.815163+0000","last_clean":"2024-04-27T03:19:34.815163+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:19:34.815163+0000","last_undegraded":"2024-04-27T03:19:34.815163+0000","last_fullsized":"2024-04-27T03:19:34.815163+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'3","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.883363+0000","last_change":"2024-04-27T03:17:57.080943+0000","last_active":"2024-04-27T03:19:57.883363+0000","last_peered":"2024-04-27T03:19:57.883363+0000","last_clean":"2024-04-27T03:19:57.883363+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:19:57.883363+0000","last_undegraded":"2024-04-27T03:19:57.883363+0000","last_fullsized":"2024-04-27T03:19:57.883363+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822407+0000","last_change":"2024-04-27T03:17:49.016254+0000","last_active":"2024-04-27T03:19:34.822407+0000","last_peered":"2024-04-27T03:19:34.822407+0000","last_clean":"2024-04-27T03:19:34.822407+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:19:34.822407+0000","last_undegraded":"2024-04-27T03:19:34.822407+0000","last_fullsized":"2024-04-27T03:19:34.822407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:42:04.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817143+0000","last_change":"2024-04-27T03:17:47.026419+0000","last_active":"2024-04-27T03:19:34.817143+0000","last_peered":"2024-04-27T03:19:34.817143+0000","last_clean":"2024-04-27T03:19:34.817143+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:19:34.817143+0000","last_undegraded":"2024-04-27T03:19:34.817143+0000","last_fullsized":"2024-04-27T03:19:34.817143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:08:25.121818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377466+0000","last_change":"2024-04-27T03:17:51.036820+0000","last_active":"2024-04-27T03:19:35.377466+0000","last_peered":"2024-04-27T03:19:35.377466+0000","last_clean":"2024-04-27T03:19:35.377466+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:19:35.377466+0000","last_undegraded":"2024-04-27T03:19:35.377466+0000","last_fullsized":"2024-04-27T03:19:35.377466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:24.012671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.298340+0000","last_change":"2024-04-27T03:17:53.043412+0000","last_active":"2024-04-27T03:19:58.298340+0000","last_peered":"2024-04-27T03:19:58.298340+0000","last_clean":"2024-04-27T03:19:58.298340+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:19:58.298340+0000","last_undegraded":"2024-04-27T03:19:58.298340+0000","last_fullsized":"2024-04-27T03:19:58.298340+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:37:03.560918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'282","reported_seq":408,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.467183+0000","last_change":"2024-04-27T03:17:55.070428+0000","last_active":"2024-04-27T03:21:01.467183+0000","last_peered":"2024-04-27T03:21:01.467183+0000","last_clean":"2024-04-27T03:21:01.467183+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:01.467183+0000","last_undegraded":"2024-04-27T03:21:01.467183+0000","last_fullsized":"2024-04-27T03:21:01.467183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:20.977192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'12","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.727848+0000","last_change":"2024-04-27T03:17:57.077926+0000","last_active":"2024-04-27T03:20:02.727848+0000","last_peered":"2024-04-27T03:20:02.727848+0000","last_clean":"2024-04-27T03:20:02.727848+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:20:02.727848+0000","last_undegraded":"2024-04-27T03:20:02.727848+0000","last_fullsized":"2024-04-27T03:20:02.727848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:14:58.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.441233+0000","last_change":"2024-04-27T03:17:49.008951+0000","last_active":"2024-04-27T03:19:35.441233+0000","last_peered":"2024-04-27T03:19:35.441233+0000","last_clean":"2024-04-27T03:19:35.441233+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:19:35.441233+0000","last_undegraded":"2024-04-27T03:19:35.441233+0000","last_fullsized":"2024-04-27T03:19:35.441233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:33:15.953778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810391+0000","last_change":"2024-04-27T03:17:47.036712+0000","last_active":"2024-04-27T03:19:34.810391+0000","last_peered":"2024-04-27T03:19:34.810391+0000","last_clean":"2024-04-27T03:19:34.810391+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:19:34.810391+0000","last_undegraded":"2024-04-27T03:19:34.810391+0000","last_fullsized":"2024-04-27T03:19:34.810391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'3","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724954+0000","last_change":"2024-04-27T03:17:51.037543+0000","last_active":"2024-04-27T03:19:59.724954+0000","last_peered":"2024-04-27T03:19:59.724954+0000","last_clean":"2024-04-27T03:19:59.724954+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:19:59.724954+0000","last_undegraded":"2024-04-27T03:19:59.724954+0000","last_fullsized":"2024-04-27T03:19:59.724954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:36.305408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":381,"num_read_kb":381,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"27'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814593+0000","last_change":"2024-04-27T03:17:53.042236+0000","last_active":"2024-04-27T03:19:34.814593+0000","last_peered":"2024-04-27T03:19:34.814593+0000","last_clean":"2024-04-27T03:19:34.814593+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:19:34.814593+0000","last_undegraded":"2024-04-27T03:19:34.814593+0000","last_fullsized":"2024-04-27T03:19:34.814593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:31:24.373609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'282","reported_seq":464,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.408184+0000","last_change":"2024-04-27T03:17:55.068988+0000","last_active":"2024-04-27T03:21:01.408184+0000","last_peered":"2024-04-27T03:21:01.408184+0000","last_clean":"2024-04-27T03:21:01.408184+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:01.408184+0000","last_undegraded":"2024-04-27T03:21:01.408184+0000","last_fullsized":"2024-04-27T03:21:01.408184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.880007+0000","last_change":"2024-04-27T03:17:57.074965+0000","last_active":"2024-04-27T03:19:59.880007+0000","last_peered":"2024-04-27T03:19:59.880007+0000","last_clean":"2024-04-27T03:19:59.880007+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:19:59.880007+0000","last_undegraded":"2024-04-27T03:19:59.880007+0000","last_fullsized":"2024-04-27T03:19:59.880007+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:30.053117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386155+0000","last_change":"2024-04-27T03:17:49.018519+0000","last_active":"2024-04-27T03:19:35.386155+0000","last_peered":"2024-04-27T03:19:35.386155+0000","last_clean":"2024-04-27T03:19:35.386155+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:19:35.386155+0000","last_undegraded":"2024-04-27T03:19:35.386155+0000","last_fullsized":"2024-04-27T03:19:35.386155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:11:46.730077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809539+0000","last_change":"2024-04-27T03:17:47.031891+0000","last_active":"2024-04-27T03:19:34.809539+0000","last_peered":"2024-04-27T03:19:34.809539+0000","last_clean":"2024-04-27T03:19:34.809539+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:19:34.809539+0000","last_undegraded":"2024-04-27T03:19:34.809539+0000","last_fullsized":"2024-04-27T03:19:34.809539+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:40:12.321826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815512+0000","last_change":"2024-04-27T03:17:51.033741+0000","last_active":"2024-04-27T03:19:34.815512+0000","last_peered":"2024-04-27T03:19:34.815512+0000","last_clean":"2024-04-27T03:19:34.815512+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:19:34.815512+0000","last_undegraded":"2024-04-27T03:19:34.815512+0000","last_fullsized":"2024-04-27T03:19:34.815512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:10.915031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'38","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.276705+0000","last_change":"2024-04-27T03:17:53.048795+0000","last_active":"2024-04-27T03:19:58.276705+0000","last_peered":"2024-04-27T03:19:58.276705+0000","last_clean":"2024-04-27T03:19:58.276705+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:19:58.276705+0000","last_undegraded":"2024-04-27T03:19:58.276705+0000","last_fullsized":"2024-04-27T03:19:58.276705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:16.903014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817470+0000","last_change":"2024-04-27T03:17:55.071142+0000","last_active":"2024-04-27T03:19:34.817470+0000","last_peered":"2024-04-27T03:19:34.817470+0000","last_clean":"2024-04-27T03:19:34.817470+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:19:34.817470+0000","last_undegraded":"2024-04-27T03:19:34.817470+0000","last_fullsized":"2024-04-27T03:19:34.817470+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:27:41.712670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'8","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.368117+0000","last_change":"2024-04-27T03:17:57.071496+0000","last_active":"2024-04-27T03:21:01.368117+0000","last_peered":"2024-04-27T03:21:01.368117+0000","last_clean":"2024-04-27T03:21:01.368117+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:01.368117+0000","last_undegraded":"2024-04-27T03:21:01.368117+0000","last_fullsized":"2024-04-27T03:21:01.368117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:32:20.548838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.811995+0000","last_change":"2024-04-27T03:17:49.013855+0000","last_active":"2024-04-27T03:21:00.811995+0000","last_peered":"2024-04-27T03:21:00.811995+0000","last_clean":"2024-04-27T03:21:00.811995+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:00.811995+0000","last_undegraded":"2024-04-27T03:21:00.811995+0000","last_fullsized":"2024-04-27T03:21:00.811995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":201,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379585+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:19:35.379585+0000","last_peered":"2024-04-27T03:19:35.379585+0000","last_clean":"2024-04-27T03:19:35.379585+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:19:35.379585+0000","last_undegraded":"2024-04-27T03:19:35.379585+0000","last_fullsized":"2024-04-27T03:19:35.379585+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816708+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:19:34.816708+0000","last_peered":"2024-04-27T03:19:34.816708+0000","last_clean":"2024-04-27T03:19:34.816708+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:19:34.816708+0000","last_undegraded":"2024-04-27T03:19:34.816708+0000","last_fullsized":"2024-04-27T03:19:34.816708+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:25.777031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.100108+0000","last_change":"2024-04-27T03:17:47.039911+0000","last_active":"2024-04-27T03:21:01.100108+0000","last_peered":"2024-04-27T03:21:01.100108+0000","last_clean":"2024-04-27T03:21:01.100108+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:21:01.100108+0000","last_undegraded":"2024-04-27T03:21:01.100108+0000","last_fullsized":"2024-04-27T03:21:01.100108+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382198+0000","last_change":"2024-04-27T03:17:51.036961+0000","last_active":"2024-04-27T03:19:35.382198+0000","last_peered":"2024-04-27T03:19:35.382198+0000","last_clean":"2024-04-27T03:19:35.382198+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:19:35.382198+0000","last_undegraded":"2024-04-27T03:19:35.382198+0000","last_fullsized":"2024-04-27T03:19:35.382198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:16.046086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'82","reported_seq":161,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.296556+0000","last_change":"2024-04-27T03:17:53.051108+0000","last_active":"2024-04-27T03:19:58.296556+0000","last_peered":"2024-04-27T03:19:58.296556+0000","last_clean":"2024-04-27T03:19:58.296556+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:19:58.296556+0000","last_undegraded":"2024-04-27T03:19:58.296556+0000","last_fullsized":"2024-04-27T03:19:58.296556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:32:18.585035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":63,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812224+0000","last_change":"2024-04-27T03:17:55.071697+0000","last_active":"2024-04-27T03:19:34.812224+0000","last_peered":"2024-04-27T03:19:34.812224+0000","last_clean":"2024-04-27T03:19:34.812224+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:19:34.812224+0000","last_undegraded":"2024-04-27T03:19:34.812224+0000","last_fullsized":"2024-04-27T03:19:34.812224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":167,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.324346+0000","last_change":"2024-04-27T03:17:57.075459+0000","last_active":"2024-04-27T03:21:01.324346+0000","last_peered":"2024-04-27T03:21:01.324346+0000","last_clean":"2024-04-27T03:21:01.324346+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:01.324346+0000","last_undegraded":"2024-04-27T03:21:01.324346+0000","last_fullsized":"2024-04-27T03:21:01.324346+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:41:22.086532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.233419+0000","last_change":"2024-04-27T03:17:49.007601+0000","last_active":"2024-04-27T03:21:01.233419+0000","last_peered":"2024-04-27T03:21:01.233419+0000","last_clean":"2024-04-27T03:21:01.233419+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:21:01.233419+0000","last_undegraded":"2024-04-27T03:21:01.233419+0000","last_fullsized":"2024-04-27T03:21:01.233419+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:05:14.934011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815344+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:19:34.815344+0000","last_peered":"2024-04-27T03:19:34.815344+0000","last_clean":"2024-04-27T03:19:34.815344+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:19:34.815344+0000","last_undegraded":"2024-04-27T03:19:34.815344+0000","last_fullsized":"2024-04-27T03:19:34.815344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:47:23.800651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.380638+0000","last_change":"2024-04-27T03:17:47.015528+0000","last_active":"2024-04-27T03:19:35.380638+0000","last_peered":"2024-04-27T03:19:35.380638+0000","last_clean":"2024-04-27T03:19:35.380638+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:19:35.380638+0000","last_undegraded":"2024-04-27T03:19:35.380638+0000","last_fullsized":"2024-04-27T03:19:35.380638+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822099+0000","last_change":"2024-04-27T03:17:51.034048+0000","last_active":"2024-04-27T03:19:34.822099+0000","last_peered":"2024-04-27T03:19:34.822099+0000","last_clean":"2024-04-27T03:19:34.822099+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:19:34.822099+0000","last_undegraded":"2024-04-27T03:19:34.822099+0000","last_fullsized":"2024-04-27T03:19:34.822099+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:53.128444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'279","reported_seq":493,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.279126+0000","last_change":"2024-04-27T03:17:53.058256+0000","last_active":"2024-04-27T03:21:01.279126+0000","last_peered":"2024-04-27T03:21:01.279126+0000","last_clean":"2024-04-27T03:21:01.279126+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:01.279126+0000","last_undegraded":"2024-04-27T03:21:01.279126+0000","last_fullsized":"2024-04-27T03:21:01.279126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:08.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":366,"num_read_kb":350,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811204+0000","last_change":"2024-04-27T03:17:55.074606+0000","last_active":"2024-04-27T03:19:34.811204+0000","last_peered":"2024-04-27T03:19:34.811204+0000","last_clean":"2024-04-27T03:19:34.811204+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:19:34.811204+0000","last_undegraded":"2024-04-27T03:19:34.811204+0000","last_fullsized":"2024-04-27T03:19:34.811204+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:48:18.392300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379830+0000","last_change":"2024-04-27T03:17:57.086888+0000","last_active":"2024-04-27T03:19:35.379830+0000","last_peered":"2024-04-27T03:19:35.379830+0000","last_clean":"2024-04-27T03:19:35.379830+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:19:35.379830+0000","last_undegraded":"2024-04-27T03:19:35.379830+0000","last_fullsized":"2024-04-27T03:19:35.379830+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:30.189251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.232768+0000","last_change":"2024-04-27T03:17:49.014111+0000","last_active":"2024-04-27T03:21:01.232768+0000","last_peered":"2024-04-27T03:21:01.232768+0000","last_clean":"2024-04-27T03:21:01.232768+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:01.232768+0000","last_undegraded":"2024-04-27T03:21:01.232768+0000","last_fullsized":"2024-04-27T03:21:01.232768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:25:15.967813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436752+0000","last_change":"2024-04-27T03:17:39.633841+0000","last_active":"2024-04-27T03:19:35.436752+0000","last_peered":"2024-04-27T03:19:35.436752+0000","last_clean":"2024-04-27T03:19:35.436752+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:19:35.436752+0000","last_undegraded":"2024-04-27T03:19:35.436752+0000","last_fullsized":"2024-04-27T03:19:35.436752+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:39.585874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.559603+0000","last_change":"2024-04-27T03:17:47.011695+0000","last_active":"2024-04-27T03:21:00.559603+0000","last_peered":"2024-04-27T03:21:00.559603+0000","last_clean":"2024-04-27T03:21:00.559603+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:21:00.559603+0000","last_undegraded":"2024-04-27T03:21:00.559603+0000","last_fullsized":"2024-04-27T03:21:00.559603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:11:37.606805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"20'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.435807+0000","last_change":"2024-04-27T03:17:51.031676+0000","last_active":"2024-04-27T03:19:35.435807+0000","last_peered":"2024-04-27T03:19:35.435807+0000","last_clean":"2024-04-27T03:19:35.435807+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:19:35.435807+0000","last_undegraded":"2024-04-27T03:19:35.435807+0000","last_fullsized":"2024-04-27T03:19:35.435807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:23.681601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'16","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.278287+0000","last_change":"2024-04-27T03:17:53.051185+0000","last_active":"2024-04-27T03:21:01.278287+0000","last_peered":"2024-04-27T03:21:01.278287+0000","last_clean":"2024-04-27T03:21:01.278287+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:01.278287+0000","last_undegraded":"2024-04-27T03:21:01.278287+0000","last_fullsized":"2024-04-27T03:21:01.278287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382273+0000","last_change":"2024-04-27T03:17:55.070180+0000","last_active":"2024-04-27T03:19:35.382273+0000","last_peered":"2024-04-27T03:19:35.382273+0000","last_clean":"2024-04-27T03:19:35.382273+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:19:35.382273+0000","last_undegraded":"2024-04-27T03:19:35.382273+0000","last_fullsized":"2024-04-27T03:19:35.382273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:15.975975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.206940+0000","last_change":"2024-04-27T03:17:57.071109+0000","last_active":"2024-04-27T03:21:01.206940+0000","last_peered":"2024-04-27T03:21:01.206940+0000","last_clean":"2024-04-27T03:21:01.206940+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:01.206940+0000","last_undegraded":"2024-04-27T03:21:01.206940+0000","last_fullsized":"2024-04-27T03:21:01.206940+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802933+0000","last_change":"2024-04-27T03:17:49.009764+0000","last_active":"2024-04-27T03:21:00.802933+0000","last_peered":"2024-04-27T03:21:00.802933+0000","last_clean":"2024-04-27T03:21:00.802933+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:00.802933+0000","last_undegraded":"2024-04-27T03:21:00.802933+0000","last_fullsized":"2024-04-27T03:21:00.802933+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:57:31.202044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816459+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:19:34.816459+0000","last_peered":"2024-04-27T03:19:34.816459+0000","last_clean":"2024-04-27T03:19:34.816459+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:19:34.816459+0000","last_undegraded":"2024-04-27T03:19:34.816459+0000","last_fullsized":"2024-04-27T03:19:34.816459+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:37.354698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.133613+0000","last_change":"2024-04-27T03:17:47.015018+0000","last_active":"2024-04-27T03:21:01.133613+0000","last_peered":"2024-04-27T03:21:01.133613+0000","last_clean":"2024-04-27T03:21:01.133613+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:21:01.133613+0000","last_undegraded":"2024-04-27T03:21:01.133613+0000","last_fullsized":"2024-04-27T03:21:01.133613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.378112+0000","last_change":"2024-04-27T03:17:51.029533+0000","last_active":"2024-04-27T03:19:35.378112+0000","last_peered":"2024-04-27T03:19:35.378112+0000","last_clean":"2024-04-27T03:19:35.378112+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:19:35.378112+0000","last_undegraded":"2024-04-27T03:19:35.378112+0000","last_fullsized":"2024-04-27T03:19:35.378112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:52.883679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"29'14","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.236793+0000","last_change":"2024-04-27T03:17:53.045958+0000","last_active":"2024-04-27T03:21:00.236793+0000","last_peered":"2024-04-27T03:21:00.236793+0000","last_clean":"2024-04-27T03:21:00.236793+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:00.236793+0000","last_undegraded":"2024-04-27T03:21:00.236793+0000","last_fullsized":"2024-04-27T03:21:00.236793+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815623+0000","last_change":"2024-04-27T03:17:55.078514+0000","last_active":"2024-04-27T03:19:34.815623+0000","last_peered":"2024-04-27T03:19:34.815623+0000","last_clean":"2024-04-27T03:19:34.815623+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:19:34.815623+0000","last_undegraded":"2024-04-27T03:19:34.815623+0000","last_fullsized":"2024-04-27T03:19:34.815623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:25.561786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.953374+0000","last_change":"2024-04-27T03:17:57.089242+0000","last_active":"2024-04-27T03:21:00.953374+0000","last_peered":"2024-04-27T03:21:00.953374+0000","last_clean":"2024-04-27T03:21:00.953374+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:21:00.953374+0000","last_undegraded":"2024-04-27T03:21:00.953374+0000","last_fullsized":"2024-04-27T03:21:00.953374+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:55:12.686857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299342+0000","last_change":"2024-04-27T03:17:49.006116+0000","last_active":"2024-04-27T03:21:01.299342+0000","last_peered":"2024-04-27T03:21:01.299342+0000","last_clean":"2024-04-27T03:21:01.299342+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:01.299342+0000","last_undegraded":"2024-04-27T03:21:01.299342+0000","last_fullsized":"2024-04-27T03:21:01.299342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:05.838106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.822302+0000","last_change":"2024-04-27T03:17:47.024776+0000","last_active":"2024-04-27T03:19:34.822302+0000","last_peered":"2024-04-27T03:19:34.822302+0000","last_clean":"2024-04-27T03:19:34.822302+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:19:34.822302+0000","last_undegraded":"2024-04-27T03:19:34.822302+0000","last_fullsized":"2024-04-27T03:19:34.822302+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:43.442098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382818+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:19:35.382818+0000","last_peered":"2024-04-27T03:19:35.382818+0000","last_clean":"2024-04-27T03:19:35.382818+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:19:35.382818+0000","last_undegraded":"2024-04-27T03:19:35.382818+0000","last_fullsized":"2024-04-27T03:19:35.382818+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:01:48.896188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.733547+0000","last_change":"2024-04-27T03:17:51.037696+0000","last_active":"2024-04-27T03:19:57.733547+0000","last_peered":"2024-04-27T03:19:57.733547+0000","last_clean":"2024-04-27T03:19:57.733547+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:19:57.733547+0000","last_undegraded":"2024-04-27T03:19:57.733547+0000","last_fullsized":"2024-04-27T03:19:57.733547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:44.259718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'73","reported_seq":148,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.297267+0000","last_change":"2024-04-27T03:17:53.054938+0000","last_active":"2024-04-27T03:19:58.297267+0000","last_peered":"2024-04-27T03:19:58.297267+0000","last_clean":"2024-04-27T03:19:58.297267+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:19:58.297267+0000","last_undegraded":"2024-04-27T03:19:58.297267+0000","last_fullsized":"2024-04-27T03:19:58.297267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:38.318924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":77,"num_read_kb":63,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815814+0000","last_change":"2024-04-27T03:17:55.068676+0000","last_active":"2024-04-27T03:19:34.815814+0000","last_peered":"2024-04-27T03:19:34.815814+0000","last_clean":"2024-04-27T03:19:34.815814+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:19:34.815814+0000","last_undegraded":"2024-04-27T03:19:34.815814+0000","last_fullsized":"2024-04-27T03:19:34.815814+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:04:55.186362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.082757+0000","last_change":"2024-04-27T03:17:57.068043+0000","last_active":"2024-04-27T03:20:00.082757+0000","last_peered":"2024-04-27T03:20:00.082757+0000","last_clean":"2024-04-27T03:20:00.082757+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:20:00.082757+0000","last_undegraded":"2024-04-27T03:20:00.082757+0000","last_fullsized":"2024-04-27T03:20:00.082757+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:14.279033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382103+0000","last_change":"2024-04-27T03:17:49.009882+0000","last_active":"2024-04-27T03:19:35.382103+0000","last_peered":"2024-04-27T03:19:35.382103+0000","last_clean":"2024-04-27T03:19:35.382103+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:19:35.382103+0000","last_undegraded":"2024-04-27T03:19:35.382103+0000","last_fullsized":"2024-04-27T03:19:35.382103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:53:36.878689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.286999+0000","last_change":"2024-04-27T03:17:47.030065+0000","last_active":"2024-04-27T03:21:01.286999+0000","last_peered":"2024-04-27T03:21:01.286999+0000","last_clean":"2024-04-27T03:21:01.286999+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:01.286999+0000","last_undegraded":"2024-04-27T03:21:01.286999+0000","last_fullsized":"2024-04-27T03:21:01.286999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:45:21.469900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":21,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820336+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:19:34.820336+0000","last_peered":"2024-04-27T03:19:34.820336+0000","last_clean":"2024-04-27T03:19:34.820336+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:19:34.820336+0000","last_undegraded":"2024-04-27T03:19:34.820336+0000","last_fullsized":"2024-04-27T03:19:34.820336+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:48:41.511344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816118+0000","last_change":"2024-04-27T03:17:51.035808+0000","last_active":"2024-04-27T03:19:34.816118+0000","last_peered":"2024-04-27T03:19:34.816118+0000","last_clean":"2024-04-27T03:19:34.816118+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:19:34.816118+0000","last_undegraded":"2024-04-27T03:19:34.816118+0000","last_fullsized":"2024-04-27T03:19:34.816118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"27'20","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381588+0000","last_change":"2024-04-27T03:17:53.049471+0000","last_active":"2024-04-27T03:19:35.381588+0000","last_peered":"2024-04-27T03:19:35.381588+0000","last_clean":"2024-04-27T03:19:35.381588+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:19:35.381588+0000","last_undegraded":"2024-04-27T03:19:35.381588+0000","last_fullsized":"2024-04-27T03:19:35.381588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:42.365441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'282","reported_seq":361,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.481977+0000","last_change":"2024-04-27T03:17:55.077154+0000","last_active":"2024-04-27T03:21:01.481977+0000","last_peered":"2024-04-27T03:21:01.481977+0000","last_clean":"2024-04-27T03:21:01.481977+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:01.481977+0000","last_undegraded":"2024-04-27T03:21:01.481977+0000","last_fullsized":"2024-04-27T03:21:01.481977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:55:52.856782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.236855+0000","last_change":"2024-04-27T03:17:57.091210+0000","last_active":"2024-04-27T03:21:01.236855+0000","last_peered":"2024-04-27T03:21:01.236855+0000","last_clean":"2024-04-27T03:21:01.236855+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:21:01.236855+0000","last_undegraded":"2024-04-27T03:21:01.236855+0000","last_fullsized":"2024-04-27T03:21:01.236855+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:08:12.217298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.948922+0000","last_change":"2024-04-27T03:17:49.007942+0000","last_active":"2024-04-27T03:21:00.948922+0000","last_peered":"2024-04-27T03:21:00.948922+0000","last_clean":"2024-04-27T03:21:00.948922+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:21:00.948922+0000","last_undegraded":"2024-04-27T03:21:00.948922+0000","last_fullsized":"2024-04-27T03:21:00.948922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815379+0000","last_change":"2024-04-27T03:17:47.039532+0000","last_active":"2024-04-27T03:19:34.815379+0000","last_peered":"2024-04-27T03:19:34.815379+0000","last_clean":"2024-04-27T03:19:34.815379+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:19:34.815379+0000","last_undegraded":"2024-04-27T03:19:34.815379+0000","last_fullsized":"2024-04-27T03:19:34.815379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384884+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:19:35.384884+0000","last_peered":"2024-04-27T03:19:35.384884+0000","last_clean":"2024-04-27T03:19:35.384884+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:19:35.384884+0000","last_undegraded":"2024-04-27T03:19:35.384884+0000","last_fullsized":"2024-04-27T03:19:35.384884+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:45.567442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815568+0000","last_change":"2024-04-27T03:17:51.020247+0000","last_active":"2024-04-27T03:19:34.815568+0000","last_peered":"2024-04-27T03:19:34.815568+0000","last_clean":"2024-04-27T03:19:34.815568+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:19:34.815568+0000","last_undegraded":"2024-04-27T03:19:34.815568+0000","last_fullsized":"2024-04-27T03:19:34.815568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812598+0000","last_change":"2024-04-27T03:17:53.048021+0000","last_active":"2024-04-27T03:19:34.812598+0000","last_peered":"2024-04-27T03:19:34.812598+0000","last_clean":"2024-04-27T03:19:34.812598+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:19:34.812598+0000","last_undegraded":"2024-04-27T03:19:34.812598+0000","last_fullsized":"2024-04-27T03:19:34.812598+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:13.963398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'282","reported_seq":417,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.458427+0000","last_change":"2024-04-27T03:17:55.075368+0000","last_active":"2024-04-27T03:21:01.458427+0000","last_peered":"2024-04-27T03:21:01.458427+0000","last_clean":"2024-04-27T03:21:01.458427+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:01.458427+0000","last_undegraded":"2024-04-27T03:21:01.458427+0000","last_fullsized":"2024-04-27T03:21:01.458427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:03:11.018401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381777+0000","last_change":"2024-04-27T03:17:57.087004+0000","last_active":"2024-04-27T03:19:35.381777+0000","last_peered":"2024-04-27T03:19:35.381777+0000","last_clean":"2024-04-27T03:19:35.381777+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:19:35.381777+0000","last_undegraded":"2024-04-27T03:19:35.381777+0000","last_fullsized":"2024-04-27T03:19:35.381777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:38:17.983425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.193216+0000","last_change":"2024-04-27T03:17:49.004339+0000","last_active":"2024-04-27T03:20:00.193216+0000","last_peered":"2024-04-27T03:20:00.193216+0000","last_clean":"2024-04-27T03:20:00.193216+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:20:00.193216+0000","last_undegraded":"2024-04-27T03:20:00.193216+0000","last_fullsized":"2024-04-27T03:20:00.193216+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:08.912344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.153066+0000","last_change":"2024-04-27T03:17:47.000780+0000","last_active":"2024-04-27T03:20:00.153066+0000","last_peered":"2024-04-27T03:20:00.153066+0000","last_clean":"2024-04-27T03:20:00.153066+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:20:00.153066+0000","last_undegraded":"2024-04-27T03:20:00.153066+0000","last_fullsized":"2024-04-27T03:20:00.153066+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:57.459077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819785+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:19:34.819785+0000","last_peered":"2024-04-27T03:19:34.819785+0000","last_clean":"2024-04-27T03:19:34.819785+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:19:34.819785+0000","last_undegraded":"2024-04-27T03:19:34.819785+0000","last_fullsized":"2024-04-27T03:19:34.819785+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:34.021988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810326+0000","last_change":"2024-04-27T03:17:51.033805+0000","last_active":"2024-04-27T03:19:34.810326+0000","last_peered":"2024-04-27T03:19:34.810326+0000","last_clean":"2024-04-27T03:19:34.810326+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:19:34.810326+0000","last_undegraded":"2024-04-27T03:19:34.810326+0000","last_fullsized":"2024-04-27T03:19:34.810326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:46:41.312442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'92","reported_seq":174,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.728448+0000","last_change":"2024-04-27T03:17:53.038345+0000","last_active":"2024-04-27T03:21:00.728448+0000","last_peered":"2024-04-27T03:21:00.728448+0000","last_clean":"2024-04-27T03:21:00.728448+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:00.728448+0000","last_undegraded":"2024-04-27T03:21:00.728448+0000","last_fullsized":"2024-04-27T03:21:00.728448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:47.882669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6026,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":86,"num_read_kb":71,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814813+0000","last_change":"2024-04-27T03:17:55.081924+0000","last_active":"2024-04-27T03:19:34.814813+0000","last_peered":"2024-04-27T03:19:34.814813+0000","last_clean":"2024-04-27T03:19:34.814813+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:19:34.814813+0000","last_undegraded":"2024-04-27T03:19:34.814813+0000","last_fullsized":"2024-04-27T03:19:34.814813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439932+0000","last_change":"2024-04-27T03:17:57.068568+0000","last_active":"2024-04-27T03:19:35.439932+0000","last_peered":"2024-04-27T03:19:35.439932+0000","last_clean":"2024-04-27T03:19:35.439932+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:19:35.439932+0000","last_undegraded":"2024-04-27T03:19:35.439932+0000","last_fullsized":"2024-04-27T03:19:35.439932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:43.785644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442918+0000","last_change":"2024-04-27T03:17:57.068516+0000","last_active":"2024-04-27T03:19:35.442918+0000","last_peered":"2024-04-27T03:19:35.442918+0000","last_clean":"2024-04-27T03:19:35.442918+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:19:35.442918+0000","last_undegraded":"2024-04-27T03:19:35.442918+0000","last_fullsized":"2024-04-27T03:19:35.442918+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:58.924125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.802035+0000","last_change":"2024-04-27T03:17:49.009366+0000","last_active":"2024-04-27T03:21:00.802035+0000","last_peered":"2024-04-27T03:21:00.802035+0000","last_clean":"2024-04-27T03:21:00.802035+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:00.802035+0000","last_undegraded":"2024-04-27T03:21:00.802035+0000","last_fullsized":"2024-04-27T03:21:00.802035+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:58.301334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.124874+0000","last_change":"2024-04-27T03:17:47.023352+0000","last_active":"2024-04-27T03:21:01.124874+0000","last_peered":"2024-04-27T03:21:01.124874+0000","last_clean":"2024-04-27T03:21:01.124874+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:01.124874+0000","last_undegraded":"2024-04-27T03:21:01.124874+0000","last_fullsized":"2024-04-27T03:21:01.124874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.416461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"20'1","reported_seq":222,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.724858+0000","last_change":"2024-04-27T03:17:51.031849+0000","last_active":"2024-04-27T03:19:59.724858+0000","last_peered":"2024-04-27T03:19:59.724858+0000","last_clean":"2024-04-27T03:19:59.724858+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:19:59.724858+0000","last_undegraded":"2024-04-27T03:19:59.724858+0000","last_fullsized":"2024-04-27T03:19:59.724858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:05.179093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":282,"num_read_kb":282,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'38","reported_seq":91,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.283287+0000","last_change":"2024-04-27T03:17:53.041357+0000","last_active":"2024-04-27T03:19:58.283287+0000","last_peered":"2024-04-27T03:19:58.283287+0000","last_clean":"2024-04-27T03:19:58.283287+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:19:58.283287+0000","last_undegraded":"2024-04-27T03:19:58.283287+0000","last_fullsized":"2024-04-27T03:19:58.283287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:05:57.603435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809115+0000","last_change":"2024-04-27T03:17:55.074798+0000","last_active":"2024-04-27T03:19:34.809115+0000","last_peered":"2024-04-27T03:19:34.809115+0000","last_clean":"2024-04-27T03:19:34.809115+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:19:34.809115+0000","last_undegraded":"2024-04-27T03:19:34.809115+0000","last_fullsized":"2024-04-27T03:19:34.809115+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:15.275458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.885734+0000","last_change":"2024-04-27T03:17:57.087180+0000","last_active":"2024-04-27T03:19:57.885734+0000","last_peered":"2024-04-27T03:19:57.885734+0000","last_clean":"2024-04-27T03:19:57.885734+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:19:57.885734+0000","last_undegraded":"2024-04-27T03:19:57.885734+0000","last_fullsized":"2024-04-27T03:19:57.885734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:50.266677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146716+0000","last_change":"2024-04-27T03:17:49.005188+0000","last_active":"2024-04-27T03:21:01.146716+0000","last_peered":"2024-04-27T03:21:01.146716+0000","last_clean":"2024-04-27T03:21:01.146716+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:01.146716+0000","last_undegraded":"2024-04-27T03:21:01.146716+0000","last_fullsized":"2024-04-27T03:21:01.146716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:24.812863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.440214+0000","last_change":"2024-04-27T03:17:47.022474+0000","last_active":"2024-04-27T03:19:35.440214+0000","last_peered":"2024-04-27T03:19:35.440214+0000","last_clean":"2024-04-27T03:19:35.440214+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:19:35.440214+0000","last_undegraded":"2024-04-27T03:19:35.440214+0000","last_fullsized":"2024-04-27T03:19:35.440214+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:13.293355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811181+0000","last_change":"2024-04-27T03:17:51.024948+0000","last_active":"2024-04-27T03:19:34.811181+0000","last_peered":"2024-04-27T03:19:34.811181+0000","last_clean":"2024-04-27T03:19:34.811181+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:19:34.811181+0000","last_undegraded":"2024-04-27T03:19:34.811181+0000","last_fullsized":"2024-04-27T03:19:34.811181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'6","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.271365+0000","last_change":"2024-04-27T03:17:53.053658+0000","last_active":"2024-04-27T03:21:00.271365+0000","last_peered":"2024-04-27T03:21:00.271365+0000","last_clean":"2024-04-27T03:21:00.271365+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:00.271365+0000","last_undegraded":"2024-04-27T03:21:00.271365+0000","last_fullsized":"2024-04-27T03:21:00.271365+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:12.928546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438320+0000","last_change":"2024-04-27T03:17:55.051277+0000","last_active":"2024-04-27T03:19:35.438320+0000","last_peered":"2024-04-27T03:19:35.438320+0000","last_clean":"2024-04-27T03:19:35.438320+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:19:35.438320+0000","last_undegraded":"2024-04-27T03:19:35.438320+0000","last_fullsized":"2024-04-27T03:19:35.438320+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:49.716708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382470+0000","last_change":"2024-04-27T03:17:57.089220+0000","last_active":"2024-04-27T03:19:35.382470+0000","last_peered":"2024-04-27T03:19:35.382470+0000","last_clean":"2024-04-27T03:19:35.382470+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:19:35.382470+0000","last_undegraded":"2024-04-27T03:19:35.382470+0000","last_fullsized":"2024-04-27T03:19:35.382470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:59:17.289812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949086+0000","last_change":"2024-04-27T03:18:34.799245+0000","last_active":"2024-04-27T03:21:00.949086+0000","last_peered":"2024-04-27T03:21:00.949086+0000","last_clean":"2024-04-27T03:21:00.949086+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:21:00.949086+0000","last_undegraded":"2024-04-27T03:21:00.949086+0000","last_fullsized":"2024-04-27T03:21:00.949086+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:44:17.961788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.107590+0000","last_change":"2024-04-27T03:17:47.012224+0000","last_active":"2024-04-27T03:21:01.107590+0000","last_peered":"2024-04-27T03:21:01.107590+0000","last_clean":"2024-04-27T03:21:01.107590+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:21:01.107590+0000","last_undegraded":"2024-04-27T03:21:01.107590+0000","last_fullsized":"2024-04-27T03:21:01.107590+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813595+0000","last_change":"2024-04-27T03:17:51.037763+0000","last_active":"2024-04-27T03:19:34.813595+0000","last_peered":"2024-04-27T03:19:34.813595+0000","last_clean":"2024-04-27T03:19:34.813595+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:19:34.813595+0000","last_undegraded":"2024-04-27T03:19:34.813595+0000","last_fullsized":"2024-04-27T03:19:34.813595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:19.793632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'40","reported_seq":103,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.321256+0000","last_change":"2024-04-27T03:17:53.047021+0000","last_active":"2024-04-27T03:21:00.321256+0000","last_peered":"2024-04-27T03:21:00.321256+0000","last_clean":"2024-04-27T03:21:00.321256+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:00.321256+0000","last_undegraded":"2024-04-27T03:21:00.321256+0000","last_fullsized":"2024-04-27T03:21:00.321256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":61,"num_read_kb":64,"num_write":46,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379567+0000","last_change":"2024-04-27T03:17:55.075762+0000","last_active":"2024-04-27T03:19:35.379567+0000","last_peered":"2024-04-27T03:19:35.379567+0000","last_clean":"2024-04-27T03:19:35.379567+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:19:35.379567+0000","last_undegraded":"2024-04-27T03:19:35.379567+0000","last_fullsized":"2024-04-27T03:19:35.379567+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:49:01.187937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812651+0000","last_change":"2024-04-27T03:17:57.086967+0000","last_active":"2024-04-27T03:19:34.812651+0000","last_peered":"2024-04-27T03:19:34.812651+0000","last_clean":"2024-04-27T03:19:34.812651+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:19:34.812651+0000","last_undegraded":"2024-04-27T03:19:34.812651+0000","last_fullsized":"2024-04-27T03:19:34.812651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:54.374770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.234017+0000","last_change":"2024-04-27T03:17:49.014835+0000","last_active":"2024-04-27T03:21:01.234017+0000","last_peered":"2024-04-27T03:21:01.234017+0000","last_clean":"2024-04-27T03:21:01.234017+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:01.234017+0000","last_undegraded":"2024-04-27T03:21:01.234017+0000","last_fullsized":"2024-04-27T03:21:01.234017+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.673527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385250+0000","last_change":"2024-04-27T03:17:47.026622+0000","last_active":"2024-04-27T03:19:35.385250+0000","last_peered":"2024-04-27T03:19:35.385250+0000","last_clean":"2024-04-27T03:19:35.385250+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:19:35.385250+0000","last_undegraded":"2024-04-27T03:19:35.385250+0000","last_fullsized":"2024-04-27T03:19:35.385250+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:05:57.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809701+0000","last_change":"2024-04-27T03:17:51.014069+0000","last_active":"2024-04-27T03:19:34.809701+0000","last_peered":"2024-04-27T03:19:34.809701+0000","last_clean":"2024-04-27T03:19:34.809701+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:19:34.809701+0000","last_undegraded":"2024-04-27T03:19:34.809701+0000","last_fullsized":"2024-04-27T03:19:34.809701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:21:25.135902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'31","reported_seq":77,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745864+0000","last_change":"2024-04-27T03:17:53.038792+0000","last_active":"2024-04-27T03:20:12.745864+0000","last_peered":"2024-04-27T03:20:12.745864+0000","last_clean":"2024-04-27T03:20:12.745864+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:20:12.745864+0000","last_undegraded":"2024-04-27T03:20:12.745864+0000","last_fullsized":"2024-04-27T03:20:12.745864+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:14:29.150154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438653+0000","last_change":"2024-04-27T03:17:55.055483+0000","last_active":"2024-04-27T03:19:35.438653+0000","last_peered":"2024-04-27T03:19:35.438653+0000","last_clean":"2024-04-27T03:19:35.438653+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:19:35.438653+0000","last_undegraded":"2024-04-27T03:19:35.438653+0000","last_fullsized":"2024-04-27T03:19:35.438653+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:17.695216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'1","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742486+0000","last_change":"2024-04-27T03:17:49.022141+0000","last_active":"2024-04-27T03:19:57.742486+0000","last_peered":"2024-04-27T03:19:57.742486+0000","last_clean":"2024-04-27T03:19:57.742486+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:19:57.742486+0000","last_undegraded":"2024-04-27T03:19:57.742486+0000","last_fullsized":"2024-04-27T03:19:57.742486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:15.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.117502+0000","last_change":"2024-04-27T03:17:47.032082+0000","last_active":"2024-04-27T03:21:01.117502+0000","last_peered":"2024-04-27T03:21:01.117502+0000","last_clean":"2024-04-27T03:21:01.117502+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:21:01.117502+0000","last_undegraded":"2024-04-27T03:21:01.117502+0000","last_fullsized":"2024-04-27T03:21:01.117502+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'4","reported_seq":230,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.696608+0000","last_change":"2024-04-27T03:17:51.024084+0000","last_active":"2024-04-27T03:19:59.696608+0000","last_peered":"2024-04-27T03:19:59.696608+0000","last_clean":"2024-04-27T03:19:59.696608+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:19:59.696608+0000","last_undegraded":"2024-04-27T03:19:59.696608+0000","last_fullsized":"2024-04-27T03:19:59.696608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:37.745198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":298,"num_read_kb":298,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'62","reported_seq":128,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.293689+0000","last_change":"2024-04-27T03:17:53.035870+0000","last_active":"2024-04-27T03:19:58.293689+0000","last_peered":"2024-04-27T03:19:58.293689+0000","last_clean":"2024-04-27T03:19:58.293689+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:19:58.293689+0000","last_undegraded":"2024-04-27T03:19:58.293689+0000","last_fullsized":"2024-04-27T03:19:58.293689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:10.295737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":48,"num_write":24,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.439324+0000","last_change":"2024-04-27T03:17:55.075718+0000","last_active":"2024-04-27T03:19:35.439324+0000","last_peered":"2024-04-27T03:19:35.439324+0000","last_clean":"2024-04-27T03:19:35.439324+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:19:35.439324+0000","last_undegraded":"2024-04-27T03:19:35.439324+0000","last_fullsized":"2024-04-27T03:19:35.439324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:58.929938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.074643+0000","last_change":"2024-04-27T03:17:57.070510+0000","last_active":"2024-04-27T03:20:00.074643+0000","last_peered":"2024-04-27T03:20:00.074643+0000","last_clean":"2024-04-27T03:20:00.074643+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:20:00.074643+0000","last_undegraded":"2024-04-27T03:20:00.074643+0000","last_fullsized":"2024-04-27T03:20:00.074643+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111625+0000","last_change":"2024-04-27T03:17:49.000295+0000","last_active":"2024-04-27T03:19:56.111625+0000","last_peered":"2024-04-27T03:19:56.111625+0000","last_clean":"2024-04-27T03:19:56.111625+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:19:56.111625+0000","last_undegraded":"2024-04-27T03:19:56.111625+0000","last_fullsized":"2024-04-27T03:19:56.111625+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379919+0000","last_change":"2024-04-27T03:17:47.021864+0000","last_active":"2024-04-27T03:19:35.379919+0000","last_peered":"2024-04-27T03:19:35.379919+0000","last_clean":"2024-04-27T03:19:35.379919+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:19:35.379919+0000","last_undegraded":"2024-04-27T03:19:35.379919+0000","last_fullsized":"2024-04-27T03:19:35.379919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:14.918204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.820383+0000","last_change":"2024-04-27T03:17:51.033166+0000","last_active":"2024-04-27T03:19:34.820383+0000","last_peered":"2024-04-27T03:19:34.820383+0000","last_clean":"2024-04-27T03:19:34.820383+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:19:34.820383+0000","last_undegraded":"2024-04-27T03:19:34.820383+0000","last_fullsized":"2024-04-27T03:19:34.820383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:09:40.351925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'36","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.260042+0000","last_change":"2024-04-27T03:17:53.038085+0000","last_active":"2024-04-27T03:19:58.260042+0000","last_peered":"2024-04-27T03:19:58.260042+0000","last_clean":"2024-04-27T03:19:58.260042+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:19:58.260042+0000","last_undegraded":"2024-04-27T03:19:58.260042+0000","last_fullsized":"2024-04-27T03:19:58.260042+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.383103+0000","last_change":"2024-04-27T03:17:55.069433+0000","last_active":"2024-04-27T03:19:35.383103+0000","last_peered":"2024-04-27T03:19:35.383103+0000","last_clean":"2024-04-27T03:19:35.383103+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:19:35.383103+0000","last_undegraded":"2024-04-27T03:19:35.383103+0000","last_fullsized":"2024-04-27T03:19:35.383103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:17.345394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814942+0000","last_change":"2024-04-27T03:17:57.070067+0000","last_active":"2024-04-27T03:19:34.814942+0000","last_peered":"2024-04-27T03:19:34.814942+0000","last_clean":"2024-04-27T03:19:34.814942+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:19:34.814942+0000","last_undegraded":"2024-04-27T03:19:34.814942+0000","last_fullsized":"2024-04-27T03:19:34.814942+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:55:49.238709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814892+0000","last_change":"2024-04-27T03:17:49.004464+0000","last_active":"2024-04-27T03:19:34.814892+0000","last_peered":"2024-04-27T03:19:34.814892+0000","last_clean":"2024-04-27T03:19:34.814892+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:19:34.814892+0000","last_undegraded":"2024-04-27T03:19:34.814892+0000","last_fullsized":"2024-04-27T03:19:34.814892+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.095190+0000","last_change":"2024-04-27T03:17:47.027156+0000","last_active":"2024-04-27T03:19:56.095190+0000","last_peered":"2024-04-27T03:19:56.095190+0000","last_clean":"2024-04-27T03:19:56.095190+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:19:56.095190+0000","last_undegraded":"2024-04-27T03:19:56.095190+0000","last_fullsized":"2024-04-27T03:19:56.095190+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:13:57.761315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816982+0000","last_change":"2024-04-27T03:17:51.021222+0000","last_active":"2024-04-27T03:19:34.816982+0000","last_peered":"2024-04-27T03:19:34.816982+0000","last_clean":"2024-04-27T03:19:34.816982+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:19:34.816982+0000","last_undegraded":"2024-04-27T03:19:34.816982+0000","last_fullsized":"2024-04-27T03:19:34.816982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:40:40.067249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'39","reported_seq":92,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.872099+0000","last_change":"2024-04-27T03:17:53.041372+0000","last_active":"2024-04-27T03:19:57.872099+0000","last_peered":"2024-04-27T03:19:57.872099+0000","last_clean":"2024-04-27T03:19:57.872099+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:19:57.872099+0000","last_undegraded":"2024-04-27T03:19:57.872099+0000","last_fullsized":"2024-04-27T03:19:57.872099+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:43:07.599380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.384616+0000","last_change":"2024-04-27T03:17:55.077381+0000","last_active":"2024-04-27T03:19:35.384616+0000","last_peered":"2024-04-27T03:19:35.384616+0000","last_clean":"2024-04-27T03:19:35.384616+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:19:35.384616+0000","last_undegraded":"2024-04-27T03:19:35.384616+0000","last_fullsized":"2024-04-27T03:19:35.384616+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:08:00.332438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.205156+0000","last_change":"2024-04-27T03:17:57.066218+0000","last_active":"2024-04-27T03:20:00.205156+0000","last_peered":"2024-04-27T03:20:00.205156+0000","last_clean":"2024-04-27T03:20:00.205156+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:20:00.205156+0000","last_undegraded":"2024-04-27T03:20:00.205156+0000","last_fullsized":"2024-04-27T03:20:00.205156+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:03.602846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.739365+0000","last_change":"2024-04-27T03:17:49.019967+0000","last_active":"2024-04-27T03:20:22.739365+0000","last_peered":"2024-04-27T03:20:22.739365+0000","last_clean":"2024-04-27T03:20:22.739365+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:20:22.739365+0000","last_undegraded":"2024-04-27T03:20:22.739365+0000","last_fullsized":"2024-04-27T03:20:22.739365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:45.458581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812087+0000","last_change":"2024-04-27T03:17:47.018817+0000","last_active":"2024-04-27T03:19:34.812087+0000","last_peered":"2024-04-27T03:19:34.812087+0000","last_clean":"2024-04-27T03:19:34.812087+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:19:34.812087+0000","last_undegraded":"2024-04-27T03:19:34.812087+0000","last_fullsized":"2024-04-27T03:19:34.812087+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:16:06.895861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377658+0000","last_change":"2024-04-27T03:17:51.030031+0000","last_active":"2024-04-27T03:19:35.377658+0000","last_peered":"2024-04-27T03:19:35.377658+0000","last_clean":"2024-04-27T03:19:35.377658+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:19:35.377658+0000","last_undegraded":"2024-04-27T03:19:35.377658+0000","last_fullsized":"2024-04-27T03:19:35.377658+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:11:08.238587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"27'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814172+0000","last_change":"2024-04-27T03:17:53.046967+0000","last_active":"2024-04-27T03:19:34.814172+0000","last_peered":"2024-04-27T03:19:34.814172+0000","last_clean":"2024-04-27T03:19:34.814172+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:19:34.814172+0000","last_undegraded":"2024-04-27T03:19:34.814172+0000","last_fullsized":"2024-04-27T03:19:34.814172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:03:27.264950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'282","reported_seq":380,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.277975+0000","last_change":"2024-04-27T03:17:55.060259+0000","last_active":"2024-04-27T03:21:01.277975+0000","last_peered":"2024-04-27T03:21:01.277975+0000","last_clean":"2024-04-27T03:21:01.277975+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:01.277975+0000","last_undegraded":"2024-04-27T03:21:01.277975+0000","last_fullsized":"2024-04-27T03:21:01.277975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:04:03.670215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.813560+0000","last_change":"2024-04-27T03:17:57.084816+0000","last_active":"2024-04-27T03:19:34.813560+0000","last_peered":"2024-04-27T03:19:34.813560+0000","last_clean":"2024-04-27T03:19:34.813560+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:19:34.813560+0000","last_undegraded":"2024-04-27T03:19:34.813560+0000","last_fullsized":"2024-04-27T03:19:34.813560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:18:03.843907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248654+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.248654+0000","last_peered":"2024-04-27T03:21:01.248654+0000","last_clean":"2024-04-27T03:21:01.248654+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:01.248654+0000","last_undegraded":"2024-04-27T03:21:01.248654+0000","last_fullsized":"2024-04-27T03:21:01.248654+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:22.252077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382608+0000","last_change":"2024-04-27T03:17:47.027621+0000","last_active":"2024-04-27T03:19:35.382608+0000","last_peered":"2024-04-27T03:19:35.382608+0000","last_clean":"2024-04-27T03:19:35.382608+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:19:35.382608+0000","last_undegraded":"2024-04-27T03:19:35.382608+0000","last_fullsized":"2024-04-27T03:19:35.382608+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377762+0000","last_change":"2024-04-27T03:17:51.032091+0000","last_active":"2024-04-27T03:19:35.377762+0000","last_peered":"2024-04-27T03:19:35.377762+0000","last_clean":"2024-04-27T03:19:35.377762+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:19:35.377762+0000","last_undegraded":"2024-04-27T03:19:35.377762+0000","last_fullsized":"2024-04-27T03:19:35.377762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:06.149814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'60","reported_seq":133,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.243349+0000","last_change":"2024-04-27T03:17:53.039949+0000","last_active":"2024-04-27T03:19:58.243349+0000","last_peered":"2024-04-27T03:19:58.243349+0000","last_clean":"2024-04-27T03:19:58.243349+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:19:58.243349+0000","last_undegraded":"2024-04-27T03:19:58.243349+0000","last_fullsized":"2024-04-27T03:19:58.243349+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:07:58.319861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":74,"num_read_kb":61,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.808839+0000","last_change":"2024-04-27T03:17:55.071558+0000","last_active":"2024-04-27T03:19:34.808839+0000","last_peered":"2024-04-27T03:19:34.808839+0000","last_clean":"2024-04-27T03:19:34.808839+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:19:34.808839+0000","last_undegraded":"2024-04-27T03:19:34.808839+0000","last_fullsized":"2024-04-27T03:19:34.808839+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:12.887800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.348234+0000","last_change":"2024-04-27T03:17:57.066613+0000","last_active":"2024-04-27T03:21:00.348234+0000","last_peered":"2024-04-27T03:21:00.348234+0000","last_clean":"2024-04-27T03:21:00.348234+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:21:00.348234+0000","last_undegraded":"2024-04-27T03:21:00.348234+0000","last_fullsized":"2024-04-27T03:21:00.348234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:02:34.623206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949296+0000","last_change":"2024-04-27T03:17:49.017718+0000","last_active":"2024-04-27T03:21:00.949296+0000","last_peered":"2024-04-27T03:21:00.949296+0000","last_clean":"2024-04-27T03:21:00.949296+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:00.949296+0000","last_undegraded":"2024-04-27T03:21:00.949296+0000","last_fullsized":"2024-04-27T03:21:00.949296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:40.378270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.819810+0000","last_change":"2024-04-27T03:17:47.022941+0000","last_active":"2024-04-27T03:19:34.819810+0000","last_peered":"2024-04-27T03:19:34.819810+0000","last_clean":"2024-04-27T03:19:34.819810+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:19:34.819810+0000","last_undegraded":"2024-04-27T03:19:34.819810+0000","last_fullsized":"2024-04-27T03:19:34.819810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:47:09.787441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816507+0000","last_change":"2024-04-27T03:17:51.033578+0000","last_active":"2024-04-27T03:19:34.816507+0000","last_peered":"2024-04-27T03:19:34.816507+0000","last_clean":"2024-04-27T03:19:34.816507+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:19:34.816507+0000","last_undegraded":"2024-04-27T03:19:34.816507+0000","last_fullsized":"2024-04-27T03:19:34.816507+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:05.625484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'38","reported_seq":90,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.244009+0000","last_change":"2024-04-27T03:17:53.040115+0000","last_active":"2024-04-27T03:19:58.244009+0000","last_peered":"2024-04-27T03:19:58.244009+0000","last_clean":"2024-04-27T03:19:58.244009+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:19:58.244009+0000","last_undegraded":"2024-04-27T03:19:58.244009+0000","last_fullsized":"2024-04-27T03:19:58.244009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:26:28.843466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":31,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815617+0000","last_change":"2024-04-27T03:17:55.058615+0000","last_active":"2024-04-27T03:19:34.815617+0000","last_peered":"2024-04-27T03:19:34.815617+0000","last_clean":"2024-04-27T03:19:34.815617+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:19:34.815617+0000","last_undegraded":"2024-04-27T03:19:34.815617+0000","last_fullsized":"2024-04-27T03:19:34.815617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:49.588801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'2","reported_seq":23,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.401928+0000","last_change":"2024-04-27T03:17:57.087706+0000","last_active":"2024-04-27T03:21:01.401928+0000","last_peered":"2024-04-27T03:21:01.401928+0000","last_clean":"2024-04-27T03:21:01.401928+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:21:01.401928+0000","last_undegraded":"2024-04-27T03:21:01.401928+0000","last_fullsized":"2024-04-27T03:21:01.401928+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:57.341894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.950567+0000","last_change":"2024-04-27T03:17:49.011902+0000","last_active":"2024-04-27T03:21:00.950567+0000","last_peered":"2024-04-27T03:21:00.950567+0000","last_clean":"2024-04-27T03:21:00.950567+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:21:00.950567+0000","last_undegraded":"2024-04-27T03:21:00.950567+0000","last_fullsized":"2024-04-27T03:21:00.950567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'1","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.735456+0000","last_change":"2024-04-27T03:17:47.016344+0000","last_active":"2024-04-27T03:19:47.735456+0000","last_peered":"2024-04-27T03:19:47.735456+0000","last_clean":"2024-04-27T03:19:47.735456+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:19:47.735456+0000","last_undegraded":"2024-04-27T03:19:47.735456+0000","last_fullsized":"2024-04-27T03:19:47.735456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:36:51.138307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.377191+0000","last_change":"2024-04-27T03:17:51.029804+0000","last_active":"2024-04-27T03:19:35.377191+0000","last_peered":"2024-04-27T03:19:35.377191+0000","last_clean":"2024-04-27T03:19:35.377191+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:19:35.377191+0000","last_undegraded":"2024-04-27T03:19:35.377191+0000","last_fullsized":"2024-04-27T03:19:35.377191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:13.111551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'32","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.286319+0000","last_change":"2024-04-27T03:17:53.041146+0000","last_active":"2024-04-27T03:19:58.286319+0000","last_peered":"2024-04-27T03:19:58.286319+0000","last_clean":"2024-04-27T03:19:58.286319+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:19:58.286319+0000","last_undegraded":"2024-04-27T03:19:58.286319+0000","last_fullsized":"2024-04-27T03:19:58.286319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:19:15.332817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":27,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.816372+0000","last_change":"2024-04-27T03:17:55.071034+0000","last_active":"2024-04-27T03:19:34.816372+0000","last_peered":"2024-04-27T03:19:34.816372+0000","last_clean":"2024-04-27T03:19:34.816372+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:19:34.816372+0000","last_undegraded":"2024-04-27T03:19:34.816372+0000","last_fullsized":"2024-04-27T03:19:34.816372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:38.476851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.302058+0000","last_change":"2024-04-27T03:17:57.086785+0000","last_active":"2024-04-27T03:19:58.302058+0000","last_peered":"2024-04-27T03:19:58.302058+0000","last_clean":"2024-04-27T03:19:58.302058+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:19:58.302058+0000","last_undegraded":"2024-04-27T03:19:58.302058+0000","last_fullsized":"2024-04-27T03:19:58.302058+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:25:08.779674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'14","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397096+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.397096+0000","last_peered":"2024-04-27T03:21:01.397096+0000","last_clean":"2024-04-27T03:21:01.397096+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:01.397096+0000","last_undegraded":"2024-04-27T03:21:01.397096+0000","last_fullsized":"2024-04-27T03:21:01.397096+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:53.264796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.019895+0000","last_change":"2024-04-27T03:17:47.002631+0000","last_active":"2024-04-27T03:20:00.019895+0000","last_peered":"2024-04-27T03:20:00.019895+0000","last_clean":"2024-04-27T03:20:00.019895+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:20:00.019895+0000","last_undegraded":"2024-04-27T03:20:00.019895+0000","last_fullsized":"2024-04-27T03:20:00.019895+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:56.619457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'2","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.673843+0000","last_change":"2024-04-27T03:18:34.794992+0000","last_active":"2024-04-27T03:19:56.673843+0000","last_peered":"2024-04-27T03:19:56.673843+0000","last_clean":"2024-04-27T03:19:56.673843+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:19:56.673843+0000","last_undegraded":"2024-04-27T03:19:56.673843+0000","last_fullsized":"2024-04-27T03:19:56.673843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:56:59.713686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'44","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.246215+0000","last_change":"2024-04-27T03:17:53.039095+0000","last_active":"2024-04-27T03:21:00.246215+0000","last_peered":"2024-04-27T03:21:00.246215+0000","last_clean":"2024-04-27T03:21:00.246215+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:00.246215+0000","last_undegraded":"2024-04-27T03:21:00.246215+0000","last_fullsized":"2024-04-27T03:21:00.246215+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:49.353150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":30,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438638+0000","last_change":"2024-04-27T03:17:55.063864+0000","last_active":"2024-04-27T03:19:35.438638+0000","last_peered":"2024-04-27T03:19:35.438638+0000","last_clean":"2024-04-27T03:19:35.438638+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:19:35.438638+0000","last_undegraded":"2024-04-27T03:19:35.438638+0000","last_fullsized":"2024-04-27T03:19:35.438638+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:46.421450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:58.307196+0000","last_change":"2024-04-27T03:17:57.061982+0000","last_active":"2024-04-27T03:19:58.307196+0000","last_peered":"2024-04-27T03:19:58.307196+0000","last_clean":"2024-04-27T03:19:58.307196+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:19:58.307196+0000","last_undegraded":"2024-04-27T03:19:58.307196+0000","last_fullsized":"2024-04-27T03:19:58.307196+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:37.720697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.386099+0000","last_change":"2024-04-27T03:17:49.001808+0000","last_active":"2024-04-27T03:19:35.386099+0000","last_peered":"2024-04-27T03:19:35.386099+0000","last_clean":"2024-04-27T03:19:35.386099+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:19:35.386099+0000","last_undegraded":"2024-04-27T03:19:35.386099+0000","last_fullsized":"2024-04-27T03:19:35.386099+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:19:33.807122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.436337+0000","last_change":"2024-04-27T03:17:47.032632+0000","last_active":"2024-04-27T03:19:35.436337+0000","last_peered":"2024-04-27T03:19:35.436337+0000","last_clean":"2024-04-27T03:19:35.436337+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:19:35.436337+0000","last_undegraded":"2024-04-27T03:19:35.436337+0000","last_fullsized":"2024-04-27T03:19:35.436337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:53:43.154787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385434+0000","last_change":"2024-04-27T03:17:49.011651+0000","last_active":"2024-04-27T03:19:35.385434+0000","last_peered":"2024-04-27T03:19:35.385434+0000","last_clean":"2024-04-27T03:19:35.385434+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:19:35.385434+0000","last_undegraded":"2024-04-27T03:19:35.385434+0000","last_fullsized":"2024-04-27T03:19:35.385434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:44.213409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'1","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.732801+0000","last_change":"2024-04-27T03:18:34.773740+0000","last_active":"2024-04-27T03:19:52.732801+0000","last_peered":"2024-04-27T03:19:52.732801+0000","last_clean":"2024-04-27T03:19:52.732801+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:19:52.732801+0000","last_undegraded":"2024-04-27T03:19:52.732801+0000","last_fullsized":"2024-04-27T03:19:52.732801+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:04.248651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639872,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111155+0000","last_change":"2024-04-27T03:18:34.784905+0000","last_active":"2024-04-27T03:19:56.111155+0000","last_peered":"2024-04-27T03:19:56.111155+0000","last_clean":"2024-04-27T03:19:56.111155+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:19:56.111155+0000","last_undegraded":"2024-04-27T03:19:56.111155+0000","last_fullsized":"2024-04-27T03:19:56.111155+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:28.273007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810481+0000","last_change":"2024-04-27T03:17:47.036148+0000","last_active":"2024-04-27T03:19:34.810481+0000","last_peered":"2024-04-27T03:19:34.810481+0000","last_clean":"2024-04-27T03:19:34.810481+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:19:34.810481+0000","last_undegraded":"2024-04-27T03:19:34.810481+0000","last_fullsized":"2024-04-27T03:19:34.810481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:00:20.235594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382422+0000","last_change":"2024-04-27T03:17:49.017582+0000","last_active":"2024-04-27T03:19:35.382422+0000","last_peered":"2024-04-27T03:19:35.382422+0000","last_clean":"2024-04-27T03:19:35.382422+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:19:35.382422+0000","last_undegraded":"2024-04-27T03:19:35.382422+0000","last_fullsized":"2024-04-27T03:19:35.382422+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:20.541354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.438828+0000","last_change":"2024-04-27T03:17:47.016123+0000","last_active":"2024-04-27T03:19:35.438828+0000","last_peered":"2024-04-27T03:19:35.438828+0000","last_clean":"2024-04-27T03:19:35.438828+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:19:35.438828+0000","last_undegraded":"2024-04-27T03:19:35.438828+0000","last_fullsized":"2024-04-27T03:19:35.438828+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:59.109823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300171+0000","last_change":"2024-04-27T03:17:49.021184+0000","last_active":"2024-04-27T03:21:01.300171+0000","last_peered":"2024-04-27T03:21:01.300171+0000","last_clean":"2024-04-27T03:21:01.300171+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:01.300171+0000","last_undegraded":"2024-04-27T03:21:01.300171+0000","last_fullsized":"2024-04-27T03:21:01.300171+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:37.664031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.811874+0000","last_change":"2024-04-27T03:17:47.039423+0000","last_active":"2024-04-27T03:19:34.811874+0000","last_peered":"2024-04-27T03:19:34.811874+0000","last_clean":"2024-04-27T03:19:34.811874+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:19:34.811874+0000","last_undegraded":"2024-04-27T03:19:34.811874+0000","last_fullsized":"2024-04-27T03:19:34.811874+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:21.697442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.381685+0000","last_change":"2024-04-27T03:17:49.006801+0000","last_active":"2024-04-27T03:19:35.381685+0000","last_peered":"2024-04-27T03:19:35.381685+0000","last_clean":"2024-04-27T03:19:35.381685+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:19:35.381685+0000","last_undegraded":"2024-04-27T03:19:35.381685+0000","last_fullsized":"2024-04-27T03:19:35.381685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:39:13.312552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.947064+0000","last_change":"2024-04-27T03:17:47.030288+0000","last_active":"2024-04-27T03:19:59.947064+0000","last_peered":"2024-04-27T03:19:59.947064+0000","last_clean":"2024-04-27T03:19:59.947064+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:19:59.947064+0000","last_undegraded":"2024-04-27T03:19:59.947064+0000","last_fullsized":"2024-04-27T03:19:59.947064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:52:43.758948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300194+0000","last_change":"2024-04-27T03:17:49.015796+0000","last_active":"2024-04-27T03:21:01.300194+0000","last_peered":"2024-04-27T03:21:01.300194+0000","last_clean":"2024-04-27T03:21:01.300194+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:01.300194+0000","last_undegraded":"2024-04-27T03:21:01.300194+0000","last_fullsized":"2024-04-27T03:21:01.300194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'3","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.756327+0000","last_change":"2024-04-27T03:17:47.031857+0000","last_active":"2024-04-27T03:21:00.756327+0000","last_peered":"2024-04-27T03:21:00.756327+0000","last_clean":"2024-04-27T03:21:00.756327+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:00.756327+0000","last_undegraded":"2024-04-27T03:21:00.756327+0000","last_fullsized":"2024-04-27T03:21:00.756327+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":11,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":78,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949139+0000","last_change":"2024-04-27T03:17:49.017220+0000","last_active":"2024-04-27T03:21:00.949139+0000","last_peered":"2024-04-27T03:21:00.949139+0000","last_clean":"2024-04-27T03:21:00.949139+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:00.949139+0000","last_undegraded":"2024-04-27T03:21:00.949139+0000","last_fullsized":"2024-04-27T03:21:00.949139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815261+0000","last_change":"2024-04-27T03:17:47.039005+0000","last_active":"2024-04-27T03:19:34.815261+0000","last_peered":"2024-04-27T03:19:34.815261+0000","last_clean":"2024-04-27T03:19:34.815261+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:19:34.815261+0000","last_undegraded":"2024-04-27T03:19:34.815261+0000","last_fullsized":"2024-04-27T03:19:34.815261+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:11.970742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.299434+0000","last_change":"2024-04-27T03:17:49.011708+0000","last_active":"2024-04-27T03:21:01.299434+0000","last_peered":"2024-04-27T03:21:01.299434+0000","last_clean":"2024-04-27T03:21:01.299434+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:01.299434+0000","last_undegraded":"2024-04-27T03:21:01.299434+0000","last_fullsized":"2024-04-27T03:21:01.299434+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:52.139159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.809399+0000","last_change":"2024-04-27T03:17:47.016024+0000","last_active":"2024-04-27T03:19:34.809399+0000","last_peered":"2024-04-27T03:19:34.809399+0000","last_clean":"2024-04-27T03:19:34.809399+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:19:34.809399+0000","last_undegraded":"2024-04-27T03:19:34.809399+0000","last_fullsized":"2024-04-27T03:19:34.809399+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:21:26.457330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.300134+0000","last_change":"2024-04-27T03:17:49.010865+0000","last_active":"2024-04-27T03:21:01.300134+0000","last_peered":"2024-04-27T03:21:01.300134+0000","last_clean":"2024-04-27T03:21:01.300134+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:21:01.300134+0000","last_undegraded":"2024-04-27T03:21:01.300134+0000","last_fullsized":"2024-04-27T03:21:01.300134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:16.824300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.810923+0000","last_change":"2024-04-27T03:17:47.019986+0000","last_active":"2024-04-27T03:19:34.810923+0000","last_peered":"2024-04-27T03:19:34.810923+0000","last_clean":"2024-04-27T03:19:34.810923+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:19:34.810923+0000","last_undegraded":"2024-04-27T03:19:34.810923+0000","last_fullsized":"2024-04-27T03:19:34.810923+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:51:05.189684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.949386+0000","last_change":"2024-04-27T03:17:49.005667+0000","last_active":"2024-04-27T03:21:00.949386+0000","last_peered":"2024-04-27T03:21:00.949386+0000","last_clean":"2024-04-27T03:21:00.949386+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:00.949386+0000","last_undegraded":"2024-04-27T03:21:00.949386+0000","last_fullsized":"2024-04-27T03:21:00.949386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:07.432944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.817074+0000","last_change":"2024-04-27T03:17:47.023505+0000","last_active":"2024-04-27T03:19:34.817074+0000","last_peered":"2024-04-27T03:19:34.817074+0000","last_clean":"2024-04-27T03:19:34.817074+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:19:34.817074+0000","last_undegraded":"2024-04-27T03:19:34.817074+0000","last_fullsized":"2024-04-27T03:19:34.817074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382540+0000","last_change":"2024-04-27T03:17:49.009462+0000","last_active":"2024-04-27T03:19:35.382540+0000","last_peered":"2024-04-27T03:19:35.382540+0000","last_clean":"2024-04-27T03:19:35.382540+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:19:35.382540+0000","last_undegraded":"2024-04-27T03:19:35.382540+0000","last_fullsized":"2024-04-27T03:19:35.382540+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:32.201166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.382267+0000","last_change":"2024-04-27T03:17:47.015754+0000","last_active":"2024-04-27T03:19:35.382267+0000","last_peered":"2024-04-27T03:19:35.382267+0000","last_clean":"2024-04-27T03:19:35.382267+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:19:35.382267+0000","last_undegraded":"2024-04-27T03:19:35.382267+0000","last_fullsized":"2024-04-27T03:19:35.382267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:01.063116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'7","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744638+0000","last_change":"2024-04-27T03:17:49.017194+0000","last_active":"2024-04-27T03:20:07.744638+0000","last_peered":"2024-04-27T03:20:07.744638+0000","last_clean":"2024-04-27T03:20:07.744638+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:20:07.744638+0000","last_undegraded":"2024-04-27T03:20:07.744638+0000","last_fullsized":"2024-04-27T03:20:07.744638+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:19.952083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.812147+0000","last_change":"2024-04-27T03:17:47.038806+0000","last_active":"2024-04-27T03:19:34.812147+0000","last_peered":"2024-04-27T03:19:34.812147+0000","last_clean":"2024-04-27T03:19:34.812147+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:19:34.812147+0000","last_undegraded":"2024-04-27T03:19:34.812147+0000","last_fullsized":"2024-04-27T03:19:34.812147+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:36.367478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.808067+0000","last_change":"2024-04-27T03:17:49.006535+0000","last_active":"2024-04-27T03:21:00.808067+0000","last_peered":"2024-04-27T03:21:00.808067+0000","last_clean":"2024-04-27T03:21:00.808067+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:00.808067+0000","last_undegraded":"2024-04-27T03:21:00.808067+0000","last_fullsized":"2024-04-27T03:21:00.808067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:45:21.664514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442247+0000","last_change":"2024-04-27T03:19:35.441020+0000","last_active":"2024-04-27T03:19:35.442247+0000","last_peered":"2024-04-27T03:19:35.442247+0000","last_clean":"2024-04-27T03:19:35.442247+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:19:35.442247+0000","last_undegraded":"2024-04-27T03:19:35.442247+0000","last_fullsized":"2024-04-27T03:19:35.442247+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:39:26.708815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814170+0000","last_change":"2024-04-27T03:17:49.016545+0000","last_active":"2024-04-27T03:19:34.814170+0000","last_peered":"2024-04-27T03:19:34.814170+0000","last_clean":"2024-04-27T03:19:34.814170+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:19:34.814170+0000","last_undegraded":"2024-04-27T03:19:34.814170+0000","last_fullsized":"2024-04-27T03:19:34.814170+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:45:52.886758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.442831+0000","last_change":"2024-04-27T03:17:47.022441+0000","last_active":"2024-04-27T03:19:35.442831+0000","last_peered":"2024-04-27T03:19:35.442831+0000","last_clean":"2024-04-27T03:19:35.442831+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:19:35.442831+0000","last_undegraded":"2024-04-27T03:19:35.442831+0000","last_fullsized":"2024-04-27T03:19:35.442831+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:42.246589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.146608+0000","last_change":"2024-04-27T03:17:49.018017+0000","last_active":"2024-04-27T03:21:01.146608+0000","last_peered":"2024-04-27T03:21:01.146608+0000","last_clean":"2024-04-27T03:21:01.146608+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:01.146608+0000","last_undegraded":"2024-04-27T03:21:01.146608+0000","last_fullsized":"2024-04-27T03:21:01.146608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:29:16.827083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.815679+0000","last_change":"2024-04-27T03:17:47.026063+0000","last_active":"2024-04-27T03:19:34.815679+0000","last_peered":"2024-04-27T03:19:34.815679+0000","last_clean":"2024-04-27T03:19:34.815679+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:19:34.815679+0000","last_undegraded":"2024-04-27T03:19:34.815679+0000","last_fullsized":"2024-04-27T03:19:34.815679+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:05:20.514508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:34.814107+0000","last_change":"2024-04-27T03:17:47.002843+0000","last_active":"2024-04-27T03:19:34.814107+0000","last_peered":"2024-04-27T03:19:34.814107+0000","last_clean":"2024-04-27T03:19:34.814107+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:19:34.814107+0000","last_undegraded":"2024-04-27T03:19:34.814107+0000","last_fullsized":"2024-04-27T03:19:34.814107+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:14:13.287916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'11","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164859+0000","last_change":"2024-04-27T03:17:49.014133+0000","last_active":"2024-04-27T03:21:01.164859+0000","last_peered":"2024-04-27T03:21:01.164859+0000","last_clean":"2024-04-27T03:21:01.164859+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:01.164859+0000","last_undegraded":"2024-04-27T03:21:01.164859+0000","last_fullsized":"2024-04-27T03:21:01.164859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:14:38.383961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.379782+0000","last_change":"2024-04-27T03:17:47.022513+0000","last_active":"2024-04-27T03:19:35.379782+0000","last_peered":"2024-04-27T03:19:35.379782+0000","last_clean":"2024-04-27T03:19:35.379782+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:19:35.379782+0000","last_undegraded":"2024-04-27T03:19:35.379782+0000","last_fullsized":"2024-04-27T03:19:35.379782+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:24.322205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:35.385155+0000","last_change":"2024-04-27T03:17:49.012955+0000","last_active":"2024-04-27T03:19:35.385155+0000","last_peered":"2024-04-27T03:19:35.385155+0000","last_clean":"2024-04-27T03:19:35.385155+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:19:35.385155+0000","last_undegraded":"2024-04-27T03:19:35.385155+0000","last_fullsized":"2024-04-27T03:19:35.385155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:58:35.198710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2256,"ondisk_log_size":2256,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":13384,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1690,"num_read_kb":1444,"num_write":723,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":10824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1470,"ondisk_log_size":1470,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":961,"num_read_kb":961,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":473,"ondisk_log_size":473,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":25805696,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":138,"num_read_kb":87,"num_write":309,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38719488,"data_stored":38719488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46056,"kb_used_data":6560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94325784,"statfs":{"total":96636764160,"available":96589602816,"internally_reserved":0,"allocated":6717440,"data_stored":6440144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57224,"kb_used_data":13568,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94314616,"statfs":{"total":96636764160,"available":96578166784,"internally_reserved":0,"allocated":13893632,"data_stored":13644798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705709,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36540,"kb_used_data":9524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335300,"statfs":{"total":96636764160,"available":96599347200,"internally_reserved":0,"allocated":9752576,"data_stored":9451527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50504,"kb_used_data":11008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94321336,"statfs":{"total":96636764160,"available":96585048064,"internally_reserved":0,"allocated":11272192,"data_stored":10958989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10809344,"data_stored":10809344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8712192,"data_stored":8712192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":12906496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":4422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:21:13.131 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.131 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.131 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.131 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.131 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.132 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.133 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.134 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.135 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 8.1b last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.136 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.137 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.138 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 8.7 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.139 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 8.1 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.140 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 8.3 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.141 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 8.9 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.142 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 8.a last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 8.b last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.143 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.144 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 8.e last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.145 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.146 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 8.d last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 8.1f last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.147 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.148 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.149 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 8.13 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.150 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.151 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.152 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.153 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.154 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.155 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.156 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.157 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.158 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.158 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.158 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.158 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.158 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.158 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:13.158 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-27T03:21:33.159 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:21:33.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff1a1e8b640 1 Processor -- start 2024-04-27T03:21:33.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff1a1e8b640 1 -- start start 2024-04-27T03:21:33.291 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff1a1e8b640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff19c06f2a0 0x7ff19c06f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff1a1e8b640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19c06ffd0 con 0x7ff19c06fbc0 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff1a1e8b640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7ff19c070cf0 con 0x7ff19c06f2a0 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff19b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7ff19c06f2a0 0x7ff19c06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff19affd640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7ff19c06fbc0 0x7ff19c12e3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:42594/0 (socket says 172.21.15.77:42594) 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.281+0000 7ff19affd640 1 -- 172.21.15.77:0/1819626330 learned_addr learned my addr 172.21.15.77:0/1819626330 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff19b7fe640 1 -- 172.21.15.77:0/1819626330 >> v1:172.21.15.77:6789/0 conn(0x7ff19c06fbc0 legacy=0x7ff19c12e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff19b7fe640 1 -- 172.21.15.77:0/1819626330 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c12eaf0 con 0x7ff19c06f2a0 2024-04-27T03:21:33.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff19a7fc640 1 -- 172.21.15.77:0/1819626330 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009961181 0 0) 0x7ff19c06ffd0 con 0x7ff19c06fbc0 2024-04-27T03:21:33.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff19b7fe640 1 --2- 172.21.15.77:0/1819626330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c06f680 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009740 tx=0x7ff18c01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5122c3223e12360 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:33.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff19a7fc640 1 -- 172.21.15.77:0/1819626330 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff18c0b0070 con 0x7ff19c06f2a0 2024-04-27T03:21:33.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff19a7fc640 1 -- 172.21.15.77:0/1819626330 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff18c0ab530 con 0x7ff19c06f2a0 2024-04-27T03:21:33.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff19a7fc640 1 -- 172.21.15.77:0/1819626330 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff18c0ab830 con 0x7ff19c06f2a0 2024-04-27T03:21:33.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/1819626330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 msgr2=0x7ff19c06f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:33.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff1a1e8b640 1 --2- 172.21.15.77:0/1819626330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c06f680 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009740 tx=0x7ff18c01c370 comp rx=0 tx=0).stop 2024-04-27T03:21:33.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/1819626330 shutdown_connections 2024-04-27T03:21:33.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff1a1e8b640 1 --2- 172.21.15.77:0/1819626330 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c06f680 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:33.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/1819626330 >> 172.21.15.77:0/1819626330 conn(0x7ff19c09e0d0 msgr2=0x7ff19c06d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:33.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/1819626330 shutdown_connections 2024-04-27T03:21:33.296 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.285+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/1819626330 wait complete. 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff1a1e8b640 1 Processor -- start 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff1a1e8b640 1 -- start start 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff1a1e8b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c1c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff1a1e8b640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c070ab0 con 0x7ff19c06f2a0 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff19b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c1c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff19b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c1c6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47894/0 (socket says 172.21.15.77:47894) 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff19b7fe640 1 -- 172.21.15.77:0/3813375503 learned_addr learned my addr 172.21.15.77:0/3813375503 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff19b7fe640 1 -- 172.21.15.77:0/3813375503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c12dff0 con 0x7ff19c06f2a0 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff19b7fe640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c1c6450 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff18c019fa0 tx=0x7ff18c01abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff183fff640 1 -- 172.21.15.77:0/3813375503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff18c0b0050 con 0x7ff19c06f2a0 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff183fff640 1 -- 172.21.15.77:0/3813375503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff18c0b1040 con 0x7ff19c06f2a0 2024-04-27T03:21:33.298 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff183fff640 1 -- 172.21.15.77:0/3813375503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff18c004170 con 0x7ff19c06f2a0 2024-04-27T03:21:33.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19c1177e0 con 0x7ff19c06f2a0 2024-04-27T03:21:33.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19c1c6990 con 0x7ff19c06f2a0 2024-04-27T03:21:33.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff19c117fb0 con 0x7ff19c06f2a0 2024-04-27T03:21:33.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff183fff640 1 -- 172.21.15.77:0/3813375503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff18c0be060 con 0x7ff19c06f2a0 2024-04-27T03:21:33.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff183fff640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff188035d60 0x7ff188038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:33.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.289+0000 7ff183fff640 1 -- 172.21.15.77:0/3813375503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7ff18c0e3580 con 0x7ff19c06f2a0 2024-04-27T03:21:33.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.293+0000 7ff19affd640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff188035d60 0x7ff188038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:33.300 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.293+0000 7ff19affd640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff188035d60 0x7ff188038220 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff190007fd0 tx=0x7ff190007920 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:33.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.297+0000 7ff183fff640 1 -- 172.21.15.77:0/3813375503 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff19c12dff0 con 0x7ff19c06f2a0 2024-04-27T03:21:33.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.453+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff19c062e50 con 0x7ff188035d60 2024-04-27T03:21:33.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.501+0000 7ff183fff640 1 -- 172.21.15.77:0/3813375503 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+658201 (secure 0 0 0) 0x7ff19c062e50 con 0x7ff188035d60 2024-04-27T03:21:33.515 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:21:33.521 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:21:33.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff188035d60 msgr2=0x7ff188038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:33.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff188035d60 0x7ff188038220 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff190007fd0 tx=0x7ff190007920 comp rx=0 tx=0).stop 2024-04-27T03:21:33.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 msgr2=0x7ff19c1c6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:33.522 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c1c6450 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7ff18c019fa0 tx=0x7ff18c01abb0 comp rx=0 tx=0).stop 2024-04-27T03:21:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 shutdown_connections 2024-04-27T03:21:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7ff188035d60 0x7ff188038220 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 --2- 172.21.15.77:0/3813375503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff19c06f2a0 0x7ff19c1c6450 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 >> 172.21.15.77:0/3813375503 conn(0x7ff19c09e0d0 msgr2=0x7ff19c1295d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 shutdown_connections 2024-04-27T03:21:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:33.509+0000 7ff1a1e8b640 1 -- 172.21.15.77:0/3813375503 wait complete. 2024-04-27T03:21:33.534 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-04-27T03:21:31.744945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605383,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":5393,"num_read_kb":4942,"num_write":2371,"num_write_kb":61374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":542,"num_omap_keys":4,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4911,"ondisk_log_size":4911,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":135864,"kb_used_data":2840,"kb_used_omap":6,"kb_used_meta":132985,"kb_avail":377351496,"statfs":{"total":386547056640,"available":386407931904,"internally_reserved":0,"allocated":2908160,"data_stored":1772550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":136177430},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":542,"num_omap_keys":4,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006870"},"pg_stats":[{"pgid":"3.3f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.986140+0000","last_change":"2024-04-27T03:17:47.025622+0000","last_active":"2024-04-27T03:21:07.986140+0000","last_peered":"2024-04-27T03:21:07.986140+0000","last_clean":"2024-04-27T03:21:07.986140+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:21:07.986140+0000","last_undegraded":"2024-04-27T03:21:07.986140+0000","last_fullsized":"2024-04-27T03:21:07.986140+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'5","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.398557+0000","last_change":"2024-04-27T03:17:49.019025+0000","last_active":"2024-04-27T03:21:01.398557+0000","last_peered":"2024-04-27T03:21:01.398557+0000","last_clean":"2024-04-27T03:21:01.398557+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:01.398557+0000","last_undegraded":"2024-04-27T03:21:01.398557+0000","last_fullsized":"2024-04-27T03:21:01.398557+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.726062+0000","last_change":"2024-04-27T03:17:47.023458+0000","last_active":"2024-04-27T03:19:52.726062+0000","last_peered":"2024-04-27T03:19:52.726062+0000","last_clean":"2024-04-27T03:19:52.726062+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:19:52.726062+0000","last_undegraded":"2024-04-27T03:19:52.726062+0000","last_fullsized":"2024-04-27T03:19:52.726062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.744411+0000","last_change":"2024-04-27T03:17:49.020940+0000","last_active":"2024-04-27T03:20:22.744411+0000","last_peered":"2024-04-27T03:20:22.744411+0000","last_clean":"2024-04-27T03:20:22.744411+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:20:22.744411+0000","last_undegraded":"2024-04-27T03:20:22.744411+0000","last_fullsized":"2024-04-27T03:20:22.744411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.736697+0000","last_change":"2024-04-27T03:17:47.039594+0000","last_active":"2024-04-27T03:19:52.736697+0000","last_peered":"2024-04-27T03:19:52.736697+0000","last_clean":"2024-04-27T03:19:52.736697+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:19:52.736697+0000","last_undegraded":"2024-04-27T03:19:52.736697+0000","last_fullsized":"2024-04-27T03:19:52.736697+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.744483+0000","last_change":"2024-04-27T03:17:49.016480+0000","last_active":"2024-04-27T03:20:22.744483+0000","last_peered":"2024-04-27T03:20:22.744483+0000","last_clean":"2024-04-27T03:20:22.744483+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:20:22.744483+0000","last_undegraded":"2024-04-27T03:20:22.744483+0000","last_fullsized":"2024-04-27T03:20:22.744483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.731795+0000","last_change":"2024-04-27T03:17:47.021409+0000","last_active":"2024-04-27T03:19:47.731795+0000","last_peered":"2024-04-27T03:19:47.731795+0000","last_clean":"2024-04-27T03:19:47.731795+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:19:47.731795+0000","last_undegraded":"2024-04-27T03:19:47.731795+0000","last_fullsized":"2024-04-27T03:19:47.731795+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.963292+0000","last_change":"2024-04-27T03:17:49.021774+0000","last_active":"2024-04-27T03:21:00.963292+0000","last_peered":"2024-04-27T03:21:00.963292+0000","last_clean":"2024-04-27T03:21:00.963292+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:00.963292+0000","last_undegraded":"2024-04-27T03:21:00.963292+0000","last_fullsized":"2024-04-27T03:21:00.963292+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747231+0000","last_change":"2024-04-27T03:17:47.015865+0000","last_active":"2024-04-27T03:20:17.747231+0000","last_peered":"2024-04-27T03:20:17.747231+0000","last_clean":"2024-04-27T03:20:17.747231+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:20:17.747231+0000","last_undegraded":"2024-04-27T03:20:17.747231+0000","last_fullsized":"2024-04-27T03:20:17.747231+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:37.742362+0000","last_change":"2024-04-27T03:18:34.776780+0000","last_active":"2024-04-27T03:20:37.742362+0000","last_peered":"2024-04-27T03:20:37.742362+0000","last_clean":"2024-04-27T03:20:37.742362+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:20:37.742362+0000","last_undegraded":"2024-04-27T03:20:37.742362+0000","last_fullsized":"2024-04-27T03:20:37.742362+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747191+0000","last_change":"2024-04-27T03:17:47.026206+0000","last_active":"2024-04-27T03:20:17.747191+0000","last_peered":"2024-04-27T03:20:17.747191+0000","last_clean":"2024-04-27T03:20:17.747191+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:20:17.747191+0000","last_undegraded":"2024-04-27T03:20:17.747191+0000","last_fullsized":"2024-04-27T03:20:17.747191+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'11","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397113+0000","last_change":"2024-04-27T03:17:49.015210+0000","last_active":"2024-04-27T03:21:01.397113+0000","last_peered":"2024-04-27T03:21:01.397113+0000","last_clean":"2024-04-27T03:21:01.397113+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:21:01.397113+0000","last_undegraded":"2024-04-27T03:21:01.397113+0000","last_fullsized":"2024-04-27T03:21:01.397113+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.742935+0000","last_change":"2024-04-27T03:17:47.036460+0000","last_active":"2024-04-27T03:20:17.742935+0000","last_peered":"2024-04-27T03:20:17.742935+0000","last_clean":"2024-04-27T03:20:17.742935+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:20:17.742935+0000","last_undegraded":"2024-04-27T03:20:17.742935+0000","last_fullsized":"2024-04-27T03:20:17.742935+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.316299+0000","last_change":"2024-04-27T03:17:49.024190+0000","last_active":"2024-04-27T03:21:01.316299+0000","last_peered":"2024-04-27T03:21:01.316299+0000","last_clean":"2024-04-27T03:21:01.316299+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:01.316299+0000","last_undegraded":"2024-04-27T03:21:01.316299+0000","last_fullsized":"2024-04-27T03:21:01.316299+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.742856+0000","last_change":"2024-04-27T03:17:47.013877+0000","last_active":"2024-04-27T03:20:17.742856+0000","last_peered":"2024-04-27T03:20:17.742856+0000","last_clean":"2024-04-27T03:20:17.742856+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:20:17.742856+0000","last_undegraded":"2024-04-27T03:20:17.742856+0000","last_fullsized":"2024-04-27T03:20:17.742856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.733618+0000","last_change":"2024-04-27T03:17:49.020265+0000","last_active":"2024-04-27T03:19:57.733618+0000","last_peered":"2024-04-27T03:19:57.733618+0000","last_clean":"2024-04-27T03:19:57.733618+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:19:57.733618+0000","last_undegraded":"2024-04-27T03:19:57.733618+0000","last_fullsized":"2024-04-27T03:19:57.733618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.316439+0000","last_change":"2024-04-27T03:17:49.019484+0000","last_active":"2024-04-27T03:21:01.316439+0000","last_peered":"2024-04-27T03:21:01.316439+0000","last_clean":"2024-04-27T03:21:01.316439+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:01.316439+0000","last_undegraded":"2024-04-27T03:21:01.316439+0000","last_fullsized":"2024-04-27T03:21:01.316439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.731193+0000","last_change":"2024-04-27T03:17:47.027209+0000","last_active":"2024-04-27T03:20:12.731193+0000","last_peered":"2024-04-27T03:20:12.731193+0000","last_clean":"2024-04-27T03:20:12.731193+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:20:12.731193+0000","last_undegraded":"2024-04-27T03:20:12.731193+0000","last_fullsized":"2024-04-27T03:20:12.731193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745824+0000","last_change":"2024-04-27T03:17:49.013701+0000","last_active":"2024-04-27T03:20:12.745824+0000","last_peered":"2024-04-27T03:20:12.745824+0000","last_clean":"2024-04-27T03:20:12.745824+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:20:12.745824+0000","last_undegraded":"2024-04-27T03:20:12.745824+0000","last_fullsized":"2024-04-27T03:20:12.745824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.737026+0000","last_change":"2024-04-27T03:17:47.015536+0000","last_active":"2024-04-27T03:20:12.737026+0000","last_peered":"2024-04-27T03:20:12.737026+0000","last_clean":"2024-04-27T03:20:12.737026+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:20:12.737026+0000","last_undegraded":"2024-04-27T03:20:12.737026+0000","last_fullsized":"2024-04-27T03:20:12.737026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.825947+0000","last_change":"2024-04-27T03:17:49.011747+0000","last_active":"2024-04-27T03:21:00.825947+0000","last_peered":"2024-04-27T03:21:00.825947+0000","last_clean":"2024-04-27T03:21:00.825947+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:21:00.825947+0000","last_undegraded":"2024-04-27T03:21:00.825947+0000","last_fullsized":"2024-04-27T03:21:00.825947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.746067+0000","last_change":"2024-04-27T03:17:47.023848+0000","last_active":"2024-04-27T03:20:12.746067+0000","last_peered":"2024-04-27T03:20:12.746067+0000","last_clean":"2024-04-27T03:20:12.746067+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:20:12.746067+0000","last_undegraded":"2024-04-27T03:20:12.746067+0000","last_fullsized":"2024-04-27T03:20:12.746067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248888+0000","last_change":"2024-04-27T03:17:49.019463+0000","last_active":"2024-04-27T03:21:01.248888+0000","last_peered":"2024-04-27T03:21:01.248888+0000","last_clean":"2024-04-27T03:21:01.248888+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:01.248888+0000","last_undegraded":"2024-04-27T03:21:01.248888+0000","last_fullsized":"2024-04-27T03:21:01.248888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745970+0000","last_change":"2024-04-27T03:17:47.012402+0000","last_active":"2024-04-27T03:20:12.745970+0000","last_peered":"2024-04-27T03:20:12.745970+0000","last_clean":"2024-04-27T03:20:12.745970+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:20:12.745970+0000","last_undegraded":"2024-04-27T03:20:12.745970+0000","last_fullsized":"2024-04-27T03:20:12.745970+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.164522+0000","last_change":"2024-04-27T03:17:49.008483+0000","last_active":"2024-04-27T03:21:01.164522+0000","last_peered":"2024-04-27T03:21:01.164522+0000","last_clean":"2024-04-27T03:21:01.164522+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:01.164522+0000","last_undegraded":"2024-04-27T03:21:01.164522+0000","last_fullsized":"2024-04-27T03:21:01.164522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.741798+0000","last_change":"2024-04-27T03:17:47.036082+0000","last_active":"2024-04-27T03:20:12.741798+0000","last_peered":"2024-04-27T03:20:12.741798+0000","last_clean":"2024-04-27T03:20:12.741798+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:20:12.741798+0000","last_undegraded":"2024-04-27T03:20:12.741798+0000","last_fullsized":"2024-04-27T03:20:12.741798+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.165090+0000","last_change":"2024-04-27T03:17:49.005822+0000","last_active":"2024-04-27T03:21:01.165090+0000","last_peered":"2024-04-27T03:21:01.165090+0000","last_clean":"2024-04-27T03:21:01.165090+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:01.165090+0000","last_undegraded":"2024-04-27T03:21:01.165090+0000","last_fullsized":"2024-04-27T03:21:01.165090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740064+0000","last_change":"2024-04-27T03:17:47.033466+0000","last_active":"2024-04-27T03:20:07.740064+0000","last_peered":"2024-04-27T03:20:07.740064+0000","last_clean":"2024-04-27T03:20:07.740064+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:20:07.740064+0000","last_undegraded":"2024-04-27T03:20:07.740064+0000","last_fullsized":"2024-04-27T03:20:07.740064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'37","reported_seq":106,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.340362+0000","last_change":"2024-04-27T03:17:57.088698+0000","last_active":"2024-04-27T03:21:01.340362+0000","last_peered":"2024-04-27T03:21:01.340362+0000","last_clean":"2024-04-27T03:21:01.340362+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:21:01.340362+0000","last_undegraded":"2024-04-27T03:21:01.340362+0000","last_fullsized":"2024-04-27T03:21:01.340362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.247699+0000","last_change":"2024-04-27T03:17:49.015482+0000","last_active":"2024-04-27T03:21:01.247699+0000","last_peered":"2024-04-27T03:21:01.247699+0000","last_clean":"2024-04-27T03:21:01.247699+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:01.247699+0000","last_undegraded":"2024-04-27T03:21:01.247699+0000","last_fullsized":"2024-04-27T03:21:01.247699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740181+0000","last_change":"2024-04-27T03:17:47.018824+0000","last_active":"2024-04-27T03:20:07.740181+0000","last_peered":"2024-04-27T03:20:07.740181+0000","last_clean":"2024-04-27T03:20:07.740181+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:20:07.740181+0000","last_undegraded":"2024-04-27T03:20:07.740181+0000","last_fullsized":"2024-04-27T03:20:07.740181+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.743320+0000","last_change":"2024-04-27T03:17:51.032891+0000","last_active":"2024-04-27T03:20:17.743320+0000","last_peered":"2024-04-27T03:20:17.743320+0000","last_clean":"2024-04-27T03:20:17.743320+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:20:17.743320+0000","last_undegraded":"2024-04-27T03:20:17.743320+0000","last_fullsized":"2024-04-27T03:20:17.743320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'51","reported_seq":122,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.542440+0000","last_change":"2024-04-27T03:21:17.542518+0000","last_active":"2024-04-27T03:21:17.542440+0000","last_peered":"2024-04-27T03:21:17.542440+0000","last_clean":"2024-04-27T03:21:17.542440+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:21:17.542440+0000","last_undegraded":"2024-04-27T03:21:17.542440+0000","last_fullsized":"2024-04-27T03:21:17.542440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'51","last_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_deep_scrub":"31'51","last_deep_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.542266+0000","objects_scrubbed":6,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:24.169053+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.748594+0000","last_change":"2024-04-27T03:17:55.072849+0000","last_active":"2024-04-27T03:20:22.748594+0000","last_peered":"2024-04-27T03:20:22.748594+0000","last_clean":"2024-04-27T03:20:22.748594+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:20:22.748594+0000","last_undegraded":"2024-04-27T03:20:22.748594+0000","last_fullsized":"2024-04-27T03:20:22.748594+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.157534+0000","last_change":"2024-04-27T03:17:57.061550+0000","last_active":"2024-04-27T03:21:01.157534+0000","last_peered":"2024-04-27T03:21:01.157534+0000","last_clean":"2024-04-27T03:21:01.157534+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:01.157534+0000","last_undegraded":"2024-04-27T03:21:01.157534+0000","last_fullsized":"2024-04-27T03:21:01.157534+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.743273+0000","last_change":"2024-04-27T03:17:49.017246+0000","last_active":"2024-04-27T03:20:17.743273+0000","last_peered":"2024-04-27T03:20:17.743273+0000","last_clean":"2024-04-27T03:20:17.743273+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:20:17.743273+0000","last_undegraded":"2024-04-27T03:20:17.743273+0000","last_fullsized":"2024-04-27T03:20:17.743273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.735578+0000","last_change":"2024-04-27T03:17:47.032159+0000","last_active":"2024-04-27T03:20:07.735578+0000","last_peered":"2024-04-27T03:20:07.735578+0000","last_clean":"2024-04-27T03:20:07.735578+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:20:07.735578+0000","last_undegraded":"2024-04-27T03:20:07.735578+0000","last_fullsized":"2024-04-27T03:20:07.735578+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.741892+0000","last_change":"2024-04-27T03:17:51.012211+0000","last_active":"2024-04-27T03:20:12.741892+0000","last_peered":"2024-04-27T03:20:12.741892+0000","last_clean":"2024-04-27T03:20:12.741892+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:20:12.741892+0000","last_undegraded":"2024-04-27T03:20:12.741892+0000","last_fullsized":"2024-04-27T03:20:12.741892+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"31'16","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.287986+0000","last_change":"2024-04-27T03:21:22.288067+0000","last_active":"2024-04-27T03:21:22.287986+0000","last_peered":"2024-04-27T03:21:22.287986+0000","last_clean":"2024-04-27T03:21:22.287986+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:21:22.287986+0000","last_undegraded":"2024-04-27T03:21:22.287986+0000","last_fullsized":"2024-04-27T03:21:22.287986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.287788+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:31.752629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.205494+0000","last_change":"2024-04-27T03:21:23.205547+0000","last_active":"2024-04-27T03:21:23.205494+0000","last_peered":"2024-04-27T03:21:23.205494+0000","last_clean":"2024-04-27T03:21:23.205494+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:21:23.205494+0000","last_undegraded":"2024-04-27T03:21:23.205494+0000","last_fullsized":"2024-04-27T03:21:23.205494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.205266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:32.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.745803+0000","last_change":"2024-04-27T03:17:57.074831+0000","last_active":"2024-04-27T03:20:27.745803+0000","last_peered":"2024-04-27T03:20:27.745803+0000","last_clean":"2024-04-27T03:20:27.745803+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:20:27.745803+0000","last_undegraded":"2024-04-27T03:20:27.745803+0000","last_fullsized":"2024-04-27T03:20:27.745803+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'15","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397950+0000","last_change":"2024-04-27T03:17:49.022969+0000","last_active":"2024-04-27T03:21:01.397950+0000","last_peered":"2024-04-27T03:21:01.397950+0000","last_clean":"2024-04-27T03:21:01.397950+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:01.397950+0000","last_undegraded":"2024-04-27T03:21:01.397950+0000","last_fullsized":"2024-04-27T03:21:01.397950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744546+0000","last_change":"2024-04-27T03:17:47.009721+0000","last_active":"2024-04-27T03:20:07.744546+0000","last_peered":"2024-04-27T03:20:07.744546+0000","last_clean":"2024-04-27T03:20:07.744546+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:20:07.744546+0000","last_undegraded":"2024-04-27T03:20:07.744546+0000","last_fullsized":"2024-04-27T03:20:07.744546+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.555607+0000","last_change":"2024-04-27T03:21:18.555657+0000","last_active":"2024-04-27T03:21:18.555607+0000","last_peered":"2024-04-27T03:21:18.555607+0000","last_clean":"2024-04-27T03:21:18.555607+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:21:18.555607+0000","last_undegraded":"2024-04-27T03:21:18.555607+0000","last_fullsized":"2024-04-27T03:21:18.555607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.555415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:17.141075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"27'22","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.736815+0000","last_change":"2024-04-27T03:17:53.032443+0000","last_active":"2024-04-27T03:20:12.736815+0000","last_peered":"2024-04-27T03:20:12.736815+0000","last_clean":"2024-04-27T03:20:12.736815+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:20:12.736815+0000","last_undegraded":"2024-04-27T03:20:12.736815+0000","last_fullsized":"2024-04-27T03:20:12.736815+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'285","reported_seq":430,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.249048+0000","last_change":"2024-04-27T03:17:55.069049+0000","last_active":"2024-04-27T03:21:08.249048+0000","last_peered":"2024-04-27T03:21:08.249048+0000","last_clean":"2024-04-27T03:21:08.249048+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:08.249048+0000","last_undegraded":"2024-04-27T03:21:08.249048+0000","last_fullsized":"2024-04-27T03:21:08.249048+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.797908+0000","last_change":"2024-04-27T03:21:13.797972+0000","last_active":"2024-04-27T03:21:13.797908+0000","last_peered":"2024-04-27T03:21:13.797908+0000","last_clean":"2024-04-27T03:21:13.797908+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:21:13.797908+0000","last_undegraded":"2024-04-27T03:21:13.797908+0000","last_fullsized":"2024-04-27T03:21:13.797908+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.797701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:15.106812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.964742+0000","last_change":"2024-04-27T03:17:49.017573+0000","last_active":"2024-04-27T03:21:00.964742+0000","last_peered":"2024-04-27T03:21:00.964742+0000","last_clean":"2024-04-27T03:21:00.964742+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:00.964742+0000","last_undegraded":"2024-04-27T03:21:00.964742+0000","last_fullsized":"2024-04-27T03:21:00.964742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744463+0000","last_change":"2024-04-27T03:17:47.003169+0000","last_active":"2024-04-27T03:20:07.744463+0000","last_peered":"2024-04-27T03:20:07.744463+0000","last_clean":"2024-04-27T03:20:07.744463+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:20:07.744463+0000","last_undegraded":"2024-04-27T03:20:07.744463+0000","last_fullsized":"2024-04-27T03:20:07.744463+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747394+0000","last_change":"2024-04-27T03:17:51.020242+0000","last_active":"2024-04-27T03:20:17.747394+0000","last_peered":"2024-04-27T03:20:17.747394+0000","last_clean":"2024-04-27T03:20:17.747394+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:20:17.747394+0000","last_undegraded":"2024-04-27T03:20:17.747394+0000","last_fullsized":"2024-04-27T03:20:17.747394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'43","reported_seq":108,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.228263+0000","last_change":"2024-04-27T03:21:21.228344+0000","last_active":"2024-04-27T03:21:21.228263+0000","last_peered":"2024-04-27T03:21:21.228263+0000","last_clean":"2024-04-27T03:21:21.228263+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:21:21.228263+0000","last_undegraded":"2024-04-27T03:21:21.228263+0000","last_fullsized":"2024-04-27T03:21:21.228263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'43","last_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_deep_scrub":"31'43","last_deep_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.228072+0000","objects_scrubbed":5,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:05.540210+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'285","reported_seq":418,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.596919+0000","last_change":"2024-04-27T03:21:24.596980+0000","last_active":"2024-04-27T03:21:24.596919+0000","last_peered":"2024-04-27T03:21:24.596919+0000","last_clean":"2024-04-27T03:21:24.596919+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:24.596919+0000","last_undegraded":"2024-04-27T03:21:24.596919+0000","last_fullsized":"2024-04-27T03:21:24.596919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.596708+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:43.207566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'31","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.398240+0000","last_change":"2024-04-27T03:18:34.790257+0000","last_active":"2024-04-27T03:21:01.398240+0000","last_peered":"2024-04-27T03:21:01.398240+0000","last_clean":"2024-04-27T03:21:01.398240+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:21:01.398240+0000","last_undegraded":"2024-04-27T03:21:01.398240+0000","last_fullsized":"2024-04-27T03:21:01.398240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.723623+0000","last_change":"2024-04-27T03:17:47.013403+0000","last_active":"2024-04-27T03:21:00.723623+0000","last_peered":"2024-04-27T03:21:00.723623+0000","last_clean":"2024-04-27T03:21:00.723623+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:21:00.723623+0000","last_undegraded":"2024-04-27T03:21:00.723623+0000","last_fullsized":"2024-04-27T03:21:00.723623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.727897+0000","last_change":"2024-04-27T03:17:51.027190+0000","last_active":"2024-04-27T03:20:02.727897+0000","last_peered":"2024-04-27T03:20:02.727897+0000","last_clean":"2024-04-27T03:20:02.727897+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:20:02.727897+0000","last_undegraded":"2024-04-27T03:20:02.727897+0000","last_fullsized":"2024-04-27T03:20:02.727897+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'59","reported_seq":129,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.203730+0000","last_change":"2024-04-27T03:21:26.203825+0000","last_active":"2024-04-27T03:21:26.203730+0000","last_peered":"2024-04-27T03:21:26.203730+0000","last_clean":"2024-04-27T03:21:26.203730+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:21:26.203730+0000","last_undegraded":"2024-04-27T03:21:26.203730+0000","last_fullsized":"2024-04-27T03:21:26.203730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'59","last_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_deep_scrub":"31'59","last_deep_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.203534+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:44.814392+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":47,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.260194+0000","last_change":"2024-04-27T03:21:15.260248+0000","last_active":"2024-04-27T03:21:15.260194+0000","last_peered":"2024-04-27T03:21:15.260194+0000","last_clean":"2024-04-27T03:21:15.260194+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:21:15.260194+0000","last_undegraded":"2024-04-27T03:21:15.260194+0000","last_fullsized":"2024-04-27T03:21:15.260194+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.259998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:59.062074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.825785+0000","last_change":"2024-04-27T03:21:25.825848+0000","last_active":"2024-04-27T03:21:25.825785+0000","last_peered":"2024-04-27T03:21:25.825785+0000","last_clean":"2024-04-27T03:21:25.825785+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:21:25.825785+0000","last_undegraded":"2024-04-27T03:21:25.825785+0000","last_fullsized":"2024-04-27T03:21:25.825785+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'3","last_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_deep_scrub":"29'3","last_deep_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.825552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:43:06.575600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.315602+0000","last_change":"2024-04-27T03:17:49.003206+0000","last_active":"2024-04-27T03:21:01.315602+0000","last_peered":"2024-04-27T03:21:01.315602+0000","last_clean":"2024-04-27T03:21:01.315602+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:01.315602+0000","last_undegraded":"2024-04-27T03:21:01.315602+0000","last_fullsized":"2024-04-27T03:21:01.315602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743585+0000","last_change":"2024-04-27T03:17:46.995080+0000","last_active":"2024-04-27T03:20:02.743585+0000","last_peered":"2024-04-27T03:20:02.743585+0000","last_clean":"2024-04-27T03:20:02.743585+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:20:02.743585+0000","last_undegraded":"2024-04-27T03:20:02.743585+0000","last_fullsized":"2024-04-27T03:20:02.743585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.734327+0000","last_change":"2024-04-27T03:17:51.036969+0000","last_active":"2024-04-27T03:20:02.734327+0000","last_peered":"2024-04-27T03:20:02.734327+0000","last_clean":"2024-04-27T03:20:02.734327+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:20:02.734327+0000","last_undegraded":"2024-04-27T03:20:02.734327+0000","last_fullsized":"2024-04-27T03:20:02.734327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'44","reported_seq":109,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.518797+0000","last_change":"2024-04-27T03:21:13.518875+0000","last_active":"2024-04-27T03:21:13.518797+0000","last_peered":"2024-04-27T03:21:13.518797+0000","last_clean":"2024-04-27T03:21:13.518797+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:21:13.518797+0000","last_undegraded":"2024-04-27T03:21:13.518797+0000","last_fullsized":"2024-04-27T03:21:13.518797+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'44","last_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_deep_scrub":"31'44","last_deep_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.518592+0000","objects_scrubbed":5,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:24:22.864770+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740035+0000","last_change":"2024-04-27T03:17:55.075008+0000","last_active":"2024-04-27T03:20:07.740035+0000","last_peered":"2024-04-27T03:20:07.740035+0000","last_clean":"2024-04-27T03:20:07.740035+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:20:07.740035+0000","last_undegraded":"2024-04-27T03:20:07.740035+0000","last_fullsized":"2024-04-27T03:20:07.740035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"29'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.728245+0000","last_change":"2024-04-27T03:17:57.084457+0000","last_active":"2024-04-27T03:20:02.728245+0000","last_peered":"2024-04-27T03:20:02.728245+0000","last_clean":"2024-04-27T03:20:02.728245+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:20:02.728245+0000","last_undegraded":"2024-04-27T03:20:02.728245+0000","last_fullsized":"2024-04-27T03:20:02.728245+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397663+0000","last_change":"2024-04-27T03:18:34.767220+0000","last_active":"2024-04-27T03:21:01.397663+0000","last_peered":"2024-04-27T03:21:01.397663+0000","last_clean":"2024-04-27T03:21:01.397663+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:01.397663+0000","last_undegraded":"2024-04-27T03:21:01.397663+0000","last_fullsized":"2024-04-27T03:21:01.397663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743552+0000","last_change":"2024-04-27T03:17:47.028376+0000","last_active":"2024-04-27T03:20:02.743552+0000","last_peered":"2024-04-27T03:20:02.743552+0000","last_clean":"2024-04-27T03:20:02.743552+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:20:02.743552+0000","last_undegraded":"2024-04-27T03:20:02.743552+0000","last_fullsized":"2024-04-27T03:20:02.743552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.735622+0000","last_change":"2024-04-27T03:17:51.037525+0000","last_active":"2024-04-27T03:20:07.735622+0000","last_peered":"2024-04-27T03:20:07.735622+0000","last_clean":"2024-04-27T03:20:07.735622+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:20:07.735622+0000","last_undegraded":"2024-04-27T03:20:07.735622+0000","last_fullsized":"2024-04-27T03:20:07.735622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'51","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.091926+0000","last_change":"2024-04-27T03:18:34.800954+0000","last_active":"2024-04-27T03:21:08.091926+0000","last_peered":"2024-04-27T03:21:08.091926+0000","last_clean":"2024-04-27T03:21:08.091926+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:21:08.091926+0000","last_undegraded":"2024-04-27T03:21:08.091926+0000","last_fullsized":"2024-04-27T03:21:08.091926+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'285","reported_seq":393,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.866360+0000","last_change":"2024-04-27T03:21:20.866425+0000","last_active":"2024-04-27T03:21:20.866360+0000","last_peered":"2024-04-27T03:21:20.866360+0000","last_clean":"2024-04-27T03:21:20.866360+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:20.866360+0000","last_undegraded":"2024-04-27T03:21:20.866360+0000","last_fullsized":"2024-04-27T03:21:20.866360+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.866168+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:18.678233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729676+0000","last_change":"2024-04-27T03:17:57.077044+0000","last_active":"2024-04-27T03:20:07.729676+0000","last_peered":"2024-04-27T03:20:07.729676+0000","last_clean":"2024-04-27T03:20:07.729676+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:20:07.729676+0000","last_undegraded":"2024-04-27T03:20:07.729676+0000","last_fullsized":"2024-04-27T03:20:07.729676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.317965+0000","last_change":"2024-04-27T03:17:49.007911+0000","last_active":"2024-04-27T03:21:01.317965+0000","last_peered":"2024-04-27T03:21:01.317965+0000","last_clean":"2024-04-27T03:21:01.317965+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:01.317965+0000","last_undegraded":"2024-04-27T03:21:01.317965+0000","last_fullsized":"2024-04-27T03:21:01.317965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743488+0000","last_change":"2024-04-27T03:18:34.772611+0000","last_active":"2024-04-27T03:20:02.743488+0000","last_peered":"2024-04-27T03:20:02.743488+0000","last_clean":"2024-04-27T03:20:02.743488+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:20:02.743488+0000","last_undegraded":"2024-04-27T03:20:02.743488+0000","last_fullsized":"2024-04-27T03:20:02.743488+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.734434+0000","last_change":"2024-04-27T03:17:51.038976+0000","last_active":"2024-04-27T03:20:02.734434+0000","last_peered":"2024-04-27T03:20:02.734434+0000","last_clean":"2024-04-27T03:20:02.734434+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:20:02.734434+0000","last_undegraded":"2024-04-27T03:20:02.734434+0000","last_fullsized":"2024-04-27T03:20:02.734434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'77","reported_seq":166,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.631941+0000","last_change":"2024-04-27T03:21:27.632024+0000","last_active":"2024-04-27T03:21:27.631941+0000","last_peered":"2024-04-27T03:21:27.631941+0000","last_clean":"2024-04-27T03:21:27.631941+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:21:27.631941+0000","last_undegraded":"2024-04-27T03:21:27.631941+0000","last_fullsized":"2024-04-27T03:21:27.631941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'77","last_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_deep_scrub":"31'77","last_deep_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.631741+0000","objects_scrubbed":7,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:01.393467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":34,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.822541+0000","last_change":"2024-04-27T03:21:22.822651+0000","last_active":"2024-04-27T03:21:22.822541+0000","last_peered":"2024-04-27T03:21:22.822541+0000","last_clean":"2024-04-27T03:21:22.822541+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:21:22.822541+0000","last_undegraded":"2024-04-27T03:21:22.822541+0000","last_fullsized":"2024-04-27T03:21:22.822541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.822323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:20.663089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'18","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.465826+0000","last_change":"2024-04-27T03:17:57.062348+0000","last_active":"2024-04-27T03:21:01.465826+0000","last_peered":"2024-04-27T03:21:01.465826+0000","last_clean":"2024-04-27T03:21:01.465826+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:01.465826+0000","last_undegraded":"2024-04-27T03:21:01.465826+0000","last_fullsized":"2024-04-27T03:21:01.465826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.247455+0000","last_change":"2024-04-27T03:17:49.009833+0000","last_active":"2024-04-27T03:21:01.247455+0000","last_peered":"2024-04-27T03:21:01.247455+0000","last_clean":"2024-04-27T03:21:01.247455+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:01.247455+0000","last_undegraded":"2024-04-27T03:21:01.247455+0000","last_fullsized":"2024-04-27T03:21:01.247455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:52.745487+0000","last_change":"2024-04-27T03:18:34.767518+0000","last_active":"2024-04-27T03:20:52.745487+0000","last_peered":"2024-04-27T03:20:52.745487+0000","last_clean":"2024-04-27T03:20:52.745487+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:20:52.745487+0000","last_undegraded":"2024-04-27T03:20:52.745487+0000","last_fullsized":"2024-04-27T03:20:52.745487+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.735716+0000","last_change":"2024-04-27T03:17:51.029195+0000","last_active":"2024-04-27T03:20:07.735716+0000","last_peered":"2024-04-27T03:20:07.735716+0000","last_clean":"2024-04-27T03:20:07.735716+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:20:07.735716+0000","last_undegraded":"2024-04-27T03:20:07.735716+0000","last_fullsized":"2024-04-27T03:20:07.735716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'43","reported_seq":101,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.218452+0000","last_change":"2024-04-27T03:17:53.049067+0000","last_active":"2024-04-27T03:21:08.218452+0000","last_peered":"2024-04-27T03:21:08.218452+0000","last_clean":"2024-04-27T03:21:08.218452+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:21:08.218452+0000","last_undegraded":"2024-04-27T03:21:08.218452+0000","last_fullsized":"2024-04-27T03:21:08.218452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":63,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.585696+0000","last_change":"2024-04-27T03:21:23.585759+0000","last_active":"2024-04-27T03:21:23.585696+0000","last_peered":"2024-04-27T03:21:23.585696+0000","last_clean":"2024-04-27T03:21:23.585696+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:21:23.585696+0000","last_undegraded":"2024-04-27T03:21:23.585696+0000","last_fullsized":"2024-04-27T03:21:23.585696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.585484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:34.196580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.406842+0000","last_change":"2024-04-27T03:17:57.080943+0000","last_active":"2024-04-27T03:21:01.406842+0000","last_peered":"2024-04-27T03:21:01.406842+0000","last_clean":"2024-04-27T03:21:01.406842+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:21:01.406842+0000","last_undegraded":"2024-04-27T03:21:01.406842+0000","last_fullsized":"2024-04-27T03:21:01.406842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744769+0000","last_change":"2024-04-27T03:17:49.016254+0000","last_active":"2024-04-27T03:20:07.744769+0000","last_peered":"2024-04-27T03:20:07.744769+0000","last_clean":"2024-04-27T03:20:07.744769+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:20:07.744769+0000","last_undegraded":"2024-04-27T03:20:07.744769+0000","last_fullsized":"2024-04-27T03:20:07.744769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742240+0000","last_change":"2024-04-27T03:17:47.026419+0000","last_active":"2024-04-27T03:19:57.742240+0000","last_peered":"2024-04-27T03:19:57.742240+0000","last_clean":"2024-04-27T03:19:57.742240+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:19:57.742240+0000","last_undegraded":"2024-04-27T03:19:57.742240+0000","last_fullsized":"2024-04-27T03:19:57.742240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.324797+0000","last_change":"2024-04-27T03:21:12.324853+0000","last_active":"2024-04-27T03:21:12.324797+0000","last_peered":"2024-04-27T03:21:12.324797+0000","last_clean":"2024-04-27T03:21:12.324797+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:21:12.324797+0000","last_undegraded":"2024-04-27T03:21:12.324797+0000","last_fullsized":"2024-04-27T03:21:12.324797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.324599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:41:24.652668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'45","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.786964+0000","last_change":"2024-04-27T03:21:26.787057+0000","last_active":"2024-04-27T03:21:26.786964+0000","last_peered":"2024-04-27T03:21:26.786964+0000","last_clean":"2024-04-27T03:21:26.786964+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:21:26.786964+0000","last_undegraded":"2024-04-27T03:21:26.786964+0000","last_fullsized":"2024-04-27T03:21:26.786964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.786765+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:28.251865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'285","reported_seq":420,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.779843+0000","last_change":"2024-04-27T03:21:18.779964+0000","last_active":"2024-04-27T03:21:18.779843+0000","last_peered":"2024-04-27T03:21:18.779843+0000","last_clean":"2024-04-27T03:21:18.779843+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:18.779843+0000","last_undegraded":"2024-04-27T03:21:18.779843+0000","last_fullsized":"2024-04-27T03:21:18.779843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.779620+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:58:08.955224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'12","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.412099+0000","last_change":"2024-04-27T03:17:57.077926+0000","last_active":"2024-04-27T03:21:01.412099+0000","last_peered":"2024-04-27T03:21:01.412099+0000","last_clean":"2024-04-27T03:21:01.412099+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:21:01.412099+0000","last_undegraded":"2024-04-27T03:21:01.412099+0000","last_fullsized":"2024-04-27T03:21:01.412099+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.735762+0000","last_change":"2024-04-27T03:17:49.008951+0000","last_active":"2024-04-27T03:20:07.735762+0000","last_peered":"2024-04-27T03:20:07.735762+0000","last_clean":"2024-04-27T03:20:07.735762+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:20:07.735762+0000","last_undegraded":"2024-04-27T03:20:07.735762+0000","last_fullsized":"2024-04-27T03:20:07.735762+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737562+0000","last_change":"2024-04-27T03:17:47.036712+0000","last_active":"2024-04-27T03:19:57.737562+0000","last_peered":"2024-04-27T03:19:57.737562+0000","last_clean":"2024-04-27T03:19:57.737562+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:19:57.737562+0000","last_undegraded":"2024-04-27T03:19:57.737562+0000","last_fullsized":"2024-04-27T03:19:57.737562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'5","reported_seq":432,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.080708+0000","last_change":"2024-04-27T03:21:14.080869+0000","last_active":"2024-04-27T03:21:14.080708+0000","last_peered":"2024-04-27T03:21:14.080708+0000","last_clean":"2024-04-27T03:21:14.080708+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:21:14.080708+0000","last_undegraded":"2024-04-27T03:21:14.080708+0000","last_fullsized":"2024-04-27T03:21:14.080708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.080480+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:27.352786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":385,"num_read_kb":385,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"31'16","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.621515+0000","last_change":"2024-04-27T03:21:20.621586+0000","last_active":"2024-04-27T03:21:20.621515+0000","last_peered":"2024-04-27T03:21:20.621515+0000","last_clean":"2024-04-27T03:21:20.621515+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:21:20.621515+0000","last_undegraded":"2024-04-27T03:21:20.621515+0000","last_fullsized":"2024-04-27T03:21:20.621515+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.621330+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:04.933468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'285","reported_seq":468,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.253700+0000","last_change":"2024-04-27T03:17:55.068988+0000","last_active":"2024-04-27T03:21:08.253700+0000","last_peered":"2024-04-27T03:21:08.253700+0000","last_clean":"2024-04-27T03:21:08.253700+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:08.253700+0000","last_undegraded":"2024-04-27T03:21:08.253700+0000","last_fullsized":"2024-04-27T03:21:08.253700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.766179+0000","last_change":"2024-04-27T03:21:17.766241+0000","last_active":"2024-04-27T03:21:17.766179+0000","last_peered":"2024-04-27T03:21:17.766179+0000","last_clean":"2024-04-27T03:21:17.766179+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:21:17.766179+0000","last_undegraded":"2024-04-27T03:21:17.766179+0000","last_fullsized":"2024-04-27T03:21:17.766179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.765988+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:49.164397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729814+0000","last_change":"2024-04-27T03:17:49.018519+0000","last_active":"2024-04-27T03:20:07.729814+0000","last_peered":"2024-04-27T03:20:07.729814+0000","last_clean":"2024-04-27T03:20:07.729814+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:20:07.729814+0000","last_undegraded":"2024-04-27T03:20:07.729814+0000","last_fullsized":"2024-04-27T03:20:07.729814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737526+0000","last_change":"2024-04-27T03:17:47.031891+0000","last_active":"2024-04-27T03:19:57.737526+0000","last_peered":"2024-04-27T03:19:57.737526+0000","last_clean":"2024-04-27T03:19:57.737526+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:19:57.737526+0000","last_undegraded":"2024-04-27T03:19:57.737526+0000","last_fullsized":"2024-04-27T03:19:57.737526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.628128+0000","last_change":"2024-04-27T03:21:21.628229+0000","last_active":"2024-04-27T03:21:21.628128+0000","last_peered":"2024-04-27T03:21:21.628128+0000","last_clean":"2024-04-27T03:21:21.628128+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:21:21.628128+0000","last_undegraded":"2024-04-27T03:21:21.628128+0000","last_fullsized":"2024-04-27T03:21:21.628128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.627898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:37.847354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'41","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.056086+0000","last_change":"2024-04-27T03:17:53.048795+0000","last_active":"2024-04-27T03:21:08.056086+0000","last_peered":"2024-04-27T03:21:08.056086+0000","last_clean":"2024-04-27T03:21:08.056086+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:21:08.056086+0000","last_undegraded":"2024-04-27T03:21:08.056086+0000","last_fullsized":"2024-04-27T03:21:08.056086+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":37,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745998+0000","last_change":"2024-04-27T03:17:55.071142+0000","last_active":"2024-04-27T03:20:12.745998+0000","last_peered":"2024-04-27T03:20:12.745998+0000","last_clean":"2024-04-27T03:20:12.745998+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:20:12.745998+0000","last_undegraded":"2024-04-27T03:20:12.745998+0000","last_fullsized":"2024-04-27T03:20:12.745998+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'8","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.461550+0000","last_change":"2024-04-27T03:17:57.071496+0000","last_active":"2024-04-27T03:21:01.461550+0000","last_peered":"2024-04-27T03:21:01.461550+0000","last_clean":"2024-04-27T03:21:01.461550+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:01.461550+0000","last_undegraded":"2024-04-27T03:21:01.461550+0000","last_fullsized":"2024-04-27T03:21:01.461550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.826271+0000","last_change":"2024-04-27T03:17:49.013855+0000","last_active":"2024-04-27T03:21:00.826271+0000","last_peered":"2024-04-27T03:21:00.826271+0000","last_clean":"2024-04-27T03:21:00.826271+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:00.826271+0000","last_undegraded":"2024-04-27T03:21:00.826271+0000","last_fullsized":"2024-04-27T03:21:00.826271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":202,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:42.723360+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:19:42.723360+0000","last_peered":"2024-04-27T03:19:42.723360+0000","last_clean":"2024-04-27T03:19:42.723360+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:19:42.723360+0000","last_undegraded":"2024-04-27T03:19:42.723360+0000","last_fullsized":"2024-04-27T03:19:42.723360+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.739939+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:19:47.739939+0000","last_peered":"2024-04-27T03:19:47.739939+0000","last_clean":"2024-04-27T03:19:47.739939+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:19:47.739939+0000","last_undegraded":"2024-04-27T03:19:47.739939+0000","last_fullsized":"2024-04-27T03:19:47.739939+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.106795+0000","last_change":"2024-04-27T03:17:47.039911+0000","last_active":"2024-04-27T03:21:01.106795+0000","last_peered":"2024-04-27T03:21:01.106795+0000","last_clean":"2024-04-27T03:21:01.106795+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:21:01.106795+0000","last_undegraded":"2024-04-27T03:21:01.106795+0000","last_fullsized":"2024-04-27T03:21:01.106795+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.278555+0000","last_change":"2024-04-27T03:21:25.278617+0000","last_active":"2024-04-27T03:21:25.278555+0000","last_peered":"2024-04-27T03:21:25.278555+0000","last_clean":"2024-04-27T03:21:25.278555+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:21:25.278555+0000","last_undegraded":"2024-04-27T03:21:25.278555+0000","last_fullsized":"2024-04-27T03:21:25.278555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.278344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:54.825766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'91","reported_seq":183,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.138746+0000","last_change":"2024-04-27T03:17:53.051108+0000","last_active":"2024-04-27T03:21:08.138746+0000","last_peered":"2024-04-27T03:21:08.138746+0000","last_clean":"2024-04-27T03:21:08.138746+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:21:08.138746+0000","last_undegraded":"2024-04-27T03:21:08.138746+0000","last_fullsized":"2024-04-27T03:21:08.138746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":81,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.635119+0000","last_change":"2024-04-27T03:21:25.635175+0000","last_active":"2024-04-27T03:21:25.635119+0000","last_peered":"2024-04-27T03:21:25.635119+0000","last_clean":"2024-04-27T03:21:25.635119+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:21:25.635119+0000","last_undegraded":"2024-04-27T03:21:25.635119+0000","last_fullsized":"2024-04-27T03:21:25.635119+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.634932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:40.208378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":176,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.825556+0000","last_change":"2024-04-27T03:21:21.825636+0000","last_active":"2024-04-27T03:21:21.825556+0000","last_peered":"2024-04-27T03:21:21.825556+0000","last_clean":"2024-04-27T03:21:21.825556+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:21.825556+0000","last_undegraded":"2024-04-27T03:21:21.825556+0000","last_fullsized":"2024-04-27T03:21:21.825556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'82","last_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_deep_scrub":"31'82","last_deep_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.825358+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:31:11.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248347+0000","last_change":"2024-04-27T03:17:49.007601+0000","last_active":"2024-04-27T03:21:01.248347+0000","last_peered":"2024-04-27T03:21:01.248347+0000","last_clean":"2024-04-27T03:21:01.248347+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:21:01.248347+0000","last_undegraded":"2024-04-27T03:21:01.248347+0000","last_fullsized":"2024-04-27T03:21:01.248347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:42.738740+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:19:42.738740+0000","last_peered":"2024-04-27T03:19:42.738740+0000","last_clean":"2024-04-27T03:19:42.738740+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:19:42.738740+0000","last_undegraded":"2024-04-27T03:19:42.738740+0000","last_fullsized":"2024-04-27T03:19:42.738740+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.725805+0000","last_change":"2024-04-27T03:17:47.015528+0000","last_active":"2024-04-27T03:19:52.725805+0000","last_peered":"2024-04-27T03:19:52.725805+0000","last_clean":"2024-04-27T03:19:52.725805+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:19:52.725805+0000","last_undegraded":"2024-04-27T03:19:52.725805+0000","last_fullsized":"2024-04-27T03:19:52.725805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.174097+0000","last_change":"2024-04-27T03:21:18.174150+0000","last_active":"2024-04-27T03:21:18.174097+0000","last_peered":"2024-04-27T03:21:18.174097+0000","last_clean":"2024-04-27T03:21:18.174097+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:21:18.174097+0000","last_undegraded":"2024-04-27T03:21:18.174097+0000","last_fullsized":"2024-04-27T03:21:18.174097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.173908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:30.769576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'297","reported_seq":533,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.209080+0000","last_change":"2024-04-27T03:21:17.209173+0000","last_active":"2024-04-27T03:21:17.209080+0000","last_peered":"2024-04-27T03:21:17.209080+0000","last_clean":"2024-04-27T03:21:17.209080+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:17.209080+0000","last_undegraded":"2024-04-27T03:21:17.209080+0000","last_fullsized":"2024-04-27T03:21:17.209080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'297","last_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_deep_scrub":"31'297","last_deep_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.208885+0000","objects_scrubbed":11,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:24.543836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":387,"num_read_kb":371,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.591116+0000","last_change":"2024-04-27T03:21:22.591170+0000","last_active":"2024-04-27T03:21:22.591116+0000","last_peered":"2024-04-27T03:21:22.591116+0000","last_clean":"2024-04-27T03:21:22.591116+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:21:22.591116+0000","last_undegraded":"2024-04-27T03:21:22.591116+0000","last_fullsized":"2024-04-27T03:21:22.591116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.590921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:10.482862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729433+0000","last_change":"2024-04-27T03:17:57.086888+0000","last_active":"2024-04-27T03:20:07.729433+0000","last_peered":"2024-04-27T03:20:07.729433+0000","last_clean":"2024-04-27T03:20:07.729433+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:20:07.729433+0000","last_undegraded":"2024-04-27T03:20:07.729433+0000","last_fullsized":"2024-04-27T03:20:07.729433+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248662+0000","last_change":"2024-04-27T03:17:49.014111+0000","last_active":"2024-04-27T03:21:01.248662+0000","last_peered":"2024-04-27T03:21:01.248662+0000","last_clean":"2024-04-27T03:21:01.248662+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:01.248662+0000","last_undegraded":"2024-04-27T03:21:01.248662+0000","last_fullsized":"2024-04-27T03:21:01.248662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:42.730670+0000","last_change":"2024-04-27T03:17:39.633841+0000","last_active":"2024-04-27T03:19:42.730670+0000","last_peered":"2024-04-27T03:19:42.730670+0000","last_clean":"2024-04-27T03:19:42.730670+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:19:42.730670+0000","last_undegraded":"2024-04-27T03:19:42.730670+0000","last_fullsized":"2024-04-27T03:19:42.730670+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'4","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.989957+0000","last_change":"2024-04-27T03:17:47.011695+0000","last_active":"2024-04-27T03:21:07.989957+0000","last_peered":"2024-04-27T03:21:07.989957+0000","last_clean":"2024-04-27T03:21:07.989957+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:21:07.989957+0000","last_undegraded":"2024-04-27T03:21:07.989957+0000","last_fullsized":"2024-04-27T03:21:07.989957+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"20'1","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.734603+0000","last_change":"2024-04-27T03:17:51.031676+0000","last_active":"2024-04-27T03:20:02.734603+0000","last_peered":"2024-04-27T03:20:02.734603+0000","last_clean":"2024-04-27T03:20:02.734603+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:20:02.734603+0000","last_undegraded":"2024-04-27T03:20:02.734603+0000","last_fullsized":"2024-04-27T03:20:02.734603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'23","reported_seq":105,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.192662+0000","last_change":"2024-04-27T03:21:27.192744+0000","last_active":"2024-04-27T03:21:27.192662+0000","last_peered":"2024-04-27T03:21:27.192662+0000","last_clean":"2024-04-27T03:21:27.192662+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:27.192662+0000","last_undegraded":"2024-04-27T03:21:27.192662+0000","last_fullsized":"2024-04-27T03:21:27.192662+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.192466+0000","objects_scrubbed":8,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:41.765912+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.232293+0000","last_change":"2024-04-27T03:21:16.232354+0000","last_active":"2024-04-27T03:21:16.232293+0000","last_peered":"2024-04-27T03:21:16.232293+0000","last_clean":"2024-04-27T03:21:16.232293+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:21:16.232293+0000","last_undegraded":"2024-04-27T03:21:16.232293+0000","last_fullsized":"2024-04-27T03:21:16.232293+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.232084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:05.186600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.309820+0000","last_change":"2024-04-27T03:17:57.071109+0000","last_active":"2024-04-27T03:21:01.309820+0000","last_peered":"2024-04-27T03:21:01.309820+0000","last_clean":"2024-04-27T03:21:01.309820+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:01.309820+0000","last_undegraded":"2024-04-27T03:21:01.309820+0000","last_fullsized":"2024-04-27T03:21:01.309820+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.826921+0000","last_change":"2024-04-27T03:17:49.009764+0000","last_active":"2024-04-27T03:21:00.826921+0000","last_peered":"2024-04-27T03:21:00.826921+0000","last_clean":"2024-04-27T03:21:00.826921+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:00.826921+0000","last_undegraded":"2024-04-27T03:21:00.826921+0000","last_fullsized":"2024-04-27T03:21:00.826921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.739879+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:19:47.739879+0000","last_peered":"2024-04-27T03:19:47.739879+0000","last_clean":"2024-04-27T03:19:47.739879+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:19:47.739879+0000","last_undegraded":"2024-04-27T03:19:47.739879+0000","last_fullsized":"2024-04-27T03:19:47.739879+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.140255+0000","last_change":"2024-04-27T03:17:47.015018+0000","last_active":"2024-04-27T03:21:01.140255+0000","last_peered":"2024-04-27T03:21:01.140255+0000","last_clean":"2024-04-27T03:21:01.140255+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:21:01.140255+0000","last_undegraded":"2024-04-27T03:21:01.140255+0000","last_fullsized":"2024-04-27T03:21:01.140255+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.726952+0000","last_change":"2024-04-27T03:17:51.029533+0000","last_active":"2024-04-27T03:19:57.726952+0000","last_peered":"2024-04-27T03:19:57.726952+0000","last_clean":"2024-04-27T03:19:57.726952+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:19:57.726952+0000","last_undegraded":"2024-04-27T03:19:57.726952+0000","last_fullsized":"2024-04-27T03:19:57.726952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"29'14","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.264944+0000","last_change":"2024-04-27T03:17:53.045958+0000","last_active":"2024-04-27T03:21:00.264944+0000","last_peered":"2024-04-27T03:21:00.264944+0000","last_clean":"2024-04-27T03:21:00.264944+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:00.264944+0000","last_undegraded":"2024-04-27T03:21:00.264944+0000","last_fullsized":"2024-04-27T03:21:00.264944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740471+0000","last_change":"2024-04-27T03:17:55.078514+0000","last_active":"2024-04-27T03:20:07.740471+0000","last_peered":"2024-04-27T03:20:07.740471+0000","last_clean":"2024-04-27T03:20:07.740471+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:20:07.740471+0000","last_undegraded":"2024-04-27T03:20:07.740471+0000","last_fullsized":"2024-04-27T03:20:07.740471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.968086+0000","last_change":"2024-04-27T03:17:57.089242+0000","last_active":"2024-04-27T03:21:00.968086+0000","last_peered":"2024-04-27T03:21:00.968086+0000","last_clean":"2024-04-27T03:21:00.968086+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:21:00.968086+0000","last_undegraded":"2024-04-27T03:21:00.968086+0000","last_fullsized":"2024-04-27T03:21:00.968086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.316306+0000","last_change":"2024-04-27T03:17:49.006116+0000","last_active":"2024-04-27T03:21:01.316306+0000","last_peered":"2024-04-27T03:21:01.316306+0000","last_clean":"2024-04-27T03:21:01.316306+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:01.316306+0000","last_undegraded":"2024-04-27T03:21:01.316306+0000","last_fullsized":"2024-04-27T03:21:01.316306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741133+0000","last_change":"2024-04-27T03:17:47.024776+0000","last_active":"2024-04-27T03:19:52.741133+0000","last_peered":"2024-04-27T03:19:52.741133+0000","last_clean":"2024-04-27T03:19:52.741133+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:19:52.741133+0000","last_undegraded":"2024-04-27T03:19:52.741133+0000","last_fullsized":"2024-04-27T03:19:52.741133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.724644+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:19:47.724644+0000","last_peered":"2024-04-27T03:19:47.724644+0000","last_clean":"2024-04-27T03:19:47.724644+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:19:47.724644+0000","last_undegraded":"2024-04-27T03:19:47.724644+0000","last_fullsized":"2024-04-27T03:19:47.724644+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.872299+0000","last_change":"2024-04-27T03:17:51.037696+0000","last_active":"2024-04-27T03:21:07.872299+0000","last_peered":"2024-04-27T03:21:07.872299+0000","last_clean":"2024-04-27T03:21:07.872299+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:21:07.872299+0000","last_undegraded":"2024-04-27T03:21:07.872299+0000","last_fullsized":"2024-04-27T03:21:07.872299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'87","reported_seq":179,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.199732+0000","last_change":"2024-04-27T03:21:18.199826+0000","last_active":"2024-04-27T03:21:18.199732+0000","last_peered":"2024-04-27T03:21:18.199732+0000","last_clean":"2024-04-27T03:21:18.199732+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:21:18.199732+0000","last_undegraded":"2024-04-27T03:21:18.199732+0000","last_fullsized":"2024-04-27T03:21:18.199732+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'87","last_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_deep_scrub":"31'87","last_deep_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.199538+0000","objects_scrubbed":11,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:49.872043+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":89,"num_read_kb":75,"num_write":42,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":292,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743372+0000","last_change":"2024-04-27T03:17:55.068676+0000","last_active":"2024-04-27T03:20:02.743372+0000","last_peered":"2024-04-27T03:20:02.743372+0000","last_clean":"2024-04-27T03:20:02.743372+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:20:02.743372+0000","last_undegraded":"2024-04-27T03:20:02.743372+0000","last_fullsized":"2024-04-27T03:20:02.743372+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.745180+0000","last_change":"2024-04-27T03:21:15.745242+0000","last_active":"2024-04-27T03:21:15.745180+0000","last_peered":"2024-04-27T03:21:15.745180+0000","last_clean":"2024-04-27T03:21:15.745180+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:21:15.745180+0000","last_undegraded":"2024-04-27T03:21:15.745180+0000","last_fullsized":"2024-04-27T03:21:15.745180+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.744992+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:00.286559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.726883+0000","last_change":"2024-04-27T03:17:49.009882+0000","last_active":"2024-04-27T03:19:57.726883+0000","last_peered":"2024-04-27T03:19:57.726883+0000","last_clean":"2024-04-27T03:19:57.726883+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:19:57.726883+0000","last_undegraded":"2024-04-27T03:19:57.726883+0000","last_fullsized":"2024-04-27T03:19:57.726883+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'6","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.986972+0000","last_change":"2024-04-27T03:17:47.030065+0000","last_active":"2024-04-27T03:21:07.986972+0000","last_peered":"2024-04-27T03:21:07.986972+0000","last_clean":"2024-04-27T03:21:07.986972+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:07.986972+0000","last_undegraded":"2024-04-27T03:21:07.986972+0000","last_fullsized":"2024-04-27T03:21:07.986972+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.740011+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:19:47.740011+0000","last_peered":"2024-04-27T03:19:47.740011+0000","last_clean":"2024-04-27T03:19:47.740011+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:19:47.740011+0000","last_undegraded":"2024-04-27T03:19:47.740011+0000","last_fullsized":"2024-04-27T03:19:47.740011+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.128323+0000","last_change":"2024-04-27T03:21:16.128384+0000","last_active":"2024-04-27T03:21:16.128323+0000","last_peered":"2024-04-27T03:21:16.128323+0000","last_clean":"2024-04-27T03:21:16.128323+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:21:16.128323+0000","last_undegraded":"2024-04-27T03:21:16.128323+0000","last_fullsized":"2024-04-27T03:21:16.128323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.128135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:44.502862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"31'28","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.261647+0000","last_change":"2024-04-27T03:21:21.261734+0000","last_active":"2024-04-27T03:21:21.261647+0000","last_peered":"2024-04-27T03:21:21.261647+0000","last_clean":"2024-04-27T03:21:21.261647+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:21:21.261647+0000","last_undegraded":"2024-04-27T03:21:21.261647+0000","last_fullsized":"2024-04-27T03:21:21.261647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'28","last_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_deep_scrub":"31'28","last_deep_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.261420+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:52.474587+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'285","reported_seq":373,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.251214+0000","last_change":"2024-04-27T03:21:24.251366+0000","last_active":"2024-04-27T03:21:24.251214+0000","last_peered":"2024-04-27T03:21:24.251214+0000","last_clean":"2024-04-27T03:21:24.251214+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:24.251214+0000","last_undegraded":"2024-04-27T03:21:24.251214+0000","last_fullsized":"2024-04-27T03:21:24.251214+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.250979+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:52.625705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.251899+0000","last_change":"2024-04-27T03:17:57.091210+0000","last_active":"2024-04-27T03:21:01.251899+0000","last_peered":"2024-04-27T03:21:01.251899+0000","last_clean":"2024-04-27T03:21:01.251899+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:21:01.251899+0000","last_undegraded":"2024-04-27T03:21:01.251899+0000","last_fullsized":"2024-04-27T03:21:01.251899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.964775+0000","last_change":"2024-04-27T03:17:49.007942+0000","last_active":"2024-04-27T03:21:00.964775+0000","last_peered":"2024-04-27T03:21:00.964775+0000","last_clean":"2024-04-27T03:21:00.964775+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:21:00.964775+0000","last_undegraded":"2024-04-27T03:21:00.964775+0000","last_fullsized":"2024-04-27T03:21:00.964775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737821+0000","last_change":"2024-04-27T03:17:47.039532+0000","last_active":"2024-04-27T03:19:57.737821+0000","last_peered":"2024-04-27T03:19:57.737821+0000","last_clean":"2024-04-27T03:19:57.737821+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:19:57.737821+0000","last_undegraded":"2024-04-27T03:19:57.737821+0000","last_fullsized":"2024-04-27T03:19:57.737821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.724711+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:19:47.724711+0000","last_peered":"2024-04-27T03:19:47.724711+0000","last_clean":"2024-04-27T03:19:47.724711+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:19:47.724711+0000","last_undegraded":"2024-04-27T03:19:47.724711+0000","last_fullsized":"2024-04-27T03:19:47.724711+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742149+0000","last_change":"2024-04-27T03:17:51.020247+0000","last_active":"2024-04-27T03:19:57.742149+0000","last_peered":"2024-04-27T03:19:57.742149+0000","last_clean":"2024-04-27T03:19:57.742149+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:19:57.742149+0000","last_undegraded":"2024-04-27T03:19:57.742149+0000","last_fullsized":"2024-04-27T03:19:57.742149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"27'16","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737626+0000","last_change":"2024-04-27T03:17:53.048021+0000","last_active":"2024-04-27T03:19:57.737626+0000","last_peered":"2024-04-27T03:19:57.737626+0000","last_clean":"2024-04-27T03:19:57.737626+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:19:57.737626+0000","last_undegraded":"2024-04-27T03:19:57.737626+0000","last_fullsized":"2024-04-27T03:19:57.737626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'285","reported_seq":421,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.256555+0000","last_change":"2024-04-27T03:17:55.075368+0000","last_active":"2024-04-27T03:21:08.256555+0000","last_peered":"2024-04-27T03:21:08.256555+0000","last_clean":"2024-04-27T03:21:08.256555+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:08.256555+0000","last_undegraded":"2024-04-27T03:21:08.256555+0000","last_fullsized":"2024-04-27T03:21:08.256555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.731000+0000","last_change":"2024-04-27T03:17:57.087004+0000","last_active":"2024-04-27T03:20:12.731000+0000","last_peered":"2024-04-27T03:20:12.731000+0000","last_clean":"2024-04-27T03:20:12.731000+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:20:12.731000+0000","last_undegraded":"2024-04-27T03:20:12.731000+0000","last_fullsized":"2024-04-27T03:20:12.731000+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":104,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.214842+0000","last_change":"2024-04-27T03:17:49.004339+0000","last_active":"2024-04-27T03:20:00.214842+0000","last_peered":"2024-04-27T03:20:00.214842+0000","last_clean":"2024-04-27T03:20:00.214842+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:20:00.214842+0000","last_undegraded":"2024-04-27T03:20:00.214842+0000","last_fullsized":"2024-04-27T03:20:00.214842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.158584+0000","last_change":"2024-04-27T03:17:47.000780+0000","last_active":"2024-04-27T03:20:00.158584+0000","last_peered":"2024-04-27T03:20:00.158584+0000","last_clean":"2024-04-27T03:20:00.158584+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:20:00.158584+0000","last_undegraded":"2024-04-27T03:20:00.158584+0000","last_fullsized":"2024-04-27T03:20:00.158584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741186+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:19:52.741186+0000","last_peered":"2024-04-27T03:19:52.741186+0000","last_clean":"2024-04-27T03:19:52.741186+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:19:52.741186+0000","last_undegraded":"2024-04-27T03:19:52.741186+0000","last_fullsized":"2024-04-27T03:19:52.741186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.600203+0000","last_change":"2024-04-27T03:21:19.600253+0000","last_active":"2024-04-27T03:21:19.600203+0000","last_peered":"2024-04-27T03:21:19.600203+0000","last_clean":"2024-04-27T03:21:19.600203+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:21:19.600203+0000","last_undegraded":"2024-04-27T03:21:19.600203+0000","last_fullsized":"2024-04-27T03:21:19.600203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.600004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:53:52.295521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'108","reported_seq":211,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.121243+0000","last_change":"2024-04-27T03:21:15.121323+0000","last_active":"2024-04-27T03:21:15.121243+0000","last_peered":"2024-04-27T03:21:15.121243+0000","last_clean":"2024-04-27T03:21:15.121243+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:15.121243+0000","last_undegraded":"2024-04-27T03:21:15.121243+0000","last_fullsized":"2024-04-27T03:21:15.121243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'108","last_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_deep_scrub":"31'108","last_deep_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.121034+0000","objects_scrubbed":9,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:20.807083+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4316,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":107,"num_read_kb":100,"num_write":44,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.738701+0000","last_change":"2024-04-27T03:17:55.081924+0000","last_active":"2024-04-27T03:20:02.738701+0000","last_peered":"2024-04-27T03:20:02.738701+0000","last_clean":"2024-04-27T03:20:02.738701+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:20:02.738701+0000","last_undegraded":"2024-04-27T03:20:02.738701+0000","last_fullsized":"2024-04-27T03:20:02.738701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.824584+0000","last_change":"2024-04-27T03:21:12.824647+0000","last_active":"2024-04-27T03:21:12.824584+0000","last_peered":"2024-04-27T03:21:12.824584+0000","last_clean":"2024-04-27T03:21:12.824584+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:21:12.824584+0000","last_undegraded":"2024-04-27T03:21:12.824584+0000","last_fullsized":"2024-04-27T03:21:12.824584+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.824385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:28:25.571276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.779914+0000","last_change":"2024-04-27T03:21:14.779972+0000","last_active":"2024-04-27T03:21:14.779914+0000","last_peered":"2024-04-27T03:21:14.779914+0000","last_clean":"2024-04-27T03:21:14.779914+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:21:14.779914+0000","last_undegraded":"2024-04-27T03:21:14.779914+0000","last_fullsized":"2024-04-27T03:21:14.779914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.779705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:11:09.535832+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.826787+0000","last_change":"2024-04-27T03:17:49.009366+0000","last_active":"2024-04-27T03:21:00.826787+0000","last_peered":"2024-04-27T03:21:00.826787+0000","last_clean":"2024-04-27T03:21:00.826787+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:00.826787+0000","last_undegraded":"2024-04-27T03:21:00.826787+0000","last_fullsized":"2024-04-27T03:21:00.826787+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.132425+0000","last_change":"2024-04-27T03:17:47.023352+0000","last_active":"2024-04-27T03:21:01.132425+0000","last_peered":"2024-04-27T03:21:01.132425+0000","last_clean":"2024-04-27T03:21:01.132425+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:01.132425+0000","last_undegraded":"2024-04-27T03:21:01.132425+0000","last_fullsized":"2024-04-27T03:21:01.132425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"31'2","reported_seq":234,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.286650+0000","last_change":"2024-04-27T03:21:26.286722+0000","last_active":"2024-04-27T03:21:26.286650+0000","last_peered":"2024-04-27T03:21:26.286650+0000","last_clean":"2024-04-27T03:21:26.286650+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:21:26.286650+0000","last_undegraded":"2024-04-27T03:21:26.286650+0000","last_fullsized":"2024-04-27T03:21:26.286650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.286462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:36:54.147283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":285,"num_read_kb":285,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'46","reported_seq":112,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.542394+0000","last_change":"2024-04-27T03:21:15.542505+0000","last_active":"2024-04-27T03:21:15.542394+0000","last_peered":"2024-04-27T03:21:15.542394+0000","last_clean":"2024-04-27T03:21:15.542394+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:21:15.542394+0000","last_undegraded":"2024-04-27T03:21:15.542394+0000","last_fullsized":"2024-04-27T03:21:15.542394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'46","last_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_deep_scrub":"31'46","last_deep_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.542196+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:31.074928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.742806+0000","last_change":"2024-04-27T03:17:55.074798+0000","last_active":"2024-04-27T03:20:17.742806+0000","last_peered":"2024-04-27T03:20:17.742806+0000","last_clean":"2024-04-27T03:20:17.742806+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:20:17.742806+0000","last_undegraded":"2024-04-27T03:20:17.742806+0000","last_fullsized":"2024-04-27T03:20:17.742806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.887628+0000","last_change":"2024-04-27T03:17:57.087180+0000","last_active":"2024-04-27T03:19:57.887628+0000","last_peered":"2024-04-27T03:19:57.887628+0000","last_clean":"2024-04-27T03:19:57.887628+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:19:57.887628+0000","last_undegraded":"2024-04-27T03:19:57.887628+0000","last_fullsized":"2024-04-27T03:19:57.887628+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.163593+0000","last_change":"2024-04-27T03:17:49.005188+0000","last_active":"2024-04-27T03:21:01.163593+0000","last_peered":"2024-04-27T03:21:01.163593+0000","last_clean":"2024-04-27T03:21:01.163593+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:01.163593+0000","last_undegraded":"2024-04-27T03:21:01.163593+0000","last_fullsized":"2024-04-27T03:21:01.163593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.736914+0000","last_change":"2024-04-27T03:17:47.022474+0000","last_active":"2024-04-27T03:20:12.736914+0000","last_peered":"2024-04-27T03:20:12.736914+0000","last_clean":"2024-04-27T03:20:12.736914+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:20:12.736914+0000","last_undegraded":"2024-04-27T03:20:12.736914+0000","last_fullsized":"2024-04-27T03:20:12.736914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.741747+0000","last_change":"2024-04-27T03:17:51.024948+0000","last_active":"2024-04-27T03:20:12.741747+0000","last_peered":"2024-04-27T03:20:12.741747+0000","last_clean":"2024-04-27T03:20:12.741747+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:20:12.741747+0000","last_undegraded":"2024-04-27T03:20:12.741747+0000","last_fullsized":"2024-04-27T03:20:12.741747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.185318+0000","last_change":"2024-04-27T03:21:19.185455+0000","last_active":"2024-04-27T03:21:19.185318+0000","last_peered":"2024-04-27T03:21:19.185318+0000","last_clean":"2024-04-27T03:21:19.185318+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:19.185318+0000","last_undegraded":"2024-04-27T03:21:19.185318+0000","last_fullsized":"2024-04-27T03:21:19.185318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.185100+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:12.343058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":250,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.823643+0000","last_change":"2024-04-27T03:21:24.823694+0000","last_active":"2024-04-27T03:21:24.823643+0000","last_peered":"2024-04-27T03:21:24.823643+0000","last_clean":"2024-04-27T03:21:24.823643+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:21:24.823643+0000","last_undegraded":"2024-04-27T03:21:24.823643+0000","last_fullsized":"2024-04-27T03:21:24.823643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.823449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:10:09.300634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.734550+0000","last_change":"2024-04-27T03:17:57.089220+0000","last_active":"2024-04-27T03:20:27.734550+0000","last_peered":"2024-04-27T03:20:27.734550+0000","last_clean":"2024-04-27T03:20:27.734550+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:20:27.734550+0000","last_undegraded":"2024-04-27T03:20:27.734550+0000","last_fullsized":"2024-04-27T03:20:27.734550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.964257+0000","last_change":"2024-04-27T03:18:34.799245+0000","last_active":"2024-04-27T03:21:00.964257+0000","last_peered":"2024-04-27T03:21:00.964257+0000","last_clean":"2024-04-27T03:21:00.964257+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:21:00.964257+0000","last_undegraded":"2024-04-27T03:21:00.964257+0000","last_fullsized":"2024-04-27T03:21:00.964257+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.115332+0000","last_change":"2024-04-27T03:17:47.012224+0000","last_active":"2024-04-27T03:21:01.115332+0000","last_peered":"2024-04-27T03:21:01.115332+0000","last_clean":"2024-04-27T03:21:01.115332+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:21:01.115332+0000","last_undegraded":"2024-04-27T03:21:01.115332+0000","last_fullsized":"2024-04-27T03:21:01.115332+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.096198+0000","last_change":"2024-04-27T03:21:12.096234+0000","last_active":"2024-04-27T03:21:12.096198+0000","last_peered":"2024-04-27T03:21:12.096198+0000","last_clean":"2024-04-27T03:21:12.096198+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:21:12.096198+0000","last_undegraded":"2024-04-27T03:21:12.096198+0000","last_fullsized":"2024-04-27T03:21:12.096198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.096079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:20.095840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'43","reported_seq":111,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.005411+0000","last_change":"2024-04-27T03:17:53.047021+0000","last_active":"2024-04-27T03:21:08.005411+0000","last_peered":"2024-04-27T03:21:08.005411+0000","last_clean":"2024-04-27T03:21:08.005411+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:08.005411+0000","last_undegraded":"2024-04-27T03:21:08.005411+0000","last_fullsized":"2024-04-27T03:21:08.005411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":67,"num_read_kb":70,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.199623+0000","last_change":"2024-04-27T03:21:20.199682+0000","last_active":"2024-04-27T03:21:20.199623+0000","last_peered":"2024-04-27T03:21:20.199623+0000","last_clean":"2024-04-27T03:21:20.199623+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:21:20.199623+0000","last_undegraded":"2024-04-27T03:21:20.199623+0000","last_fullsized":"2024-04-27T03:21:20.199623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.199413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:10:57.834556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.745960+0000","last_change":"2024-04-27T03:17:57.086967+0000","last_active":"2024-04-27T03:20:27.745960+0000","last_peered":"2024-04-27T03:20:27.745960+0000","last_clean":"2024-04-27T03:20:27.745960+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:20:27.745960+0000","last_undegraded":"2024-04-27T03:20:27.745960+0000","last_fullsized":"2024-04-27T03:20:27.745960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.247672+0000","last_change":"2024-04-27T03:17:49.014835+0000","last_active":"2024-04-27T03:21:01.247672+0000","last_peered":"2024-04-27T03:21:01.247672+0000","last_clean":"2024-04-27T03:21:01.247672+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:01.247672+0000","last_undegraded":"2024-04-27T03:21:01.247672+0000","last_fullsized":"2024-04-27T03:21:01.247672+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.731154+0000","last_change":"2024-04-27T03:17:47.026622+0000","last_active":"2024-04-27T03:20:12.731154+0000","last_peered":"2024-04-27T03:20:12.731154+0000","last_clean":"2024-04-27T03:20:12.731154+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:20:12.731154+0000","last_undegraded":"2024-04-27T03:20:12.731154+0000","last_fullsized":"2024-04-27T03:20:12.731154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.741502+0000","last_change":"2024-04-27T03:17:51.014069+0000","last_active":"2024-04-27T03:20:12.741502+0000","last_peered":"2024-04-27T03:20:12.741502+0000","last_clean":"2024-04-27T03:20:12.741502+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:20:12.741502+0000","last_undegraded":"2024-04-27T03:20:12.741502+0000","last_fullsized":"2024-04-27T03:20:12.741502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'37","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.159109+0000","last_change":"2024-04-27T03:21:17.159218+0000","last_active":"2024-04-27T03:21:17.159109+0000","last_peered":"2024-04-27T03:21:17.159109+0000","last_clean":"2024-04-27T03:21:17.159109+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:21:17.159109+0000","last_undegraded":"2024-04-27T03:21:17.159109+0000","last_fullsized":"2024-04-27T03:21:17.159109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.158898+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:06.103710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.871801+0000","last_change":"2024-04-27T03:21:23.871852+0000","last_active":"2024-04-27T03:21:23.871801+0000","last_peered":"2024-04-27T03:21:23.871801+0000","last_clean":"2024-04-27T03:21:23.871801+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:21:23.871801+0000","last_undegraded":"2024-04-27T03:21:23.871801+0000","last_fullsized":"2024-04-27T03:21:23.871801+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.871608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:14.918299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'1","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397141+0000","last_change":"2024-04-27T03:17:49.022141+0000","last_active":"2024-04-27T03:21:01.397141+0000","last_peered":"2024-04-27T03:21:01.397141+0000","last_clean":"2024-04-27T03:21:01.397141+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:21:01.397141+0000","last_undegraded":"2024-04-27T03:21:01.397141+0000","last_fullsized":"2024-04-27T03:21:01.397141+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.124071+0000","last_change":"2024-04-27T03:17:47.032082+0000","last_active":"2024-04-27T03:21:01.124071+0000","last_peered":"2024-04-27T03:21:01.124071+0000","last_clean":"2024-04-27T03:21:01.124071+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:21:01.124071+0000","last_undegraded":"2024-04-27T03:21:01.124071+0000","last_fullsized":"2024-04-27T03:21:01.124071+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'4","reported_seq":233,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.855799+0000","last_change":"2024-04-27T03:17:51.024084+0000","last_active":"2024-04-27T03:21:07.855799+0000","last_peered":"2024-04-27T03:21:07.855799+0000","last_clean":"2024-04-27T03:21:07.855799+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:21:07.855799+0000","last_undegraded":"2024-04-27T03:21:07.855799+0000","last_fullsized":"2024-04-27T03:21:07.855799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":301,"num_read_kb":301,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'72","reported_seq":155,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.270317+0000","last_change":"2024-04-27T03:21:23.270391+0000","last_active":"2024-04-27T03:21:23.270317+0000","last_peered":"2024-04-27T03:21:23.270317+0000","last_clean":"2024-04-27T03:21:23.270317+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:21:23.270317+0000","last_undegraded":"2024-04-27T03:21:23.270317+0000","last_fullsized":"2024-04-27T03:21:23.270317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'72","last_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_deep_scrub":"31'72","last_deep_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.270178+0000","objects_scrubbed":6,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:28.956226+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":28,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.743462+0000","last_change":"2024-04-27T03:21:16.743518+0000","last_active":"2024-04-27T03:21:16.743462+0000","last_peered":"2024-04-27T03:21:16.743462+0000","last_clean":"2024-04-27T03:21:16.743462+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:21:16.743462+0000","last_undegraded":"2024-04-27T03:21:16.743462+0000","last_fullsized":"2024-04-27T03:21:16.743462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.743267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:34:18.284825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.742988+0000","last_change":"2024-04-27T03:17:57.070510+0000","last_active":"2024-04-27T03:20:17.742988+0000","last_peered":"2024-04-27T03:20:17.742988+0000","last_clean":"2024-04-27T03:20:17.742988+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:20:17.742988+0000","last_undegraded":"2024-04-27T03:20:17.742988+0000","last_fullsized":"2024-04-27T03:20:17.742988+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:56.111767+0000","last_change":"2024-04-27T03:17:49.000295+0000","last_active":"2024-04-27T03:19:56.111767+0000","last_peered":"2024-04-27T03:19:56.111767+0000","last_clean":"2024-04-27T03:19:56.111767+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:19:56.111767+0000","last_undegraded":"2024-04-27T03:19:56.111767+0000","last_fullsized":"2024-04-27T03:19:56.111767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.730774+0000","last_change":"2024-04-27T03:17:47.021864+0000","last_active":"2024-04-27T03:20:12.730774+0000","last_peered":"2024-04-27T03:20:12.730774+0000","last_clean":"2024-04-27T03:20:12.730774+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:20:12.730774+0000","last_undegraded":"2024-04-27T03:20:12.730774+0000","last_fullsized":"2024-04-27T03:20:12.730774+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.193746+0000","last_change":"2024-04-27T03:21:20.193806+0000","last_active":"2024-04-27T03:21:20.193746+0000","last_peered":"2024-04-27T03:21:20.193746+0000","last_clean":"2024-04-27T03:21:20.193746+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:21:20.193746+0000","last_undegraded":"2024-04-27T03:21:20.193746+0000","last_fullsized":"2024-04-27T03:21:20.193746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.193546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:27:42.782149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'45","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.064594+0000","last_change":"2024-04-27T03:21:13.064640+0000","last_active":"2024-04-27T03:21:13.064594+0000","last_peered":"2024-04-27T03:21:13.064594+0000","last_clean":"2024-04-27T03:21:13.064594+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:21:13.064594+0000","last_undegraded":"2024-04-27T03:21:13.064594+0000","last_fullsized":"2024-04-27T03:21:13.064594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.064511+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:06:47.923690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.302195+0000","last_change":"2024-04-27T03:21:14.302256+0000","last_active":"2024-04-27T03:21:14.302195+0000","last_peered":"2024-04-27T03:21:14.302195+0000","last_clean":"2024-04-27T03:21:14.302195+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:21:14.302195+0000","last_undegraded":"2024-04-27T03:21:14.302195+0000","last_fullsized":"2024-04-27T03:21:14.302195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.302007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:00.292019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747121+0000","last_change":"2024-04-27T03:17:57.070067+0000","last_active":"2024-04-27T03:20:17.747121+0000","last_peered":"2024-04-27T03:20:17.747121+0000","last_clean":"2024-04-27T03:20:17.747121+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:20:17.747121+0000","last_undegraded":"2024-04-27T03:20:17.747121+0000","last_fullsized":"2024-04-27T03:20:17.747121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.744728+0000","last_change":"2024-04-27T03:17:49.004464+0000","last_active":"2024-04-27T03:20:22.744728+0000","last_peered":"2024-04-27T03:20:22.744728+0000","last_clean":"2024-04-27T03:20:22.744728+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:20:22.744728+0000","last_undegraded":"2024-04-27T03:20:22.744728+0000","last_fullsized":"2024-04-27T03:20:22.744728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747160+0000","last_change":"2024-04-27T03:17:47.027156+0000","last_active":"2024-04-27T03:20:17.747160+0000","last_peered":"2024-04-27T03:20:17.747160+0000","last_clean":"2024-04-27T03:20:17.747160+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:20:17.747160+0000","last_undegraded":"2024-04-27T03:20:17.747160+0000","last_fullsized":"2024-04-27T03:20:17.747160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.231515+0000","last_change":"2024-04-27T03:21:22.231634+0000","last_active":"2024-04-27T03:21:22.231515+0000","last_peered":"2024-04-27T03:21:22.231515+0000","last_clean":"2024-04-27T03:21:22.231515+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:21:22.231515+0000","last_undegraded":"2024-04-27T03:21:22.231515+0000","last_fullsized":"2024-04-27T03:21:22.231515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.231282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:38.450738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'47","reported_seq":113,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.572445+0000","last_change":"2024-04-27T03:21:16.572535+0000","last_active":"2024-04-27T03:21:16.572445+0000","last_peered":"2024-04-27T03:21:16.572445+0000","last_clean":"2024-04-27T03:21:16.572445+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:21:16.572445+0000","last_undegraded":"2024-04-27T03:21:16.572445+0000","last_fullsized":"2024-04-27T03:21:16.572445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.572262+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:05.517074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.726973+0000","last_change":"2024-04-27T03:17:55.077381+0000","last_active":"2024-04-27T03:19:57.726973+0000","last_peered":"2024-04-27T03:19:57.726973+0000","last_clean":"2024-04-27T03:19:57.726973+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:19:57.726973+0000","last_undegraded":"2024-04-27T03:19:57.726973+0000","last_fullsized":"2024-04-27T03:19:57.726973+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747057+0000","last_change":"2024-04-27T03:17:57.066218+0000","last_active":"2024-04-27T03:20:17.747057+0000","last_peered":"2024-04-27T03:20:17.747057+0000","last_clean":"2024-04-27T03:20:17.747057+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:20:17.747057+0000","last_undegraded":"2024-04-27T03:20:17.747057+0000","last_fullsized":"2024-04-27T03:20:17.747057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'3","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397875+0000","last_change":"2024-04-27T03:17:49.019967+0000","last_active":"2024-04-27T03:21:01.397875+0000","last_peered":"2024-04-27T03:21:01.397875+0000","last_clean":"2024-04-27T03:21:01.397875+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:21:01.397875+0000","last_undegraded":"2024-04-27T03:21:01.397875+0000","last_fullsized":"2024-04-27T03:21:01.397875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.743118+0000","last_change":"2024-04-27T03:17:47.018817+0000","last_active":"2024-04-27T03:20:17.743118+0000","last_peered":"2024-04-27T03:20:17.743118+0000","last_clean":"2024-04-27T03:20:17.743118+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:20:17.743118+0000","last_undegraded":"2024-04-27T03:20:17.743118+0000","last_fullsized":"2024-04-27T03:20:17.743118+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.301687+0000","last_change":"2024-04-27T03:21:27.301740+0000","last_active":"2024-04-27T03:21:27.301687+0000","last_peered":"2024-04-27T03:21:27.301687+0000","last_clean":"2024-04-27T03:21:27.301687+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:21:27.301687+0000","last_undegraded":"2024-04-27T03:21:27.301687+0000","last_fullsized":"2024-04-27T03:21:27.301687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.301481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:33.928268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"27'16","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.736789+0000","last_change":"2024-04-27T03:17:53.046967+0000","last_active":"2024-04-27T03:19:52.736789+0000","last_peered":"2024-04-27T03:19:52.736789+0000","last_clean":"2024-04-27T03:19:52.736789+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:19:52.736789+0000","last_undegraded":"2024-04-27T03:19:52.736789+0000","last_fullsized":"2024-04-27T03:19:52.736789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'285","reported_seq":392,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.185100+0000","last_change":"2024-04-27T03:21:25.185168+0000","last_active":"2024-04-27T03:21:25.185100+0000","last_peered":"2024-04-27T03:21:25.185100+0000","last_clean":"2024-04-27T03:21:25.185100+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:25.185100+0000","last_undegraded":"2024-04-27T03:21:25.185100+0000","last_fullsized":"2024-04-27T03:21:25.185100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.184906+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:35.796002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.746971+0000","last_change":"2024-04-27T03:17:57.084816+0000","last_active":"2024-04-27T03:20:17.746971+0000","last_peered":"2024-04-27T03:20:17.746971+0000","last_clean":"2024-04-27T03:20:17.746971+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:20:17.746971+0000","last_undegraded":"2024-04-27T03:20:17.746971+0000","last_fullsized":"2024-04-27T03:20:17.746971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.249009+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.249009+0000","last_peered":"2024-04-27T03:21:01.249009+0000","last_clean":"2024-04-27T03:21:01.249009+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:01.249009+0000","last_undegraded":"2024-04-27T03:21:01.249009+0000","last_fullsized":"2024-04-27T03:21:01.249009+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.732325+0000","last_change":"2024-04-27T03:17:47.027621+0000","last_active":"2024-04-27T03:20:17.732325+0000","last_peered":"2024-04-27T03:20:17.732325+0000","last_clean":"2024-04-27T03:20:17.732325+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:20:17.732325+0000","last_undegraded":"2024-04-27T03:20:17.732325+0000","last_fullsized":"2024-04-27T03:20:17.732325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.332423+0000","last_change":"2024-04-27T03:21:13.332476+0000","last_active":"2024-04-27T03:21:13.332423+0000","last_peered":"2024-04-27T03:21:13.332423+0000","last_clean":"2024-04-27T03:21:13.332423+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:21:13.332423+0000","last_undegraded":"2024-04-27T03:21:13.332423+0000","last_fullsized":"2024-04-27T03:21:13.332423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.332213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:31.014208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'63","reported_seq":141,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.164731+0000","last_change":"2024-04-27T03:17:53.039949+0000","last_active":"2024-04-27T03:21:08.164731+0000","last_peered":"2024-04-27T03:21:08.164731+0000","last_clean":"2024-04-27T03:21:08.164731+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:21:08.164731+0000","last_undegraded":"2024-04-27T03:21:08.164731+0000","last_fullsized":"2024-04-27T03:21:08.164731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":44,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.654794+0000","last_change":"2024-04-27T03:21:26.654845+0000","last_active":"2024-04-27T03:21:26.654794+0000","last_peered":"2024-04-27T03:21:26.654794+0000","last_clean":"2024-04-27T03:21:26.654794+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:21:26.654794+0000","last_undegraded":"2024-04-27T03:21:26.654794+0000","last_fullsized":"2024-04-27T03:21:26.654794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.654609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:01.305349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.819801+0000","last_change":"2024-04-27T03:17:57.066613+0000","last_active":"2024-04-27T03:21:00.819801+0000","last_peered":"2024-04-27T03:21:00.819801+0000","last_clean":"2024-04-27T03:21:00.819801+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:21:00.819801+0000","last_undegraded":"2024-04-27T03:21:00.819801+0000","last_fullsized":"2024-04-27T03:21:00.819801+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.963438+0000","last_change":"2024-04-27T03:17:49.017718+0000","last_active":"2024-04-27T03:21:00.963438+0000","last_peered":"2024-04-27T03:21:00.963438+0000","last_clean":"2024-04-27T03:21:00.963438+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:00.963438+0000","last_undegraded":"2024-04-27T03:21:00.963438+0000","last_fullsized":"2024-04-27T03:21:00.963438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747314+0000","last_change":"2024-04-27T03:17:47.022941+0000","last_active":"2024-04-27T03:20:17.747314+0000","last_peered":"2024-04-27T03:20:17.747314+0000","last_clean":"2024-04-27T03:20:17.747314+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:20:17.747314+0000","last_undegraded":"2024-04-27T03:20:17.747314+0000","last_fullsized":"2024-04-27T03:20:17.747314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.156966+0000","last_change":"2024-04-27T03:21:19.157032+0000","last_active":"2024-04-27T03:21:19.156966+0000","last_peered":"2024-04-27T03:21:19.156966+0000","last_clean":"2024-04-27T03:21:19.156966+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:21:19.156966+0000","last_undegraded":"2024-04-27T03:21:19.156966+0000","last_fullsized":"2024-04-27T03:21:19.156966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.156758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:49.521232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'41","reported_seq":98,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.173537+0000","last_change":"2024-04-27T03:17:53.040115+0000","last_active":"2024-04-27T03:21:08.173537+0000","last_peered":"2024-04-27T03:21:08.173537+0000","last_clean":"2024-04-27T03:21:08.173537+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:21:08.173537+0000","last_undegraded":"2024-04-27T03:21:08.173537+0000","last_fullsized":"2024-04-27T03:21:08.173537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.175376+0000","last_change":"2024-04-27T03:21:24.175430+0000","last_active":"2024-04-27T03:21:24.175376+0000","last_peered":"2024-04-27T03:21:24.175376+0000","last_clean":"2024-04-27T03:21:24.175376+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:21:24.175376+0000","last_undegraded":"2024-04-27T03:21:24.175376+0000","last_fullsized":"2024-04-27T03:21:24.175376+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.175164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:23.877311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.480429+0000","last_change":"2024-04-27T03:17:57.087706+0000","last_active":"2024-04-27T03:21:01.480429+0000","last_peered":"2024-04-27T03:21:01.480429+0000","last_clean":"2024-04-27T03:21:01.480429+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:21:01.480429+0000","last_undegraded":"2024-04-27T03:21:01.480429+0000","last_fullsized":"2024-04-27T03:21:01.480429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.962890+0000","last_change":"2024-04-27T03:17:49.011902+0000","last_active":"2024-04-27T03:21:00.962890+0000","last_peered":"2024-04-27T03:21:00.962890+0000","last_clean":"2024-04-27T03:21:00.962890+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:21:00.962890+0000","last_undegraded":"2024-04-27T03:21:00.962890+0000","last_fullsized":"2024-04-27T03:21:00.962890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.987295+0000","last_change":"2024-04-27T03:17:47.016344+0000","last_active":"2024-04-27T03:21:07.987295+0000","last_peered":"2024-04-27T03:21:07.987295+0000","last_clean":"2024-04-27T03:21:07.987295+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:21:07.987295+0000","last_undegraded":"2024-04-27T03:21:07.987295+0000","last_fullsized":"2024-04-27T03:21:07.987295+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.732183+0000","last_change":"2024-04-27T03:17:51.029804+0000","last_active":"2024-04-27T03:20:17.732183+0000","last_peered":"2024-04-27T03:20:17.732183+0000","last_clean":"2024-04-27T03:20:17.732183+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:20:17.732183+0000","last_undegraded":"2024-04-27T03:20:17.732183+0000","last_fullsized":"2024-04-27T03:20:17.732183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'37","reported_seq":100,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.538736+0000","last_change":"2024-04-27T03:21:14.538808+0000","last_active":"2024-04-27T03:21:14.538736+0000","last_peered":"2024-04-27T03:21:14.538736+0000","last_clean":"2024-04-27T03:21:14.538736+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:21:14.538736+0000","last_undegraded":"2024-04-27T03:21:14.538736+0000","last_fullsized":"2024-04-27T03:21:14.538736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.538542+0000","objects_scrubbed":3,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:32.436126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.749488+0000","last_change":"2024-04-27T03:17:55.071034+0000","last_active":"2024-04-27T03:20:27.749488+0000","last_peered":"2024-04-27T03:20:27.749488+0000","last_clean":"2024-04-27T03:20:27.749488+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:20:27.749488+0000","last_undegraded":"2024-04-27T03:20:27.749488+0000","last_fullsized":"2024-04-27T03:20:27.749488+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:54.018430+0000","last_clean_scrub_stamp":"2024-04-27T03:17:54.018430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.744683+0000","last_change":"2024-04-27T03:17:57.086785+0000","last_active":"2024-04-27T03:20:22.744683+0000","last_peered":"2024-04-27T03:20:22.744683+0000","last_clean":"2024-04-27T03:20:22.744683+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:20:22.744683+0000","last_undegraded":"2024-04-27T03:20:22.744683+0000","last_fullsized":"2024-04-27T03:20:22.744683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'14","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.398373+0000","last_change":"2024-04-27T03:17:49.013870+0000","last_active":"2024-04-27T03:21:01.398373+0000","last_peered":"2024-04-27T03:21:01.398373+0000","last_clean":"2024-04-27T03:21:01.398373+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:01.398373+0000","last_undegraded":"2024-04-27T03:21:01.398373+0000","last_fullsized":"2024-04-27T03:21:01.398373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.026513+0000","last_change":"2024-04-27T03:17:47.002631+0000","last_active":"2024-04-27T03:20:00.026513+0000","last_peered":"2024-04-27T03:20:00.026513+0000","last_clean":"2024-04-27T03:20:00.026513+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:20:00.026513+0000","last_undegraded":"2024-04-27T03:20:00.026513+0000","last_fullsized":"2024-04-27T03:20:00.026513+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'2","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:02.747502+0000","last_change":"2024-04-27T03:18:34.794992+0000","last_active":"2024-04-27T03:21:02.747502+0000","last_peered":"2024-04-27T03:21:02.747502+0000","last_clean":"2024-04-27T03:21:02.747502+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:21:02.747502+0000","last_undegraded":"2024-04-27T03:21:02.747502+0000","last_fullsized":"2024-04-27T03:21:02.747502+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'47","reported_seq":112,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.121525+0000","last_change":"2024-04-27T03:17:53.039095+0000","last_active":"2024-04-27T03:21:08.121525+0000","last_peered":"2024-04-27T03:21:08.121525+0000","last_clean":"2024-04-27T03:21:08.121525+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:08.121525+0000","last_undegraded":"2024-04-27T03:21:08.121525+0000","last_fullsized":"2024-04-27T03:21:08.121525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":52,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.822080+0000","last_change":"2024-04-27T03:21:19.822219+0000","last_active":"2024-04-27T03:21:19.822080+0000","last_peered":"2024-04-27T03:21:19.822080+0000","last_clean":"2024-04-27T03:21:19.822080+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:21:19.822080+0000","last_undegraded":"2024-04-27T03:21:19.822080+0000","last_fullsized":"2024-04-27T03:21:19.822080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.821853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:18.101674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747256+0000","last_change":"2024-04-27T03:17:57.061982+0000","last_active":"2024-04-27T03:20:17.747256+0000","last_peered":"2024-04-27T03:20:17.747256+0000","last_clean":"2024-04-27T03:20:17.747256+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:20:17.747256+0000","last_undegraded":"2024-04-27T03:20:17.747256+0000","last_fullsized":"2024-04-27T03:20:17.747256+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.728438+0000","last_change":"2024-04-27T03:17:49.001808+0000","last_active":"2024-04-27T03:20:02.728438+0000","last_peered":"2024-04-27T03:20:02.728438+0000","last_clean":"2024-04-27T03:20:02.728438+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:20:02.728438+0000","last_undegraded":"2024-04-27T03:20:02.728438+0000","last_fullsized":"2024-04-27T03:20:02.728438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.732746+0000","last_change":"2024-04-27T03:17:47.032632+0000","last_active":"2024-04-27T03:19:52.732746+0000","last_peered":"2024-04-27T03:19:52.732746+0000","last_clean":"2024-04-27T03:19:52.732746+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:19:52.732746+0000","last_undegraded":"2024-04-27T03:19:52.732746+0000","last_fullsized":"2024-04-27T03:19:52.732746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.728325+0000","last_change":"2024-04-27T03:17:49.011651+0000","last_active":"2024-04-27T03:20:02.728325+0000","last_peered":"2024-04-27T03:20:02.728325+0000","last_clean":"2024-04-27T03:20:02.728325+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:20:02.728325+0000","last_undegraded":"2024-04-27T03:20:02.728325+0000","last_fullsized":"2024-04-27T03:20:02.728325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'2","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.988329+0000","last_change":"2024-04-27T03:18:34.773740+0000","last_active":"2024-04-27T03:21:07.988329+0000","last_peered":"2024-04-27T03:21:07.988329+0000","last_clean":"2024-04-27T03:21:07.988329+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:21:07.988329+0000","last_undegraded":"2024-04-27T03:21:07.988329+0000","last_fullsized":"2024-04-27T03:21:07.988329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:47.744454+0000","last_change":"2024-04-27T03:18:34.784905+0000","last_active":"2024-04-27T03:20:47.744454+0000","last_peered":"2024-04-27T03:20:47.744454+0000","last_clean":"2024-04-27T03:20:47.744454+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:20:47.744454+0000","last_undegraded":"2024-04-27T03:20:47.744454+0000","last_fullsized":"2024-04-27T03:20:47.744454+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.736562+0000","last_change":"2024-04-27T03:17:47.036148+0000","last_active":"2024-04-27T03:19:52.736562+0000","last_peered":"2024-04-27T03:19:52.736562+0000","last_clean":"2024-04-27T03:19:52.736562+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:19:52.736562+0000","last_undegraded":"2024-04-27T03:19:52.736562+0000","last_fullsized":"2024-04-27T03:19:52.736562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.728162+0000","last_change":"2024-04-27T03:17:49.017582+0000","last_active":"2024-04-27T03:20:02.728162+0000","last_peered":"2024-04-27T03:20:02.728162+0000","last_clean":"2024-04-27T03:20:02.728162+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:20:02.728162+0000","last_undegraded":"2024-04-27T03:20:02.728162+0000","last_fullsized":"2024-04-27T03:20:02.728162+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.732873+0000","last_change":"2024-04-27T03:17:47.016123+0000","last_active":"2024-04-27T03:19:52.732873+0000","last_peered":"2024-04-27T03:19:52.732873+0000","last_clean":"2024-04-27T03:19:52.732873+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:19:52.732873+0000","last_undegraded":"2024-04-27T03:19:52.732873+0000","last_fullsized":"2024-04-27T03:19:52.732873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.315993+0000","last_change":"2024-04-27T03:17:49.021184+0000","last_active":"2024-04-27T03:21:01.315993+0000","last_peered":"2024-04-27T03:21:01.315993+0000","last_clean":"2024-04-27T03:21:01.315993+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:01.315993+0000","last_undegraded":"2024-04-27T03:21:01.315993+0000","last_fullsized":"2024-04-27T03:21:01.315993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737696+0000","last_change":"2024-04-27T03:17:47.039423+0000","last_active":"2024-04-27T03:19:57.737696+0000","last_peered":"2024-04-27T03:19:57.737696+0000","last_clean":"2024-04-27T03:19:57.737696+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:19:57.737696+0000","last_undegraded":"2024-04-27T03:19:57.737696+0000","last_fullsized":"2024-04-27T03:19:57.737696+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.726869+0000","last_change":"2024-04-27T03:17:49.006801+0000","last_active":"2024-04-27T03:19:57.726869+0000","last_peered":"2024-04-27T03:19:57.726869+0000","last_clean":"2024-04-27T03:19:57.726869+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:19:57.726869+0000","last_undegraded":"2024-04-27T03:19:57.726869+0000","last_fullsized":"2024-04-27T03:19:57.726869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:59.951123+0000","last_change":"2024-04-27T03:17:47.030288+0000","last_active":"2024-04-27T03:19:59.951123+0000","last_peered":"2024-04-27T03:19:59.951123+0000","last_clean":"2024-04-27T03:19:59.951123+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:19:59.951123+0000","last_undegraded":"2024-04-27T03:19:59.951123+0000","last_fullsized":"2024-04-27T03:19:59.951123+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.317637+0000","last_change":"2024-04-27T03:17:49.015796+0000","last_active":"2024-04-27T03:21:01.317637+0000","last_peered":"2024-04-27T03:21:01.317637+0000","last_clean":"2024-04-27T03:21:01.317637+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:01.317637+0000","last_undegraded":"2024-04-27T03:21:01.317637+0000","last_fullsized":"2024-04-27T03:21:01.317637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.987179+0000","last_change":"2024-04-27T03:17:47.031857+0000","last_active":"2024-04-27T03:21:07.987179+0000","last_peered":"2024-04-27T03:21:07.987179+0000","last_clean":"2024-04-27T03:21:07.987179+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:07.987179+0000","last_undegraded":"2024-04-27T03:21:07.987179+0000","last_fullsized":"2024-04-27T03:21:07.987179+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.963682+0000","last_change":"2024-04-27T03:17:49.017220+0000","last_active":"2024-04-27T03:21:00.963682+0000","last_peered":"2024-04-27T03:21:00.963682+0000","last_clean":"2024-04-27T03:21:00.963682+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:00.963682+0000","last_undegraded":"2024-04-27T03:21:00.963682+0000","last_fullsized":"2024-04-27T03:21:00.963682+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737807+0000","last_change":"2024-04-27T03:17:47.039005+0000","last_active":"2024-04-27T03:19:57.737807+0000","last_peered":"2024-04-27T03:19:57.737807+0000","last_clean":"2024-04-27T03:19:57.737807+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:19:57.737807+0000","last_undegraded":"2024-04-27T03:19:57.737807+0000","last_fullsized":"2024-04-27T03:19:57.737807+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.316442+0000","last_change":"2024-04-27T03:17:49.011708+0000","last_active":"2024-04-27T03:21:01.316442+0000","last_peered":"2024-04-27T03:21:01.316442+0000","last_clean":"2024-04-27T03:21:01.316442+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:01.316442+0000","last_undegraded":"2024-04-27T03:21:01.316442+0000","last_fullsized":"2024-04-27T03:21:01.316442+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737509+0000","last_change":"2024-04-27T03:17:47.016024+0000","last_active":"2024-04-27T03:19:57.737509+0000","last_peered":"2024-04-27T03:19:57.737509+0000","last_clean":"2024-04-27T03:19:57.737509+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:19:57.737509+0000","last_undegraded":"2024-04-27T03:19:57.737509+0000","last_fullsized":"2024-04-27T03:19:57.737509+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.316963+0000","last_change":"2024-04-27T03:17:49.010865+0000","last_active":"2024-04-27T03:21:01.316963+0000","last_peered":"2024-04-27T03:21:01.316963+0000","last_clean":"2024-04-27T03:21:01.316963+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:21:01.316963+0000","last_undegraded":"2024-04-27T03:21:01.316963+0000","last_fullsized":"2024-04-27T03:21:01.316963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.738611+0000","last_change":"2024-04-27T03:17:47.019986+0000","last_active":"2024-04-27T03:20:02.738611+0000","last_peered":"2024-04-27T03:20:02.738611+0000","last_clean":"2024-04-27T03:20:02.738611+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:20:02.738611+0000","last_undegraded":"2024-04-27T03:20:02.738611+0000","last_fullsized":"2024-04-27T03:20:02.738611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.965185+0000","last_change":"2024-04-27T03:17:49.005667+0000","last_active":"2024-04-27T03:21:00.965185+0000","last_peered":"2024-04-27T03:21:00.965185+0000","last_clean":"2024-04-27T03:21:00.965185+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:00.965185+0000","last_undegraded":"2024-04-27T03:21:00.965185+0000","last_fullsized":"2024-04-27T03:21:00.965185+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743433+0000","last_change":"2024-04-27T03:17:47.023505+0000","last_active":"2024-04-27T03:20:02.743433+0000","last_peered":"2024-04-27T03:20:02.743433+0000","last_clean":"2024-04-27T03:20:02.743433+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:20:02.743433+0000","last_undegraded":"2024-04-27T03:20:02.743433+0000","last_fullsized":"2024-04-27T03:20:02.743433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729544+0000","last_change":"2024-04-27T03:17:49.009462+0000","last_active":"2024-04-27T03:20:07.729544+0000","last_peered":"2024-04-27T03:20:07.729544+0000","last_clean":"2024-04-27T03:20:07.729544+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:20:07.729544+0000","last_undegraded":"2024-04-27T03:20:07.729544+0000","last_fullsized":"2024-04-27T03:20:07.729544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.728069+0000","last_change":"2024-04-27T03:17:47.015754+0000","last_active":"2024-04-27T03:20:02.728069+0000","last_peered":"2024-04-27T03:20:02.728069+0000","last_clean":"2024-04-27T03:20:02.728069+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:20:02.728069+0000","last_undegraded":"2024-04-27T03:20:02.728069+0000","last_fullsized":"2024-04-27T03:20:02.728069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'7","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.398534+0000","last_change":"2024-04-27T03:17:49.017194+0000","last_active":"2024-04-27T03:21:01.398534+0000","last_peered":"2024-04-27T03:21:01.398534+0000","last_clean":"2024-04-27T03:21:01.398534+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:21:01.398534+0000","last_undegraded":"2024-04-27T03:21:01.398534+0000","last_fullsized":"2024-04-27T03:21:01.398534+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.738842+0000","last_change":"2024-04-27T03:17:47.038806+0000","last_active":"2024-04-27T03:20:02.738842+0000","last_peered":"2024-04-27T03:20:02.738842+0000","last_clean":"2024-04-27T03:20:02.738842+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:20:02.738842+0000","last_undegraded":"2024-04-27T03:20:02.738842+0000","last_fullsized":"2024-04-27T03:20:02.738842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.825968+0000","last_change":"2024-04-27T03:17:49.006535+0000","last_active":"2024-04-27T03:21:00.825968+0000","last_peered":"2024-04-27T03:21:00.825968+0000","last_clean":"2024-04-27T03:21:00.825968+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:00.825968+0000","last_undegraded":"2024-04-27T03:21:00.825968+0000","last_fullsized":"2024-04-27T03:21:00.825968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:37.737755+0000","last_change":"2024-04-27T03:19:35.441020+0000","last_active":"2024-04-27T03:19:37.737755+0000","last_peered":"2024-04-27T03:19:37.737755+0000","last_clean":"2024-04-27T03:19:37.737755+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:19:37.737755+0000","last_undegraded":"2024-04-27T03:19:37.737755+0000","last_fullsized":"2024-04-27T03:19:37.737755+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.739995+0000","last_change":"2024-04-27T03:17:49.016545+0000","last_active":"2024-04-27T03:20:07.739995+0000","last_peered":"2024-04-27T03:20:07.739995+0000","last_clean":"2024-04-27T03:20:07.739995+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:20:07.739995+0000","last_undegraded":"2024-04-27T03:20:07.739995+0000","last_fullsized":"2024-04-27T03:20:07.739995+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.735783+0000","last_change":"2024-04-27T03:17:47.022441+0000","last_active":"2024-04-27T03:20:07.735783+0000","last_peered":"2024-04-27T03:20:07.735783+0000","last_clean":"2024-04-27T03:20:07.735783+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:20:07.735783+0000","last_undegraded":"2024-04-27T03:20:07.735783+0000","last_fullsized":"2024-04-27T03:20:07.735783+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.163942+0000","last_change":"2024-04-27T03:17:49.018017+0000","last_active":"2024-04-27T03:21:01.163942+0000","last_peered":"2024-04-27T03:21:01.163942+0000","last_clean":"2024-04-27T03:21:01.163942+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:01.163942+0000","last_undegraded":"2024-04-27T03:21:01.163942+0000","last_fullsized":"2024-04-27T03:21:01.163942+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740491+0000","last_change":"2024-04-27T03:17:47.026063+0000","last_active":"2024-04-27T03:20:07.740491+0000","last_peered":"2024-04-27T03:20:07.740491+0000","last_clean":"2024-04-27T03:20:07.740491+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:20:07.740491+0000","last_undegraded":"2024-04-27T03:20:07.740491+0000","last_fullsized":"2024-04-27T03:20:07.740491+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744497+0000","last_change":"2024-04-27T03:17:47.002843+0000","last_active":"2024-04-27T03:20:07.744497+0000","last_peered":"2024-04-27T03:20:07.744497+0000","last_clean":"2024-04-27T03:20:07.744497+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:20:07.744497+0000","last_undegraded":"2024-04-27T03:20:07.744497+0000","last_fullsized":"2024-04-27T03:20:07.744497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'11","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.397743+0000","last_change":"2024-04-27T03:17:49.014133+0000","last_active":"2024-04-27T03:21:01.397743+0000","last_peered":"2024-04-27T03:21:01.397743+0000","last_clean":"2024-04-27T03:21:01.397743+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:01.397743+0000","last_undegraded":"2024-04-27T03:21:01.397743+0000","last_fullsized":"2024-04-27T03:21:01.397743+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729385+0000","last_change":"2024-04-27T03:17:47.022513+0000","last_active":"2024-04-27T03:20:07.729385+0000","last_peered":"2024-04-27T03:20:07.729385+0000","last_clean":"2024-04-27T03:20:07.729385+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:20:07.729385+0000","last_undegraded":"2024-04-27T03:20:07.729385+0000","last_fullsized":"2024-04-27T03:20:07.729385+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.732431+0000","last_change":"2024-04-27T03:17:49.012955+0000","last_active":"2024-04-27T03:20:17.732431+0000","last_peered":"2024-04-27T03:20:17.732431+0000","last_clean":"2024-04-27T03:20:17.732431+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:20:17.732431+0000","last_undegraded":"2024-04-27T03:20:17.732431+0000","last_fullsized":"2024-04-27T03:20:17.732431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2280,"ondisk_log_size":2280,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":11674,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1888,"num_read_kb":1653,"num_write":789,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":542,"num_omap_keys":4,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1675,"ondisk_log_size":1675,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":972,"num_read_kb":972,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":473,"ondisk_log_size":473,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":87,"num_write":316,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39912,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331928,"statfs":{"total":96636764160,"available":96595894272,"internally_reserved":0,"allocated":425984,"data_stored":148688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":983040,"data_stored":736592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705713,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":739335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39944,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331896,"statfs":{"total":96636764160,"available":96595861504,"internally_reserved":0,"allocated":458752,"data_stored":147935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.583 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.584 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.585 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.586 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.587 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.588 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 8.7 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.589 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.590 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.590 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.590 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.590 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.590 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.590 INFO:tasks.ceph:pgid 8.1 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.590 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 8.3 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.591 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 8.a last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.592 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 8.b last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.593 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.594 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.595 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.596 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.597 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.598 INFO:tasks.ceph:pgid 8.13 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.599 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.600 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-04-27T03:17:54.018430+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=54, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.601 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.602 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.603 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.604 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.605 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.606 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.606 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:33.606 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-27T03:21:53.607 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:21:53.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.729+0000 7f6ebf8ce640 1 Processor -- start 2024-04-27T03:21:53.738 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.729+0000 7f6ebf8ce640 1 -- start start 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.729+0000 7f6ebf8ce640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f6eb806f0d0 0x7f6eb81304f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.729+0000 7f6ebf8ce640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6eb8098610 con 0x7f6eb806e5e0 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.729+0000 7f6ebf8ce640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f6eb8070210 con 0x7f6eb806f0d0 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.729+0000 7f6ebd66a640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f6eb806e5e0 0x7f6eb806e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:39262/0 (socket says 172.21.15.77:39262) 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.729+0000 7f6ebd66a640 1 -- 172.21.15.77:0/119404543 learned_addr learned my addr 172.21.15.77:0/119404543 (peer_addr_for_me v1:172.21.15.77:0/0) 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebce69640 1 --2- 172.21.15.77:0/119404543 >> v2:172.21.15.77:3300/0 conn(0x7f6eb806f0d0 0x7f6eb81304f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6eaffff640 1 -- 172.21.15.77:0/119404543 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789451711 0 0) 0x7f6eb8098610 con 0x7f6eb806e5e0 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6eaffff640 1 -- 172.21.15.77:0/119404543 --> v1:172.21.15.77:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ea4003530 con 0x7f6eb806e5e0 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebce69640 1 -- 172.21.15.77:0/119404543 >> v1:172.21.15.77:6789/0 conn(0x7f6eb806e5e0 legacy=0x7f6eb806e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:53.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebce69640 1 -- 172.21.15.77:0/119404543 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb806ffd0 con 0x7f6eb806f0d0 2024-04-27T03:21:53.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebce69640 1 --2- 172.21.15.77:0/119404543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806f0d0 0x7f6eb81304f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009770 tx=0x7f6eb401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860f1ddc7ca8cc4b server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:53.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6eaffff640 1 -- 172.21.15.77:0/119404543 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6eb40b0070 con 0x7f6eb806f0d0 2024-04-27T03:21:53.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6eaffff640 1 -- 172.21.15.77:0/119404543 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eb40ab530 con 0x7f6eb806f0d0 2024-04-27T03:21:53.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6eaffff640 1 -- 172.21.15.77:0/119404543 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6eb40ab830 con 0x7f6eb806f0d0 2024-04-27T03:21:53.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/119404543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806f0d0 msgr2=0x7f6eb81304f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:53.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebf8ce640 1 --2- 172.21.15.77:0/119404543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806f0d0 0x7f6eb81304f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009770 tx=0x7f6eb401c370 comp rx=0 tx=0).stop 2024-04-27T03:21:53.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/119404543 shutdown_connections 2024-04-27T03:21:53.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebf8ce640 1 --2- 172.21.15.77:0/119404543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806f0d0 0x7f6eb81304f0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:53.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/119404543 >> 172.21.15.77:0/119404543 conn(0x7f6eb8099440 msgr2=0x7f6eb8099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:53.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/119404543 shutdown_connections 2024-04-27T03:21:53.743 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.733+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/119404543 wait complete. 2024-04-27T03:21:53.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebf8ce640 1 Processor -- start 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebf8ce640 1 -- start start 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebf8ce640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806e5e0 0x7f6eb812cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebf8ce640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb8133460 con 0x7f6eb806e5e0 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebd66a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806e5e0 0x7f6eb812cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebd66a640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806e5e0 0x7f6eb812cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40128/0 (socket says 172.21.15.77:40128) 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebd66a640 1 -- 172.21.15.77:0/3069742787 learned_addr learned my addr 172.21.15.77:0/3069742787 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebd66a640 1 -- 172.21.15.77:0/3069742787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb812f500 con 0x7f6eb806e5e0 2024-04-27T03:21:53.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebd66a640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806e5e0 0x7f6eb812cfd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800c450 tx=0x7f6ea800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:53.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6eadffb640 1 -- 172.21.15.77:0/3069742787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea8017070 con 0x7f6eb806e5e0 2024-04-27T03:21:53.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6eadffb640 1 -- 172.21.15.77:0/3069742787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea8005c20 con 0x7f6eb806e5e0 2024-04-27T03:21:53.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb812e890 con 0x7f6eb806e5e0 2024-04-27T03:21:53.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6eadffb640 1 -- 172.21.15.77:0/3069742787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ea8007070 con 0x7f6eb806e5e0 2024-04-27T03:21:53.746 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb812fff0 con 0x7f6eb806e5e0 2024-04-27T03:21:53.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6eadffb640 1 -- 172.21.15.77:0/3069742787 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6ea801a070 con 0x7f6eb806e5e0 2024-04-27T03:21:53.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb8062e50 con 0x7f6eb806e5e0 2024-04-27T03:21:53.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6eadffb640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6e9c035d60 0x7f6e9c038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:21:53.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6eadffb640 1 -- 172.21.15.77:0/3069742787 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f6ea8048500 con 0x7f6eb806e5e0 2024-04-27T03:21:53.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebce69640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6e9c035d60 0x7f6e9c038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:21:53.747 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.737+0000 7f6ebce69640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6e9c035d60 0x7f6e9c038220 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40ad000 tx=0x7f6eb40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:21:53.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.741+0000 7f6eadffb640 1 -- 172.21.15.77:0/3069742787 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6ea8012450 con 0x7f6eb806e5e0 2024-04-27T03:21:53.906 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.897+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6eb8070c60 con 0x7f6e9c035d60 2024-04-27T03:21:53.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.941+0000 7f6eadffb640 1 -- 172.21.15.77:0/3069742787 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+662170 (secure 0 0 0) 0x7f6eb8070c60 con 0x7f6e9c035d60 2024-04-27T03:21:53.957 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:21:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:21:53.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6e9c035d60 msgr2=0x7f6e9c038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6e9c035d60 0x7f6e9c038220 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40ad000 tx=0x7f6eb40a6000 comp rx=0 tx=0).stop 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806e5e0 msgr2=0x7f6eb812cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806e5e0 0x7f6eb812cfd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800c450 tx=0x7f6ea800c910 comp rx=0 tx=0).stop 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 shutdown_connections 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f6e9c035d60 0x7f6e9c038220 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 --2- 172.21.15.77:0/3069742787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6eb806e5e0 0x7f6eb812cfd0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 >> 172.21.15.77:0/3069742787 conn(0x7f6eb8099440 msgr2=0x7f6eb809b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 shutdown_connections 2024-04-27T03:21:53.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:21:53.953+0000 7f6ebf8ce640 1 -- 172.21.15.77:0/3069742787 wait complete. 2024-04-27T03:21:53.980 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-04-27T03:21:53.757167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605383,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":5393,"num_read_kb":4942,"num_write":2371,"num_write_kb":61374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2872,"num_omap_keys":15,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4992,"ondisk_log_size":4992,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":140024,"kb_used_data":2840,"kb_used_omap":6,"kb_used_meta":137145,"kb_avail":377347336,"statfs":{"total":386547056640,"available":386403672064,"internally_reserved":0,"allocated":2908160,"data_stored":1772550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":140437270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":180,"num_omap_keys":1,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006219"},"pg_stats":[{"pgid":"3.3f","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.986140+0000","last_change":"2024-04-27T03:17:47.025622+0000","last_active":"2024-04-27T03:21:07.986140+0000","last_peered":"2024-04-27T03:21:07.986140+0000","last_clean":"2024-04-27T03:21:07.986140+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:21:07.986140+0000","last_undegraded":"2024-04-27T03:21:07.986140+0000","last_fullsized":"2024-04-27T03:21:07.986140+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'6","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.158576+0000","last_change":"2024-04-27T03:21:42.158641+0000","last_active":"2024-04-27T03:21:42.158576+0000","last_peered":"2024-04-27T03:21:42.158576+0000","last_clean":"2024-04-27T03:21:42.158576+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:42.158576+0000","last_undegraded":"2024-04-27T03:21:42.158576+0000","last_fullsized":"2024-04-27T03:21:42.158576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:42.158374+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:42.158374+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.158374+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:53.231269+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.726062+0000","last_change":"2024-04-27T03:17:47.023458+0000","last_active":"2024-04-27T03:19:52.726062+0000","last_peered":"2024-04-27T03:19:52.726062+0000","last_clean":"2024-04-27T03:19:52.726062+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:19:52.726062+0000","last_undegraded":"2024-04-27T03:19:52.726062+0000","last_fullsized":"2024-04-27T03:19:52.726062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.789741+0000","last_change":"2024-04-27T03:21:43.789808+0000","last_active":"2024-04-27T03:21:43.789741+0000","last_peered":"2024-04-27T03:21:43.789741+0000","last_clean":"2024-04-27T03:21:43.789741+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:21:43.789741+0000","last_undegraded":"2024-04-27T03:21:43.789741+0000","last_fullsized":"2024-04-27T03:21:43.789741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'8","last_scrub_stamp":"2024-04-27T03:21:43.789520+0000","last_deep_scrub":"27'8","last_deep_scrub_stamp":"2024-04-27T03:21:43.789520+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.789520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:16.261778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.736697+0000","last_change":"2024-04-27T03:17:47.039594+0000","last_active":"2024-04-27T03:19:52.736697+0000","last_peered":"2024-04-27T03:19:52.736697+0000","last_clean":"2024-04-27T03:19:52.736697+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:19:52.736697+0000","last_undegraded":"2024-04-27T03:19:52.736697+0000","last_fullsized":"2024-04-27T03:19:52.736697+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.749840+0000","last_change":"2024-04-27T03:21:49.749992+0000","last_active":"2024-04-27T03:21:49.749840+0000","last_peered":"2024-04-27T03:21:49.749840+0000","last_clean":"2024-04-27T03:21:49.749840+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:21:49.749840+0000","last_undegraded":"2024-04-27T03:21:49.749840+0000","last_fullsized":"2024-04-27T03:21:49.749840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:49.749616+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:49.749616+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.749616+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:22.385004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.741829+0000","last_change":"2024-04-27T03:21:50.741891+0000","last_active":"2024-04-27T03:21:50.741829+0000","last_peered":"2024-04-27T03:21:50.741829+0000","last_clean":"2024-04-27T03:21:50.741829+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:21:50.741829+0000","last_undegraded":"2024-04-27T03:21:50.741829+0000","last_fullsized":"2024-04-27T03:21:50.741829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:50.741587+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:50.741587+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.741587+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:38.633527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":90,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.864487+0000","last_change":"2024-04-27T03:21:41.864610+0000","last_active":"2024-04-27T03:21:41.864487+0000","last_peered":"2024-04-27T03:21:41.864487+0000","last_clean":"2024-04-27T03:21:41.864487+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:41.864487+0000","last_undegraded":"2024-04-27T03:21:41.864487+0000","last_fullsized":"2024-04-27T03:21:41.864487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:41.864193+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:41.864193+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.864193+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:28.818568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747231+0000","last_change":"2024-04-27T03:17:47.015865+0000","last_active":"2024-04-27T03:20:17.747231+0000","last_peered":"2024-04-27T03:20:17.747231+0000","last_clean":"2024-04-27T03:20:17.747231+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:20:17.747231+0000","last_undegraded":"2024-04-27T03:20:17.747231+0000","last_fullsized":"2024-04-27T03:20:17.747231+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":20,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:37.742362+0000","last_change":"2024-04-27T03:18:34.776780+0000","last_active":"2024-04-27T03:20:37.742362+0000","last_peered":"2024-04-27T03:20:37.742362+0000","last_clean":"2024-04-27T03:20:37.742362+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:20:37.742362+0000","last_undegraded":"2024-04-27T03:20:37.742362+0000","last_fullsized":"2024-04-27T03:20:37.742362+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747191+0000","last_change":"2024-04-27T03:17:47.026206+0000","last_active":"2024-04-27T03:20:17.747191+0000","last_peered":"2024-04-27T03:20:17.747191+0000","last_clean":"2024-04-27T03:20:17.747191+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:20:17.747191+0000","last_undegraded":"2024-04-27T03:20:17.747191+0000","last_fullsized":"2024-04-27T03:20:17.747191+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'12","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.217060+0000","last_change":"2024-04-27T03:21:50.217129+0000","last_active":"2024-04-27T03:21:50.217060+0000","last_peered":"2024-04-27T03:21:50.217060+0000","last_clean":"2024-04-27T03:21:50.217060+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:21:50.217060+0000","last_undegraded":"2024-04-27T03:21:50.217060+0000","last_fullsized":"2024-04-27T03:21:50.217060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:50.216879+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:50.216879+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.216879+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:27.888741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.742935+0000","last_change":"2024-04-27T03:17:47.036460+0000","last_active":"2024-04-27T03:20:17.742935+0000","last_peered":"2024-04-27T03:20:17.742935+0000","last_clean":"2024-04-27T03:20:17.742935+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:20:17.742935+0000","last_undegraded":"2024-04-27T03:20:17.742935+0000","last_fullsized":"2024-04-27T03:20:17.742935+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.088254+0000","last_change":"2024-04-27T03:21:38.088312+0000","last_active":"2024-04-27T03:21:38.088254+0000","last_peered":"2024-04-27T03:21:38.088254+0000","last_clean":"2024-04-27T03:21:38.088254+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:38.088254+0000","last_undegraded":"2024-04-27T03:21:38.088254+0000","last_fullsized":"2024-04-27T03:21:38.088254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:38.088067+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:38.088067+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.088067+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:30.953622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.742856+0000","last_change":"2024-04-27T03:17:47.013877+0000","last_active":"2024-04-27T03:20:17.742856+0000","last_peered":"2024-04-27T03:20:17.742856+0000","last_clean":"2024-04-27T03:20:17.742856+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:20:17.742856+0000","last_undegraded":"2024-04-27T03:20:17.742856+0000","last_fullsized":"2024-04-27T03:20:17.742856+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.734748+0000","last_change":"2024-04-27T03:21:37.734857+0000","last_active":"2024-04-27T03:21:37.734748+0000","last_peered":"2024-04-27T03:21:37.734748+0000","last_clean":"2024-04-27T03:21:37.734748+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:21:37.734748+0000","last_undegraded":"2024-04-27T03:21:37.734748+0000","last_fullsized":"2024-04-27T03:21:37.734748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:37.734535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:37.734535+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.734535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:00.453711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.163703+0000","last_change":"2024-04-27T03:21:40.163767+0000","last_active":"2024-04-27T03:21:40.163703+0000","last_peered":"2024-04-27T03:21:40.163703+0000","last_clean":"2024-04-27T03:21:40.163703+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:40.163703+0000","last_undegraded":"2024-04-27T03:21:40.163703+0000","last_fullsized":"2024-04-27T03:21:40.163703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:40.163506+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:40.163506+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.163506+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:29.161646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.731193+0000","last_change":"2024-04-27T03:17:47.027209+0000","last_active":"2024-04-27T03:20:12.731193+0000","last_peered":"2024-04-27T03:20:12.731193+0000","last_clean":"2024-04-27T03:20:12.731193+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:20:12.731193+0000","last_undegraded":"2024-04-27T03:20:12.731193+0000","last_fullsized":"2024-04-27T03:20:12.731193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745824+0000","last_change":"2024-04-27T03:17:49.013701+0000","last_active":"2024-04-27T03:20:12.745824+0000","last_peered":"2024-04-27T03:20:12.745824+0000","last_clean":"2024-04-27T03:20:12.745824+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:20:12.745824+0000","last_undegraded":"2024-04-27T03:20:12.745824+0000","last_fullsized":"2024-04-27T03:20:12.745824+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.771974+0000","last_change":"2024-04-27T03:21:48.772040+0000","last_active":"2024-04-27T03:21:48.771974+0000","last_peered":"2024-04-27T03:21:48.771974+0000","last_clean":"2024-04-27T03:21:48.771974+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:21:48.771974+0000","last_undegraded":"2024-04-27T03:21:48.771974+0000","last_fullsized":"2024-04-27T03:21:48.771974+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:48.771778+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.771778+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.771778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:10.421790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.825947+0000","last_change":"2024-04-27T03:17:49.011747+0000","last_active":"2024-04-27T03:21:00.825947+0000","last_peered":"2024-04-27T03:21:00.825947+0000","last_clean":"2024-04-27T03:21:00.825947+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:21:00.825947+0000","last_undegraded":"2024-04-27T03:21:00.825947+0000","last_fullsized":"2024-04-27T03:21:00.825947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.746067+0000","last_change":"2024-04-27T03:17:47.023848+0000","last_active":"2024-04-27T03:20:12.746067+0000","last_peered":"2024-04-27T03:20:12.746067+0000","last_clean":"2024-04-27T03:20:12.746067+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:20:12.746067+0000","last_undegraded":"2024-04-27T03:20:12.746067+0000","last_fullsized":"2024-04-27T03:20:12.746067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.710830+0000","last_change":"2024-04-27T03:21:33.710892+0000","last_active":"2024-04-27T03:21:33.710830+0000","last_peered":"2024-04-27T03:21:33.710830+0000","last_clean":"2024-04-27T03:21:33.710830+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:33.710830+0000","last_undegraded":"2024-04-27T03:21:33.710830+0000","last_fullsized":"2024-04-27T03:21:33.710830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:33.710637+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:33.710637+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.710637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:36.041679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.745970+0000","last_change":"2024-04-27T03:17:47.012402+0000","last_active":"2024-04-27T03:20:12.745970+0000","last_peered":"2024-04-27T03:20:12.745970+0000","last_clean":"2024-04-27T03:20:12.745970+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:20:12.745970+0000","last_undegraded":"2024-04-27T03:20:12.745970+0000","last_fullsized":"2024-04-27T03:20:12.745970+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.716905+0000","last_change":"2024-04-27T03:21:34.716967+0000","last_active":"2024-04-27T03:21:34.716905+0000","last_peered":"2024-04-27T03:21:34.716905+0000","last_clean":"2024-04-27T03:21:34.716905+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:34.716905+0000","last_undegraded":"2024-04-27T03:21:34.716905+0000","last_fullsized":"2024-04-27T03:21:34.716905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:34.716714+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:34.716714+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.716714+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:06:56.440270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.741798+0000","last_change":"2024-04-27T03:17:47.036082+0000","last_active":"2024-04-27T03:20:12.741798+0000","last_peered":"2024-04-27T03:20:12.741798+0000","last_clean":"2024-04-27T03:20:12.741798+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:20:12.741798+0000","last_undegraded":"2024-04-27T03:20:12.741798+0000","last_fullsized":"2024-04-27T03:20:12.741798+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.165090+0000","last_change":"2024-04-27T03:17:49.005822+0000","last_active":"2024-04-27T03:21:01.165090+0000","last_peered":"2024-04-27T03:21:01.165090+0000","last_clean":"2024-04-27T03:21:01.165090+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:01.165090+0000","last_undegraded":"2024-04-27T03:21:01.165090+0000","last_fullsized":"2024-04-27T03:21:01.165090+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740064+0000","last_change":"2024-04-27T03:17:47.033466+0000","last_active":"2024-04-27T03:20:07.740064+0000","last_peered":"2024-04-27T03:20:07.740064+0000","last_clean":"2024-04-27T03:20:07.740064+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:20:07.740064+0000","last_undegraded":"2024-04-27T03:20:07.740064+0000","last_fullsized":"2024-04-27T03:20:07.740064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'37","reported_seq":106,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.340362+0000","last_change":"2024-04-27T03:17:57.088698+0000","last_active":"2024-04-27T03:21:01.340362+0000","last_peered":"2024-04-27T03:21:01.340362+0000","last_clean":"2024-04-27T03:21:01.340362+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:21:01.340362+0000","last_undegraded":"2024-04-27T03:21:01.340362+0000","last_fullsized":"2024-04-27T03:21:01.340362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.715048+0000","last_change":"2024-04-27T03:21:51.715106+0000","last_active":"2024-04-27T03:21:51.715048+0000","last_peered":"2024-04-27T03:21:51.715048+0000","last_clean":"2024-04-27T03:21:51.715048+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:51.715048+0000","last_undegraded":"2024-04-27T03:21:51.715048+0000","last_fullsized":"2024-04-27T03:21:51.715048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:51.714857+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:51.714857+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.714857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:02:41.985409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740181+0000","last_change":"2024-04-27T03:17:47.018824+0000","last_active":"2024-04-27T03:20:07.740181+0000","last_peered":"2024-04-27T03:20:07.740181+0000","last_clean":"2024-04-27T03:20:07.740181+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:20:07.740181+0000","last_undegraded":"2024-04-27T03:20:07.740181+0000","last_fullsized":"2024-04-27T03:20:07.740181+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.758982+0000","last_change":"2024-04-27T03:21:34.759034+0000","last_active":"2024-04-27T03:21:34.758982+0000","last_peered":"2024-04-27T03:21:34.758982+0000","last_clean":"2024-04-27T03:21:34.758982+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:21:34.758982+0000","last_undegraded":"2024-04-27T03:21:34.758982+0000","last_fullsized":"2024-04-27T03:21:34.758982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:34.758786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:34.758786+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.758786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:06.643052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'51","reported_seq":122,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.542440+0000","last_change":"2024-04-27T03:21:17.542518+0000","last_active":"2024-04-27T03:21:17.542440+0000","last_peered":"2024-04-27T03:21:17.542440+0000","last_clean":"2024-04-27T03:21:17.542440+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:21:17.542440+0000","last_undegraded":"2024-04-27T03:21:17.542440+0000","last_fullsized":"2024-04-27T03:21:17.542440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'51","last_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_deep_scrub":"31'51","last_deep_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.542266+0000","objects_scrubbed":6,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:24.169053+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.186893+0000","last_change":"2024-04-27T03:21:49.186953+0000","last_active":"2024-04-27T03:21:49.186893+0000","last_peered":"2024-04-27T03:21:49.186893+0000","last_clean":"2024-04-27T03:21:49.186893+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:21:49.186893+0000","last_undegraded":"2024-04-27T03:21:49.186893+0000","last_fullsized":"2024-04-27T03:21:49.186893+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:49.186698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:49.186698+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.186698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:31.658597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.157534+0000","last_change":"2024-04-27T03:17:57.061550+0000","last_active":"2024-04-27T03:21:01.157534+0000","last_peered":"2024-04-27T03:21:01.157534+0000","last_clean":"2024-04-27T03:21:01.157534+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:01.157534+0000","last_undegraded":"2024-04-27T03:21:01.157534+0000","last_fullsized":"2024-04-27T03:21:01.157534+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.784863+0000","last_change":"2024-04-27T03:21:46.784922+0000","last_active":"2024-04-27T03:21:46.784863+0000","last_peered":"2024-04-27T03:21:46.784863+0000","last_clean":"2024-04-27T03:21:46.784863+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:21:46.784863+0000","last_undegraded":"2024-04-27T03:21:46.784863+0000","last_fullsized":"2024-04-27T03:21:46.784863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:46.784639+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:46.784639+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.784639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:29.256538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.735578+0000","last_change":"2024-04-27T03:17:47.032159+0000","last_active":"2024-04-27T03:20:07.735578+0000","last_peered":"2024-04-27T03:20:07.735578+0000","last_clean":"2024-04-27T03:20:07.735578+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:20:07.735578+0000","last_undegraded":"2024-04-27T03:20:07.735578+0000","last_fullsized":"2024-04-27T03:20:07.735578+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.767822+0000","last_change":"2024-04-27T03:21:38.767879+0000","last_active":"2024-04-27T03:21:38.767822+0000","last_peered":"2024-04-27T03:21:38.767822+0000","last_clean":"2024-04-27T03:21:38.767822+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:21:38.767822+0000","last_undegraded":"2024-04-27T03:21:38.767822+0000","last_fullsized":"2024-04-27T03:21:38.767822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:38.767589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:38.767589+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.767589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:55.841408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"31'16","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.287986+0000","last_change":"2024-04-27T03:21:22.288067+0000","last_active":"2024-04-27T03:21:22.287986+0000","last_peered":"2024-04-27T03:21:22.287986+0000","last_clean":"2024-04-27T03:21:22.287986+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:21:22.287986+0000","last_undegraded":"2024-04-27T03:21:22.287986+0000","last_fullsized":"2024-04-27T03:21:22.287986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.287788+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:31.752629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.205494+0000","last_change":"2024-04-27T03:21:23.205547+0000","last_active":"2024-04-27T03:21:23.205494+0000","last_peered":"2024-04-27T03:21:23.205494+0000","last_clean":"2024-04-27T03:21:23.205494+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:21:23.205494+0000","last_undegraded":"2024-04-27T03:21:23.205494+0000","last_fullsized":"2024-04-27T03:21:23.205494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.205266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:32.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.745803+0000","last_change":"2024-04-27T03:17:57.074831+0000","last_active":"2024-04-27T03:20:27.745803+0000","last_peered":"2024-04-27T03:20:27.745803+0000","last_clean":"2024-04-27T03:20:27.745803+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:20:27.745803+0000","last_undegraded":"2024-04-27T03:20:27.745803+0000","last_fullsized":"2024-04-27T03:20:27.745803+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'16","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.813339+0000","last_change":"2024-04-27T03:21:39.813406+0000","last_active":"2024-04-27T03:21:39.813339+0000","last_peered":"2024-04-27T03:21:39.813339+0000","last_clean":"2024-04-27T03:21:39.813339+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:39.813339+0000","last_undegraded":"2024-04-27T03:21:39.813339+0000","last_fullsized":"2024-04-27T03:21:39.813339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:39.813135+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:39.813135+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.813135+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:50.886030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744546+0000","last_change":"2024-04-27T03:17:47.009721+0000","last_active":"2024-04-27T03:20:07.744546+0000","last_peered":"2024-04-27T03:20:07.744546+0000","last_clean":"2024-04-27T03:20:07.744546+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:20:07.744546+0000","last_undegraded":"2024-04-27T03:20:07.744546+0000","last_fullsized":"2024-04-27T03:20:07.744546+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.555607+0000","last_change":"2024-04-27T03:21:18.555657+0000","last_active":"2024-04-27T03:21:18.555607+0000","last_peered":"2024-04-27T03:21:18.555607+0000","last_clean":"2024-04-27T03:21:18.555607+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:21:18.555607+0000","last_undegraded":"2024-04-27T03:21:18.555607+0000","last_fullsized":"2024-04-27T03:21:18.555607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.555415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:17.141075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"31'29","reported_seq":84,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.786827+0000","last_change":"2024-04-27T03:21:28.786909+0000","last_active":"2024-04-27T03:21:28.786827+0000","last_peered":"2024-04-27T03:21:28.786827+0000","last_clean":"2024-04-27T03:21:28.786827+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:21:28.786827+0000","last_undegraded":"2024-04-27T03:21:28.786827+0000","last_fullsized":"2024-04-27T03:21:28.786827+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'29","last_scrub_stamp":"2024-04-27T03:21:28.786620+0000","last_deep_scrub":"31'29","last_deep_scrub_stamp":"2024-04-27T03:21:28.786620+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.786620+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:33.411918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'285","reported_seq":438,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.112035+0000","last_change":"2024-04-27T03:21:37.112095+0000","last_active":"2024-04-27T03:21:37.112035+0000","last_peered":"2024-04-27T03:21:37.112035+0000","last_clean":"2024-04-27T03:21:37.112035+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:37.112035+0000","last_undegraded":"2024-04-27T03:21:37.112035+0000","last_fullsized":"2024-04-27T03:21:37.112035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:37.111852+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:37.111852+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.111852+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:27.898262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.797908+0000","last_change":"2024-04-27T03:21:13.797972+0000","last_active":"2024-04-27T03:21:13.797908+0000","last_peered":"2024-04-27T03:21:13.797908+0000","last_clean":"2024-04-27T03:21:13.797908+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:21:13.797908+0000","last_undegraded":"2024-04-27T03:21:13.797908+0000","last_fullsized":"2024-04-27T03:21:13.797908+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.797701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:15.106812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.802531+0000","last_change":"2024-04-27T03:21:45.802590+0000","last_active":"2024-04-27T03:21:45.802531+0000","last_peered":"2024-04-27T03:21:45.802531+0000","last_clean":"2024-04-27T03:21:45.802531+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:45.802531+0000","last_undegraded":"2024-04-27T03:21:45.802531+0000","last_fullsized":"2024-04-27T03:21:45.802531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:45.802345+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:45.802345+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.802345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:23.495705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744463+0000","last_change":"2024-04-27T03:17:47.003169+0000","last_active":"2024-04-27T03:20:07.744463+0000","last_peered":"2024-04-27T03:20:07.744463+0000","last_clean":"2024-04-27T03:20:07.744463+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:20:07.744463+0000","last_undegraded":"2024-04-27T03:20:07.744463+0000","last_fullsized":"2024-04-27T03:20:07.744463+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.137288+0000","last_change":"2024-04-27T03:21:36.137341+0000","last_active":"2024-04-27T03:21:36.137288+0000","last_peered":"2024-04-27T03:21:36.137288+0000","last_clean":"2024-04-27T03:21:36.137288+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:21:36.137288+0000","last_undegraded":"2024-04-27T03:21:36.137288+0000","last_fullsized":"2024-04-27T03:21:36.137288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:36.137094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:36.137094+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.137094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:18.002317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'43","reported_seq":108,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.228263+0000","last_change":"2024-04-27T03:21:21.228344+0000","last_active":"2024-04-27T03:21:21.228263+0000","last_peered":"2024-04-27T03:21:21.228263+0000","last_clean":"2024-04-27T03:21:21.228263+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:21:21.228263+0000","last_undegraded":"2024-04-27T03:21:21.228263+0000","last_fullsized":"2024-04-27T03:21:21.228263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'43","last_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_deep_scrub":"31'43","last_deep_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.228072+0000","objects_scrubbed":5,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:05.540210+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'285","reported_seq":418,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.596919+0000","last_change":"2024-04-27T03:21:24.596980+0000","last_active":"2024-04-27T03:21:24.596919+0000","last_peered":"2024-04-27T03:21:24.596919+0000","last_clean":"2024-04-27T03:21:24.596919+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:24.596919+0000","last_undegraded":"2024-04-27T03:21:24.596919+0000","last_fullsized":"2024-04-27T03:21:24.596919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.596708+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:43.207566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'31","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.398240+0000","last_change":"2024-04-27T03:18:34.790257+0000","last_active":"2024-04-27T03:21:01.398240+0000","last_peered":"2024-04-27T03:21:01.398240+0000","last_clean":"2024-04-27T03:21:01.398240+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:21:01.398240+0000","last_undegraded":"2024-04-27T03:21:01.398240+0000","last_fullsized":"2024-04-27T03:21:01.398240+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.723623+0000","last_change":"2024-04-27T03:17:47.013403+0000","last_active":"2024-04-27T03:21:00.723623+0000","last_peered":"2024-04-27T03:21:00.723623+0000","last_clean":"2024-04-27T03:21:00.723623+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:21:00.723623+0000","last_undegraded":"2024-04-27T03:21:00.723623+0000","last_fullsized":"2024-04-27T03:21:00.723623+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.413586+0000","last_change":"2024-04-27T03:21:31.413670+0000","last_active":"2024-04-27T03:21:31.413586+0000","last_peered":"2024-04-27T03:21:31.413586+0000","last_clean":"2024-04-27T03:21:31.413586+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:21:31.413586+0000","last_undegraded":"2024-04-27T03:21:31.413586+0000","last_fullsized":"2024-04-27T03:21:31.413586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:31.413366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:31.413366+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.413366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:23.060435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'59","reported_seq":129,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.203730+0000","last_change":"2024-04-27T03:21:26.203825+0000","last_active":"2024-04-27T03:21:26.203730+0000","last_peered":"2024-04-27T03:21:26.203730+0000","last_clean":"2024-04-27T03:21:26.203730+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:21:26.203730+0000","last_undegraded":"2024-04-27T03:21:26.203730+0000","last_fullsized":"2024-04-27T03:21:26.203730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'59","last_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_deep_scrub":"31'59","last_deep_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.203534+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:44.814392+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":47,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.260194+0000","last_change":"2024-04-27T03:21:15.260248+0000","last_active":"2024-04-27T03:21:15.260194+0000","last_peered":"2024-04-27T03:21:15.260194+0000","last_clean":"2024-04-27T03:21:15.260194+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:21:15.260194+0000","last_undegraded":"2024-04-27T03:21:15.260194+0000","last_fullsized":"2024-04-27T03:21:15.260194+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.259998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:59.062074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.825785+0000","last_change":"2024-04-27T03:21:25.825848+0000","last_active":"2024-04-27T03:21:25.825785+0000","last_peered":"2024-04-27T03:21:25.825785+0000","last_clean":"2024-04-27T03:21:25.825785+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:21:25.825785+0000","last_undegraded":"2024-04-27T03:21:25.825785+0000","last_fullsized":"2024-04-27T03:21:25.825785+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'3","last_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_deep_scrub":"29'3","last_deep_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.825552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:43:06.575600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":78,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.537257+0000","last_change":"2024-04-27T03:21:50.537316+0000","last_active":"2024-04-27T03:21:50.537257+0000","last_peered":"2024-04-27T03:21:50.537257+0000","last_clean":"2024-04-27T03:21:50.537257+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:50.537257+0000","last_undegraded":"2024-04-27T03:21:50.537257+0000","last_fullsized":"2024-04-27T03:21:50.537257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:50.537066+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:50.537066+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.537066+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:53.904312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743585+0000","last_change":"2024-04-27T03:17:46.995080+0000","last_active":"2024-04-27T03:20:02.743585+0000","last_peered":"2024-04-27T03:20:02.743585+0000","last_clean":"2024-04-27T03:20:02.743585+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:20:02.743585+0000","last_undegraded":"2024-04-27T03:20:02.743585+0000","last_fullsized":"2024-04-27T03:20:02.743585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.762893+0000","last_change":"2024-04-27T03:21:43.762950+0000","last_active":"2024-04-27T03:21:43.762893+0000","last_peered":"2024-04-27T03:21:43.762893+0000","last_clean":"2024-04-27T03:21:43.762893+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:21:43.762893+0000","last_undegraded":"2024-04-27T03:21:43.762893+0000","last_fullsized":"2024-04-27T03:21:43.762893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:43.762700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:43.762700+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.762700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:13.310122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'44","reported_seq":109,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.518797+0000","last_change":"2024-04-27T03:21:13.518875+0000","last_active":"2024-04-27T03:21:13.518797+0000","last_peered":"2024-04-27T03:21:13.518797+0000","last_clean":"2024-04-27T03:21:13.518797+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:21:13.518797+0000","last_undegraded":"2024-04-27T03:21:13.518797+0000","last_fullsized":"2024-04-27T03:21:13.518797+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'44","last_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_deep_scrub":"31'44","last_deep_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.518592+0000","objects_scrubbed":5,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:24:22.864770+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.708239+0000","last_change":"2024-04-27T03:21:32.708291+0000","last_active":"2024-04-27T03:21:32.708239+0000","last_peered":"2024-04-27T03:21:32.708239+0000","last_clean":"2024-04-27T03:21:32.708239+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:21:32.708239+0000","last_undegraded":"2024-04-27T03:21:32.708239+0000","last_fullsized":"2024-04-27T03:21:32.708239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:32.708044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:32.708044+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.708044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:43.162536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"29'5","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.728245+0000","last_change":"2024-04-27T03:17:57.084457+0000","last_active":"2024-04-27T03:20:02.728245+0000","last_peered":"2024-04-27T03:20:02.728245+0000","last_clean":"2024-04-27T03:20:02.728245+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:20:02.728245+0000","last_undegraded":"2024-04-27T03:20:02.728245+0000","last_fullsized":"2024-04-27T03:20:02.728245+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.774871+0000","last_change":"2024-04-27T03:21:29.774989+0000","last_active":"2024-04-27T03:21:29.774871+0000","last_peered":"2024-04-27T03:21:29.774871+0000","last_clean":"2024-04-27T03:21:29.774871+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:29.774871+0000","last_undegraded":"2024-04-27T03:21:29.774871+0000","last_fullsized":"2024-04-27T03:21:29.774871+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-04-27T03:21:29.774563+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-04-27T03:21:29.774563+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.774563+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:41.424401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743552+0000","last_change":"2024-04-27T03:17:47.028376+0000","last_active":"2024-04-27T03:20:02.743552+0000","last_peered":"2024-04-27T03:20:02.743552+0000","last_clean":"2024-04-27T03:20:02.743552+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:20:02.743552+0000","last_undegraded":"2024-04-27T03:20:02.743552+0000","last_fullsized":"2024-04-27T03:20:02.743552+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.794463+0000","last_change":"2024-04-27T03:21:45.794520+0000","last_active":"2024-04-27T03:21:45.794463+0000","last_peered":"2024-04-27T03:21:45.794463+0000","last_clean":"2024-04-27T03:21:45.794463+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:21:45.794463+0000","last_undegraded":"2024-04-27T03:21:45.794463+0000","last_fullsized":"2024-04-27T03:21:45.794463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:45.794255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:45.794255+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.794255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:52.421042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'51","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.091926+0000","last_change":"2024-04-27T03:18:34.800954+0000","last_active":"2024-04-27T03:21:08.091926+0000","last_peered":"2024-04-27T03:21:08.091926+0000","last_clean":"2024-04-27T03:21:08.091926+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:21:08.091926+0000","last_undegraded":"2024-04-27T03:21:08.091926+0000","last_fullsized":"2024-04-27T03:21:08.091926+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'285","reported_seq":393,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.866360+0000","last_change":"2024-04-27T03:21:20.866425+0000","last_active":"2024-04-27T03:21:20.866360+0000","last_peered":"2024-04-27T03:21:20.866360+0000","last_clean":"2024-04-27T03:21:20.866360+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:20.866360+0000","last_undegraded":"2024-04-27T03:21:20.866360+0000","last_fullsized":"2024-04-27T03:21:20.866360+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.866168+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:18.678233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.628232+0000","last_change":"2024-04-27T03:21:40.628282+0000","last_active":"2024-04-27T03:21:40.628232+0000","last_peered":"2024-04-27T03:21:40.628232+0000","last_clean":"2024-04-27T03:21:40.628232+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:21:40.628232+0000","last_undegraded":"2024-04-27T03:21:40.628232+0000","last_fullsized":"2024-04-27T03:21:40.628232+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:40.628044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:40.628044+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.628044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:28:28.493954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":97,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.749222+0000","last_change":"2024-04-27T03:21:40.749288+0000","last_active":"2024-04-27T03:21:40.749222+0000","last_peered":"2024-04-27T03:21:40.749222+0000","last_clean":"2024-04-27T03:21:40.749222+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:40.749222+0000","last_undegraded":"2024-04-27T03:21:40.749222+0000","last_fullsized":"2024-04-27T03:21:40.749222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:40.749042+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:40.749042+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.749042+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:50.213883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743488+0000","last_change":"2024-04-27T03:18:34.772611+0000","last_active":"2024-04-27T03:20:02.743488+0000","last_peered":"2024-04-27T03:20:02.743488+0000","last_clean":"2024-04-27T03:20:02.743488+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:20:02.743488+0000","last_undegraded":"2024-04-27T03:20:02.743488+0000","last_fullsized":"2024-04-27T03:20:02.743488+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.756299+0000","last_change":"2024-04-27T03:21:30.756406+0000","last_active":"2024-04-27T03:21:30.756299+0000","last_peered":"2024-04-27T03:21:30.756299+0000","last_clean":"2024-04-27T03:21:30.756299+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:21:30.756299+0000","last_undegraded":"2024-04-27T03:21:30.756299+0000","last_fullsized":"2024-04-27T03:21:30.756299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:30.756077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:30.756077+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.756077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:41:43.084146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'77","reported_seq":166,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.631941+0000","last_change":"2024-04-27T03:21:27.632024+0000","last_active":"2024-04-27T03:21:27.631941+0000","last_peered":"2024-04-27T03:21:27.631941+0000","last_clean":"2024-04-27T03:21:27.631941+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:21:27.631941+0000","last_undegraded":"2024-04-27T03:21:27.631941+0000","last_fullsized":"2024-04-27T03:21:27.631941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'77","last_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_deep_scrub":"31'77","last_deep_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.631741+0000","objects_scrubbed":7,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:01.393467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":34,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.822541+0000","last_change":"2024-04-27T03:21:22.822651+0000","last_active":"2024-04-27T03:21:22.822541+0000","last_peered":"2024-04-27T03:21:22.822541+0000","last_clean":"2024-04-27T03:21:22.822541+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:21:22.822541+0000","last_undegraded":"2024-04-27T03:21:22.822541+0000","last_fullsized":"2024-04-27T03:21:22.822541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.822323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:20.663089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'21","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.464675+0000","last_change":"2024-04-27T03:21:32.464841+0000","last_active":"2024-04-27T03:21:32.464675+0000","last_peered":"2024-04-27T03:21:32.464675+0000","last_clean":"2024-04-27T03:21:32.464675+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:32.464675+0000","last_undegraded":"2024-04-27T03:21:32.464675+0000","last_fullsized":"2024-04-27T03:21:32.464675+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'21","last_scrub_stamp":"2024-04-27T03:21:32.464430+0000","last_deep_scrub":"31'21","last_deep_scrub_stamp":"2024-04-27T03:21:32.464430+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.464430+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:20.356371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.649742+0000","last_change":"2024-04-27T03:21:42.649809+0000","last_active":"2024-04-27T03:21:42.649742+0000","last_peered":"2024-04-27T03:21:42.649742+0000","last_clean":"2024-04-27T03:21:42.649742+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:42.649742+0000","last_undegraded":"2024-04-27T03:21:42.649742+0000","last_fullsized":"2024-04-27T03:21:42.649742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:42.649523+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:42.649523+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.649523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:58.314334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:52.745487+0000","last_change":"2024-04-27T03:18:34.767518+0000","last_active":"2024-04-27T03:20:52.745487+0000","last_peered":"2024-04-27T03:20:52.745487+0000","last_clean":"2024-04-27T03:20:52.745487+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:20:52.745487+0000","last_undegraded":"2024-04-27T03:20:52.745487+0000","last_fullsized":"2024-04-27T03:20:52.745487+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.740116+0000","last_change":"2024-04-27T03:21:39.740168+0000","last_active":"2024-04-27T03:21:39.740116+0000","last_peered":"2024-04-27T03:21:39.740116+0000","last_clean":"2024-04-27T03:21:39.740116+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:21:39.740116+0000","last_undegraded":"2024-04-27T03:21:39.740116+0000","last_fullsized":"2024-04-27T03:21:39.740116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:39.739923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:39.739923+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.739923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:52:10.953090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'48","reported_seq":114,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.092639+0000","last_change":"2024-04-27T03:21:30.092719+0000","last_active":"2024-04-27T03:21:30.092639+0000","last_peered":"2024-04-27T03:21:30.092639+0000","last_clean":"2024-04-27T03:21:30.092639+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:21:30.092639+0000","last_undegraded":"2024-04-27T03:21:30.092639+0000","last_fullsized":"2024-04-27T03:21:30.092639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'48","last_scrub_stamp":"2024-04-27T03:21:30.092441+0000","last_deep_scrub":"31'48","last_deep_scrub_stamp":"2024-04-27T03:21:30.092441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.092441+0000","objects_scrubbed":6,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:22.386772+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":63,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.585696+0000","last_change":"2024-04-27T03:21:23.585759+0000","last_active":"2024-04-27T03:21:23.585696+0000","last_peered":"2024-04-27T03:21:23.585696+0000","last_clean":"2024-04-27T03:21:23.585696+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:21:23.585696+0000","last_undegraded":"2024-04-27T03:21:23.585696+0000","last_fullsized":"2024-04-27T03:21:23.585696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.585484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:34.196580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'3","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.406842+0000","last_change":"2024-04-27T03:17:57.080943+0000","last_active":"2024-04-27T03:21:01.406842+0000","last_peered":"2024-04-27T03:21:01.406842+0000","last_clean":"2024-04-27T03:21:01.406842+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:21:01.406842+0000","last_undegraded":"2024-04-27T03:21:01.406842+0000","last_fullsized":"2024-04-27T03:21:01.406842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.193531+0000","last_change":"2024-04-27T03:21:48.193588+0000","last_active":"2024-04-27T03:21:48.193531+0000","last_peered":"2024-04-27T03:21:48.193531+0000","last_clean":"2024-04-27T03:21:48.193531+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:21:48.193531+0000","last_undegraded":"2024-04-27T03:21:48.193531+0000","last_fullsized":"2024-04-27T03:21:48.193531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:48.193325+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.193325+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.193325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:25.886685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.742240+0000","last_change":"2024-04-27T03:17:47.026419+0000","last_active":"2024-04-27T03:19:57.742240+0000","last_peered":"2024-04-27T03:19:57.742240+0000","last_clean":"2024-04-27T03:19:57.742240+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:19:57.742240+0000","last_undegraded":"2024-04-27T03:19:57.742240+0000","last_fullsized":"2024-04-27T03:19:57.742240+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.324797+0000","last_change":"2024-04-27T03:21:12.324853+0000","last_active":"2024-04-27T03:21:12.324797+0000","last_peered":"2024-04-27T03:21:12.324797+0000","last_clean":"2024-04-27T03:21:12.324797+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:21:12.324797+0000","last_undegraded":"2024-04-27T03:21:12.324797+0000","last_fullsized":"2024-04-27T03:21:12.324797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.324599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:41:24.652668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'45","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.786964+0000","last_change":"2024-04-27T03:21:26.787057+0000","last_active":"2024-04-27T03:21:26.786964+0000","last_peered":"2024-04-27T03:21:26.786964+0000","last_clean":"2024-04-27T03:21:26.786964+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:21:26.786964+0000","last_undegraded":"2024-04-27T03:21:26.786964+0000","last_fullsized":"2024-04-27T03:21:26.786964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.786765+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:28.251865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'285","reported_seq":420,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.779843+0000","last_change":"2024-04-27T03:21:18.779964+0000","last_active":"2024-04-27T03:21:18.779843+0000","last_peered":"2024-04-27T03:21:18.779843+0000","last_clean":"2024-04-27T03:21:18.779843+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:18.779843+0000","last_undegraded":"2024-04-27T03:21:18.779843+0000","last_fullsized":"2024-04-27T03:21:18.779843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.779620+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:58:08.955224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'13","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.591103+0000","last_change":"2024-04-27T03:21:44.591263+0000","last_active":"2024-04-27T03:21:44.591103+0000","last_peered":"2024-04-27T03:21:44.591103+0000","last_clean":"2024-04-27T03:21:44.591103+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:21:44.591103+0000","last_undegraded":"2024-04-27T03:21:44.591103+0000","last_fullsized":"2024-04-27T03:21:44.591103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-04-27T03:21:44.590873+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-04-27T03:21:44.590873+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.590873+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:31.004553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":180,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.745548+0000","last_change":"2024-04-27T03:21:38.745694+0000","last_active":"2024-04-27T03:21:38.745548+0000","last_peered":"2024-04-27T03:21:38.745548+0000","last_clean":"2024-04-27T03:21:38.745548+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:21:38.745548+0000","last_undegraded":"2024-04-27T03:21:38.745548+0000","last_fullsized":"2024-04-27T03:21:38.745548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'8","last_scrub_stamp":"2024-04-27T03:21:38.745283+0000","last_deep_scrub":"27'8","last_deep_scrub_stamp":"2024-04-27T03:21:38.745283+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.745283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:11:16.380426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737562+0000","last_change":"2024-04-27T03:17:47.036712+0000","last_active":"2024-04-27T03:19:57.737562+0000","last_peered":"2024-04-27T03:19:57.737562+0000","last_clean":"2024-04-27T03:19:57.737562+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:19:57.737562+0000","last_undegraded":"2024-04-27T03:19:57.737562+0000","last_fullsized":"2024-04-27T03:19:57.737562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'5","reported_seq":432,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.080708+0000","last_change":"2024-04-27T03:21:14.080869+0000","last_active":"2024-04-27T03:21:14.080708+0000","last_peered":"2024-04-27T03:21:14.080708+0000","last_clean":"2024-04-27T03:21:14.080708+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:21:14.080708+0000","last_undegraded":"2024-04-27T03:21:14.080708+0000","last_fullsized":"2024-04-27T03:21:14.080708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.080480+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:27.352786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":385,"num_read_kb":385,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"31'16","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.621515+0000","last_change":"2024-04-27T03:21:20.621586+0000","last_active":"2024-04-27T03:21:20.621515+0000","last_peered":"2024-04-27T03:21:20.621515+0000","last_clean":"2024-04-27T03:21:20.621515+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:21:20.621515+0000","last_undegraded":"2024-04-27T03:21:20.621515+0000","last_fullsized":"2024-04-27T03:21:20.621515+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.621330+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:04.933468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'285","reported_seq":476,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.100254+0000","last_change":"2024-04-27T03:21:33.100400+0000","last_active":"2024-04-27T03:21:33.100254+0000","last_peered":"2024-04-27T03:21:33.100254+0000","last_clean":"2024-04-27T03:21:33.100254+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:33.100254+0000","last_undegraded":"2024-04-27T03:21:33.100254+0000","last_fullsized":"2024-04-27T03:21:33.100254+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:33.100022+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:33.100022+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.100022+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:20.619756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.766179+0000","last_change":"2024-04-27T03:21:17.766241+0000","last_active":"2024-04-27T03:21:17.766179+0000","last_peered":"2024-04-27T03:21:17.766179+0000","last_clean":"2024-04-27T03:21:17.766179+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:21:17.766179+0000","last_undegraded":"2024-04-27T03:21:17.766179+0000","last_fullsized":"2024-04-27T03:21:17.766179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.765988+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:49.164397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.500158+0000","last_change":"2024-04-27T03:21:34.500219+0000","last_active":"2024-04-27T03:21:34.500158+0000","last_peered":"2024-04-27T03:21:34.500158+0000","last_clean":"2024-04-27T03:21:34.500158+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:21:34.500158+0000","last_undegraded":"2024-04-27T03:21:34.500158+0000","last_fullsized":"2024-04-27T03:21:34.500158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:34.499952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:34.499952+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.499952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:49:24.665238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737526+0000","last_change":"2024-04-27T03:17:47.031891+0000","last_active":"2024-04-27T03:19:57.737526+0000","last_peered":"2024-04-27T03:19:57.737526+0000","last_clean":"2024-04-27T03:19:57.737526+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:19:57.737526+0000","last_undegraded":"2024-04-27T03:19:57.737526+0000","last_fullsized":"2024-04-27T03:19:57.737526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.628128+0000","last_change":"2024-04-27T03:21:21.628229+0000","last_active":"2024-04-27T03:21:21.628128+0000","last_peered":"2024-04-27T03:21:21.628128+0000","last_clean":"2024-04-27T03:21:21.628128+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:21:21.628128+0000","last_undegraded":"2024-04-27T03:21:21.628128+0000","last_fullsized":"2024-04-27T03:21:21.628128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.627898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:37.847354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'48","reported_seq":111,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.145209+0000","last_change":"2024-04-27T03:21:31.145327+0000","last_active":"2024-04-27T03:21:31.145209+0000","last_peered":"2024-04-27T03:21:31.145209+0000","last_clean":"2024-04-27T03:21:31.145209+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:21:31.145209+0000","last_undegraded":"2024-04-27T03:21:31.145209+0000","last_fullsized":"2024-04-27T03:21:31.145209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'48","last_scrub_stamp":"2024-04-27T03:21:31.145012+0000","last_deep_scrub":"31'48","last_deep_scrub_stamp":"2024-04-27T03:21:31.145012+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.145012+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:50:53.240778+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":37,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.235688+0000","last_change":"2024-04-27T03:21:45.235740+0000","last_active":"2024-04-27T03:21:45.235688+0000","last_peered":"2024-04-27T03:21:45.235688+0000","last_clean":"2024-04-27T03:21:45.235688+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:21:45.235688+0000","last_undegraded":"2024-04-27T03:21:45.235688+0000","last_fullsized":"2024-04-27T03:21:45.235688+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:45.235457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:45.235457+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.235457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:42.520721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'9","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.523741+0000","last_change":"2024-04-27T03:21:36.523804+0000","last_active":"2024-04-27T03:21:36.523741+0000","last_peered":"2024-04-27T03:21:36.523741+0000","last_clean":"2024-04-27T03:21:36.523741+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:36.523741+0000","last_undegraded":"2024-04-27T03:21:36.523741+0000","last_fullsized":"2024-04-27T03:21:36.523741+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:36.523556+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:36.523556+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.523556+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:00.443161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.191277+0000","last_change":"2024-04-27T03:21:52.191335+0000","last_active":"2024-04-27T03:21:52.191277+0000","last_peered":"2024-04-27T03:21:52.191277+0000","last_clean":"2024-04-27T03:21:52.191277+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:52.191277+0000","last_undegraded":"2024-04-27T03:21:52.191277+0000","last_fullsized":"2024-04-27T03:21:52.191277+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:52.191088+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:52.191088+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.191088+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:24.826476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"11'117","reported_seq":203,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.859533+0000","last_change":"2024-04-27T03:17:39.633957+0000","last_active":"2024-04-27T03:21:11.859533+0000","last_peered":"2024-04-27T03:21:11.859533+0000","last_clean":"2024-04-27T03:21:11.859533+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:21:11.859533+0000","last_undegraded":"2024-04-27T03:21:11.859533+0000","last_fullsized":"2024-04-27T03:21:11.859533+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406527+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:21:11.406527+0000","last_peered":"2024-04-27T03:21:11.406527+0000","last_clean":"2024-04-27T03:21:11.406527+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:21:11.406527+0000","last_undegraded":"2024-04-27T03:21:11.406527+0000","last_fullsized":"2024-04-27T03:21:11.406527+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.106795+0000","last_change":"2024-04-27T03:17:47.039911+0000","last_active":"2024-04-27T03:21:01.106795+0000","last_peered":"2024-04-27T03:21:01.106795+0000","last_clean":"2024-04-27T03:21:01.106795+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:21:01.106795+0000","last_undegraded":"2024-04-27T03:21:01.106795+0000","last_fullsized":"2024-04-27T03:21:01.106795+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.278555+0000","last_change":"2024-04-27T03:21:25.278617+0000","last_active":"2024-04-27T03:21:25.278555+0000","last_peered":"2024-04-27T03:21:25.278555+0000","last_clean":"2024-04-27T03:21:25.278555+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:21:25.278555+0000","last_undegraded":"2024-04-27T03:21:25.278555+0000","last_fullsized":"2024-04-27T03:21:25.278555+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.278344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:54.825766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'94","reported_seq":194,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.155140+0000","last_change":"2024-04-27T03:21:28.155217+0000","last_active":"2024-04-27T03:21:28.155140+0000","last_peered":"2024-04-27T03:21:28.155140+0000","last_clean":"2024-04-27T03:21:28.155140+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:21:28.155140+0000","last_undegraded":"2024-04-27T03:21:28.155140+0000","last_fullsized":"2024-04-27T03:21:28.155140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'94","last_scrub_stamp":"2024-04-27T03:21:28.154943+0000","last_deep_scrub":"31'94","last_deep_scrub_stamp":"2024-04-27T03:21:28.154943+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.154943+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:02.805683+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":81,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.635119+0000","last_change":"2024-04-27T03:21:25.635175+0000","last_active":"2024-04-27T03:21:25.635119+0000","last_peered":"2024-04-27T03:21:25.635119+0000","last_clean":"2024-04-27T03:21:25.635119+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:21:25.635119+0000","last_undegraded":"2024-04-27T03:21:25.635119+0000","last_fullsized":"2024-04-27T03:21:25.635119+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.634932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:40.208378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":176,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.825556+0000","last_change":"2024-04-27T03:21:21.825636+0000","last_active":"2024-04-27T03:21:21.825556+0000","last_peered":"2024-04-27T03:21:21.825556+0000","last_clean":"2024-04-27T03:21:21.825556+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:21.825556+0000","last_undegraded":"2024-04-27T03:21:21.825556+0000","last_fullsized":"2024-04-27T03:21:21.825556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'82","last_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_deep_scrub":"31'82","last_deep_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.825358+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:31:11.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.248347+0000","last_change":"2024-04-27T03:17:49.007601+0000","last_active":"2024-04-27T03:21:01.248347+0000","last_peered":"2024-04-27T03:21:01.248347+0000","last_clean":"2024-04-27T03:21:01.248347+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:21:01.248347+0000","last_undegraded":"2024-04-27T03:21:01.248347+0000","last_fullsized":"2024-04-27T03:21:01.248347+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406448+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:21:11.406448+0000","last_peered":"2024-04-27T03:21:11.406448+0000","last_clean":"2024-04-27T03:21:11.406448+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:21:11.406448+0000","last_undegraded":"2024-04-27T03:21:11.406448+0000","last_fullsized":"2024-04-27T03:21:11.406448+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.725805+0000","last_change":"2024-04-27T03:17:47.015528+0000","last_active":"2024-04-27T03:19:52.725805+0000","last_peered":"2024-04-27T03:19:52.725805+0000","last_clean":"2024-04-27T03:19:52.725805+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:19:52.725805+0000","last_undegraded":"2024-04-27T03:19:52.725805+0000","last_fullsized":"2024-04-27T03:19:52.725805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.174097+0000","last_change":"2024-04-27T03:21:18.174150+0000","last_active":"2024-04-27T03:21:18.174097+0000","last_peered":"2024-04-27T03:21:18.174097+0000","last_clean":"2024-04-27T03:21:18.174097+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:21:18.174097+0000","last_undegraded":"2024-04-27T03:21:18.174097+0000","last_fullsized":"2024-04-27T03:21:18.174097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.173908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:30.769576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'297","reported_seq":533,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.209080+0000","last_change":"2024-04-27T03:21:17.209173+0000","last_active":"2024-04-27T03:21:17.209080+0000","last_peered":"2024-04-27T03:21:17.209080+0000","last_clean":"2024-04-27T03:21:17.209080+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:17.209080+0000","last_undegraded":"2024-04-27T03:21:17.209080+0000","last_fullsized":"2024-04-27T03:21:17.209080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'297","last_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_deep_scrub":"31'297","last_deep_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.208885+0000","objects_scrubbed":11,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:24.543836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":387,"num_read_kb":371,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.591116+0000","last_change":"2024-04-27T03:21:22.591170+0000","last_active":"2024-04-27T03:21:22.591116+0000","last_peered":"2024-04-27T03:21:22.591116+0000","last_clean":"2024-04-27T03:21:22.591116+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:21:22.591116+0000","last_undegraded":"2024-04-27T03:21:22.591116+0000","last_fullsized":"2024-04-27T03:21:22.591116+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.590921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:10.482862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729433+0000","last_change":"2024-04-27T03:17:57.086888+0000","last_active":"2024-04-27T03:20:07.729433+0000","last_peered":"2024-04-27T03:20:07.729433+0000","last_clean":"2024-04-27T03:20:07.729433+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:20:07.729433+0000","last_undegraded":"2024-04-27T03:20:07.729433+0000","last_fullsized":"2024-04-27T03:20:07.729433+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.558089+0000","last_change":"2024-04-27T03:21:37.558256+0000","last_active":"2024-04-27T03:21:37.558089+0000","last_peered":"2024-04-27T03:21:37.558089+0000","last_clean":"2024-04-27T03:21:37.558089+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:37.558089+0000","last_undegraded":"2024-04-27T03:21:37.558089+0000","last_fullsized":"2024-04-27T03:21:37.558089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:37.557861+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:37.557861+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.557861+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:29.852192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.792507+0000","last_change":"2024-04-27T03:21:47.792561+0000","last_active":"2024-04-27T03:21:47.792507+0000","last_peered":"2024-04-27T03:21:47.792507+0000","last_clean":"2024-04-27T03:21:47.792507+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:21:47.792507+0000","last_undegraded":"2024-04-27T03:21:47.792507+0000","last_fullsized":"2024-04-27T03:21:47.792507+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:47.792322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:47.792322+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.792322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:15:30.180059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.859333+0000","last_change":"2024-04-27T03:17:47.011695+0000","last_active":"2024-04-27T03:21:11.859333+0000","last_peered":"2024-04-27T03:21:11.859333+0000","last_clean":"2024-04-27T03:21:11.859333+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:21:11.859333+0000","last_undegraded":"2024-04-27T03:21:11.859333+0000","last_fullsized":"2024-04-27T03:21:11.859333+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"31'2","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.787899+0000","last_change":"2024-04-27T03:21:41.787965+0000","last_active":"2024-04-27T03:21:41.787899+0000","last_peered":"2024-04-27T03:21:41.787899+0000","last_clean":"2024-04-27T03:21:41.787899+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:21:41.787899+0000","last_undegraded":"2024-04-27T03:21:41.787899+0000","last_fullsized":"2024-04-27T03:21:41.787899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:41.787706+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:41.787706+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.787706+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:24:51.133885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'23","reported_seq":105,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.192662+0000","last_change":"2024-04-27T03:21:27.192744+0000","last_active":"2024-04-27T03:21:27.192662+0000","last_peered":"2024-04-27T03:21:27.192662+0000","last_clean":"2024-04-27T03:21:27.192662+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:27.192662+0000","last_undegraded":"2024-04-27T03:21:27.192662+0000","last_fullsized":"2024-04-27T03:21:27.192662+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.192466+0000","objects_scrubbed":8,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:41.765912+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.232293+0000","last_change":"2024-04-27T03:21:16.232354+0000","last_active":"2024-04-27T03:21:16.232293+0000","last_peered":"2024-04-27T03:21:16.232293+0000","last_clean":"2024-04-27T03:21:16.232293+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:21:16.232293+0000","last_undegraded":"2024-04-27T03:21:16.232293+0000","last_fullsized":"2024-04-27T03:21:16.232293+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.232084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:05.186600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.711510+0000","last_change":"2024-04-27T03:21:52.711568+0000","last_active":"2024-04-27T03:21:52.711510+0000","last_peered":"2024-04-27T03:21:52.711510+0000","last_clean":"2024-04-27T03:21:52.711510+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:52.711510+0000","last_undegraded":"2024-04-27T03:21:52.711510+0000","last_fullsized":"2024-04-27T03:21:52.711510+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:52.711315+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:52.711315+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.711315+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:37:09.185124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.564206+0000","last_change":"2024-04-27T03:21:45.564265+0000","last_active":"2024-04-27T03:21:45.564206+0000","last_peered":"2024-04-27T03:21:45.564206+0000","last_clean":"2024-04-27T03:21:45.564206+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:45.564206+0000","last_undegraded":"2024-04-27T03:21:45.564206+0000","last_fullsized":"2024-04-27T03:21:45.564206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:45.564008+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:45.564008+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.564008+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:34:05.340194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406448+0000","last_change":"2024-04-27T03:17:39.633541+0000","last_active":"2024-04-27T03:21:11.406448+0000","last_peered":"2024-04-27T03:21:11.406448+0000","last_clean":"2024-04-27T03:21:11.406448+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:21:11.406448+0000","last_undegraded":"2024-04-27T03:21:11.406448+0000","last_fullsized":"2024-04-27T03:21:11.406448+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.140255+0000","last_change":"2024-04-27T03:17:47.015018+0000","last_active":"2024-04-27T03:21:01.140255+0000","last_peered":"2024-04-27T03:21:01.140255+0000","last_clean":"2024-04-27T03:21:01.140255+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:21:01.140255+0000","last_undegraded":"2024-04-27T03:21:01.140255+0000","last_fullsized":"2024-04-27T03:21:01.140255+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.366941+0000","last_change":"2024-04-27T03:21:30.367056+0000","last_active":"2024-04-27T03:21:30.366941+0000","last_peered":"2024-04-27T03:21:30.366941+0000","last_clean":"2024-04-27T03:21:30.366941+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:21:30.366941+0000","last_undegraded":"2024-04-27T03:21:30.366941+0000","last_fullsized":"2024-04-27T03:21:30.366941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:30.366712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:30.366712+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.366712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:19:52.016724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"31'21","reported_seq":67,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.748709+0000","last_change":"2024-04-27T03:21:33.748898+0000","last_active":"2024-04-27T03:21:33.748709+0000","last_peered":"2024-04-27T03:21:33.748709+0000","last_clean":"2024-04-27T03:21:33.748709+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:33.748709+0000","last_undegraded":"2024-04-27T03:21:33.748709+0000","last_fullsized":"2024-04-27T03:21:33.748709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'21","last_scrub_stamp":"2024-04-27T03:21:33.748366+0000","last_deep_scrub":"31'21","last_deep_scrub_stamp":"2024-04-27T03:21:33.748366+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.748366+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:49.413176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":350,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.816607+0000","last_change":"2024-04-27T03:21:40.816658+0000","last_active":"2024-04-27T03:21:40.816607+0000","last_peered":"2024-04-27T03:21:40.816607+0000","last_clean":"2024-04-27T03:21:40.816607+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:21:40.816607+0000","last_undegraded":"2024-04-27T03:21:40.816607+0000","last_fullsized":"2024-04-27T03:21:40.816607+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:40.816420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:40.816420+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.816420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:44.183666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.968086+0000","last_change":"2024-04-27T03:17:57.089242+0000","last_active":"2024-04-27T03:21:00.968086+0000","last_peered":"2024-04-27T03:21:00.968086+0000","last_clean":"2024-04-27T03:21:00.968086+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:21:00.968086+0000","last_undegraded":"2024-04-27T03:21:00.968086+0000","last_fullsized":"2024-04-27T03:21:00.968086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.316306+0000","last_change":"2024-04-27T03:17:49.006116+0000","last_active":"2024-04-27T03:21:01.316306+0000","last_peered":"2024-04-27T03:21:01.316306+0000","last_clean":"2024-04-27T03:21:01.316306+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:01.316306+0000","last_undegraded":"2024-04-27T03:21:01.316306+0000","last_fullsized":"2024-04-27T03:21:01.316306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741133+0000","last_change":"2024-04-27T03:17:47.024776+0000","last_active":"2024-04-27T03:19:52.741133+0000","last_peered":"2024-04-27T03:19:52.741133+0000","last_clean":"2024-04-27T03:19:52.741133+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:19:52.741133+0000","last_undegraded":"2024-04-27T03:19:52.741133+0000","last_fullsized":"2024-04-27T03:19:52.741133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.859157+0000","last_change":"2024-04-27T03:17:39.636125+0000","last_active":"2024-04-27T03:21:11.859157+0000","last_peered":"2024-04-27T03:21:11.859157+0000","last_clean":"2024-04-27T03:21:11.859157+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:21:11.859157+0000","last_undegraded":"2024-04-27T03:21:11.859157+0000","last_fullsized":"2024-04-27T03:21:11.859157+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'1","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.872299+0000","last_change":"2024-04-27T03:17:51.037696+0000","last_active":"2024-04-27T03:21:07.872299+0000","last_peered":"2024-04-27T03:21:07.872299+0000","last_clean":"2024-04-27T03:21:07.872299+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:21:07.872299+0000","last_undegraded":"2024-04-27T03:21:07.872299+0000","last_fullsized":"2024-04-27T03:21:07.872299+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'87","reported_seq":180,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.199826+0000","last_change":"2024-04-27T03:21:18.199826+0000","last_active":"2024-04-27T03:21:18.199826+0000","last_peered":"2024-04-27T03:21:18.199826+0000","last_clean":"2024-04-27T03:21:18.199826+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:21:18.199826+0000","last_undegraded":"2024-04-27T03:21:18.199826+0000","last_fullsized":"2024-04-27T03:21:18.199826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'87","last_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_deep_scrub":"31'87","last_deep_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.199538+0000","objects_scrubbed":11,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:49.872043+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":89,"num_read_kb":75,"num_write":42,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":292,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.101164+0000","last_change":"2024-04-27T03:21:29.101226+0000","last_active":"2024-04-27T03:21:29.101164+0000","last_peered":"2024-04-27T03:21:29.101164+0000","last_clean":"2024-04-27T03:21:29.101164+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:21:29.101164+0000","last_undegraded":"2024-04-27T03:21:29.101164+0000","last_fullsized":"2024-04-27T03:21:29.101164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:29.100967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:29.100967+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.100967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:02.862694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.745180+0000","last_change":"2024-04-27T03:21:15.745242+0000","last_active":"2024-04-27T03:21:15.745180+0000","last_peered":"2024-04-27T03:21:15.745180+0000","last_clean":"2024-04-27T03:21:15.745180+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:21:15.745180+0000","last_undegraded":"2024-04-27T03:21:15.745180+0000","last_fullsized":"2024-04-27T03:21:15.745180+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.744992+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:00.286559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.623061+0000","last_change":"2024-04-27T03:21:43.623119+0000","last_active":"2024-04-27T03:21:43.623061+0000","last_peered":"2024-04-27T03:21:43.623061+0000","last_clean":"2024-04-27T03:21:43.623061+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:21:43.623061+0000","last_undegraded":"2024-04-27T03:21:43.623061+0000","last_fullsized":"2024-04-27T03:21:43.623061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:43.622863+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:43.622863+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.622863+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:25.488085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'6","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.509989+0000","last_change":"2024-04-27T03:21:49.510055+0000","last_active":"2024-04-27T03:21:49.509989+0000","last_peered":"2024-04-27T03:21:49.509989+0000","last_clean":"2024-04-27T03:21:49.509989+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:49.509989+0000","last_undegraded":"2024-04-27T03:21:49.509989+0000","last_fullsized":"2024-04-27T03:21:49.509989+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:49.509757+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:49.509757+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.509757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:00.582652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406515+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:21:11.406515+0000","last_peered":"2024-04-27T03:21:11.406515+0000","last_clean":"2024-04-27T03:21:11.406515+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:21:11.406515+0000","last_undegraded":"2024-04-27T03:21:11.406515+0000","last_fullsized":"2024-04-27T03:21:11.406515+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.128323+0000","last_change":"2024-04-27T03:21:16.128384+0000","last_active":"2024-04-27T03:21:16.128323+0000","last_peered":"2024-04-27T03:21:16.128323+0000","last_clean":"2024-04-27T03:21:16.128323+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:21:16.128323+0000","last_undegraded":"2024-04-27T03:21:16.128323+0000","last_fullsized":"2024-04-27T03:21:16.128323+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.128135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:44.502862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"31'28","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.261647+0000","last_change":"2024-04-27T03:21:21.261734+0000","last_active":"2024-04-27T03:21:21.261647+0000","last_peered":"2024-04-27T03:21:21.261647+0000","last_clean":"2024-04-27T03:21:21.261647+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:21:21.261647+0000","last_undegraded":"2024-04-27T03:21:21.261647+0000","last_fullsized":"2024-04-27T03:21:21.261647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'28","last_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_deep_scrub":"31'28","last_deep_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.261420+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:52.474587+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'285","reported_seq":373,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.251214+0000","last_change":"2024-04-27T03:21:24.251366+0000","last_active":"2024-04-27T03:21:24.251214+0000","last_peered":"2024-04-27T03:21:24.251214+0000","last_clean":"2024-04-27T03:21:24.251214+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:24.251214+0000","last_undegraded":"2024-04-27T03:21:24.251214+0000","last_fullsized":"2024-04-27T03:21:24.251214+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.250979+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:52.625705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.251899+0000","last_change":"2024-04-27T03:17:57.091210+0000","last_active":"2024-04-27T03:21:01.251899+0000","last_peered":"2024-04-27T03:21:01.251899+0000","last_clean":"2024-04-27T03:21:01.251899+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:21:01.251899+0000","last_undegraded":"2024-04-27T03:21:01.251899+0000","last_fullsized":"2024-04-27T03:21:01.251899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.748000+0000","last_change":"2024-04-27T03:17:49.007942+0000","last_active":"2024-04-27T03:21:12.748000+0000","last_peered":"2024-04-27T03:21:12.748000+0000","last_clean":"2024-04-27T03:21:12.748000+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:21:12.748000+0000","last_undegraded":"2024-04-27T03:21:12.748000+0000","last_fullsized":"2024-04-27T03:21:12.748000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737821+0000","last_change":"2024-04-27T03:17:47.039532+0000","last_active":"2024-04-27T03:19:57.737821+0000","last_peered":"2024-04-27T03:19:57.737821+0000","last_clean":"2024-04-27T03:19:57.737821+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:19:57.737821+0000","last_undegraded":"2024-04-27T03:19:57.737821+0000","last_fullsized":"2024-04-27T03:19:57.737821+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:47.724711+0000","last_change":"2024-04-27T03:17:39.629920+0000","last_active":"2024-04-27T03:19:47.724711+0000","last_peered":"2024-04-27T03:19:47.724711+0000","last_clean":"2024-04-27T03:19:47.724711+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:19:47.724711+0000","last_undegraded":"2024-04-27T03:19:47.724711+0000","last_fullsized":"2024-04-27T03:19:47.724711+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.178141+0000","last_change":"2024-04-27T03:21:35.178222+0000","last_active":"2024-04-27T03:21:35.178141+0000","last_peered":"2024-04-27T03:21:35.178141+0000","last_clean":"2024-04-27T03:21:35.178141+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:21:35.178141+0000","last_undegraded":"2024-04-27T03:21:35.178141+0000","last_fullsized":"2024-04-27T03:21:35.178141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:35.177942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:35.177942+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.177942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:50.842753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"31'22","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.688697+0000","last_change":"2024-04-27T03:21:29.688786+0000","last_active":"2024-04-27T03:21:29.688697+0000","last_peered":"2024-04-27T03:21:29.688697+0000","last_clean":"2024-04-27T03:21:29.688697+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:21:29.688697+0000","last_undegraded":"2024-04-27T03:21:29.688697+0000","last_fullsized":"2024-04-27T03:21:29.688697+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'22","last_scrub_stamp":"2024-04-27T03:21:29.688480+0000","last_deep_scrub":"31'22","last_deep_scrub_stamp":"2024-04-27T03:21:29.688480+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.688480+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:50:51.784245+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'285","reported_seq":429,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.765110+0000","last_change":"2024-04-27T03:21:36.765171+0000","last_active":"2024-04-27T03:21:36.765110+0000","last_peered":"2024-04-27T03:21:36.765110+0000","last_clean":"2024-04-27T03:21:36.765110+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:36.765110+0000","last_undegraded":"2024-04-27T03:21:36.765110+0000","last_fullsized":"2024-04-27T03:21:36.765110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:36.764897+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:36.764897+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.764897+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:07.152179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.731000+0000","last_change":"2024-04-27T03:17:57.087004+0000","last_active":"2024-04-27T03:20:12.731000+0000","last_peered":"2024-04-27T03:20:12.731000+0000","last_clean":"2024-04-27T03:20:12.731000+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:20:12.731000+0000","last_undegraded":"2024-04-27T03:20:12.731000+0000","last_fullsized":"2024-04-27T03:20:12.731000+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":112,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.791869+0000","last_change":"2024-04-27T03:21:44.791930+0000","last_active":"2024-04-27T03:21:44.791869+0000","last_peered":"2024-04-27T03:21:44.791869+0000","last_clean":"2024-04-27T03:21:44.791869+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:21:44.791869+0000","last_undegraded":"2024-04-27T03:21:44.791869+0000","last_fullsized":"2024-04-27T03:21:44.791869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:44.791685+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:44.791685+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.791685+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:12.652507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.158584+0000","last_change":"2024-04-27T03:17:47.000780+0000","last_active":"2024-04-27T03:20:00.158584+0000","last_peered":"2024-04-27T03:20:00.158584+0000","last_clean":"2024-04-27T03:20:00.158584+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:20:00.158584+0000","last_undegraded":"2024-04-27T03:20:00.158584+0000","last_fullsized":"2024-04-27T03:20:00.158584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.741186+0000","last_change":"2024-04-27T03:17:39.633493+0000","last_active":"2024-04-27T03:19:52.741186+0000","last_peered":"2024-04-27T03:19:52.741186+0000","last_clean":"2024-04-27T03:19:52.741186+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:19:52.741186+0000","last_undegraded":"2024-04-27T03:19:52.741186+0000","last_fullsized":"2024-04-27T03:19:52.741186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.600253+0000","last_change":"2024-04-27T03:21:19.600253+0000","last_active":"2024-04-27T03:21:19.600253+0000","last_peered":"2024-04-27T03:21:19.600253+0000","last_clean":"2024-04-27T03:21:19.600253+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:21:19.600253+0000","last_undegraded":"2024-04-27T03:21:19.600253+0000","last_fullsized":"2024-04-27T03:21:19.600253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.600004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:53:52.295521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'108","reported_seq":211,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.121243+0000","last_change":"2024-04-27T03:21:15.121323+0000","last_active":"2024-04-27T03:21:15.121243+0000","last_peered":"2024-04-27T03:21:15.121243+0000","last_clean":"2024-04-27T03:21:15.121243+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:15.121243+0000","last_undegraded":"2024-04-27T03:21:15.121243+0000","last_fullsized":"2024-04-27T03:21:15.121243+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'108","last_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_deep_scrub":"31'108","last_deep_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.121034+0000","objects_scrubbed":9,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:20.807083+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4316,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":107,"num_read_kb":100,"num_write":44,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.752988+0000","last_change":"2024-04-27T03:21:44.753050+0000","last_active":"2024-04-27T03:21:44.752988+0000","last_peered":"2024-04-27T03:21:44.752988+0000","last_clean":"2024-04-27T03:21:44.752988+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:21:44.752988+0000","last_undegraded":"2024-04-27T03:21:44.752988+0000","last_fullsized":"2024-04-27T03:21:44.752988+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:44.752732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:44.752732+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.752732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:27.679483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.824584+0000","last_change":"2024-04-27T03:21:12.824647+0000","last_active":"2024-04-27T03:21:12.824584+0000","last_peered":"2024-04-27T03:21:12.824584+0000","last_clean":"2024-04-27T03:21:12.824584+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:21:12.824584+0000","last_undegraded":"2024-04-27T03:21:12.824584+0000","last_fullsized":"2024-04-27T03:21:12.824584+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.824385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:28:25.571276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.779914+0000","last_change":"2024-04-27T03:21:14.779972+0000","last_active":"2024-04-27T03:21:14.779914+0000","last_peered":"2024-04-27T03:21:14.779914+0000","last_clean":"2024-04-27T03:21:14.779914+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:21:14.779914+0000","last_undegraded":"2024-04-27T03:21:14.779914+0000","last_fullsized":"2024-04-27T03:21:14.779914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.779705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:11:09.535832+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.696486+0000","last_change":"2024-04-27T03:21:36.696548+0000","last_active":"2024-04-27T03:21:36.696486+0000","last_peered":"2024-04-27T03:21:36.696486+0000","last_clean":"2024-04-27T03:21:36.696486+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:36.696486+0000","last_undegraded":"2024-04-27T03:21:36.696486+0000","last_fullsized":"2024-04-27T03:21:36.696486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:36.696292+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:36.696292+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.696292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:12:08.368797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.132425+0000","last_change":"2024-04-27T03:17:47.023352+0000","last_active":"2024-04-27T03:21:01.132425+0000","last_peered":"2024-04-27T03:21:01.132425+0000","last_clean":"2024-04-27T03:21:01.132425+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:01.132425+0000","last_undegraded":"2024-04-27T03:21:01.132425+0000","last_fullsized":"2024-04-27T03:21:01.132425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"31'2","reported_seq":234,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.286650+0000","last_change":"2024-04-27T03:21:26.286722+0000","last_active":"2024-04-27T03:21:26.286650+0000","last_peered":"2024-04-27T03:21:26.286650+0000","last_clean":"2024-04-27T03:21:26.286650+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:21:26.286650+0000","last_undegraded":"2024-04-27T03:21:26.286650+0000","last_fullsized":"2024-04-27T03:21:26.286650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.286462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:36:54.147283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":285,"num_read_kb":285,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'46","reported_seq":112,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.542394+0000","last_change":"2024-04-27T03:21:15.542505+0000","last_active":"2024-04-27T03:21:15.542394+0000","last_peered":"2024-04-27T03:21:15.542394+0000","last_clean":"2024-04-27T03:21:15.542394+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:21:15.542394+0000","last_undegraded":"2024-04-27T03:21:15.542394+0000","last_fullsized":"2024-04-27T03:21:15.542394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'46","last_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_deep_scrub":"31'46","last_deep_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.542196+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:31.074928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.640360+0000","last_change":"2024-04-27T03:21:28.640422+0000","last_active":"2024-04-27T03:21:28.640360+0000","last_peered":"2024-04-27T03:21:28.640360+0000","last_clean":"2024-04-27T03:21:28.640360+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:21:28.640360+0000","last_undegraded":"2024-04-27T03:21:28.640360+0000","last_fullsized":"2024-04-27T03:21:28.640360+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:28.640146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:28.640146+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.640146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:20.934477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.887628+0000","last_change":"2024-04-27T03:17:57.087180+0000","last_active":"2024-04-27T03:19:57.887628+0000","last_peered":"2024-04-27T03:19:57.887628+0000","last_clean":"2024-04-27T03:19:57.887628+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:19:57.887628+0000","last_undegraded":"2024-04-27T03:19:57.887628+0000","last_fullsized":"2024-04-27T03:19:57.887628+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.536766+0000","last_change":"2024-04-27T03:21:52.536831+0000","last_active":"2024-04-27T03:21:52.536766+0000","last_peered":"2024-04-27T03:21:52.536766+0000","last_clean":"2024-04-27T03:21:52.536766+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:52.536766+0000","last_undegraded":"2024-04-27T03:21:52.536766+0000","last_fullsized":"2024-04-27T03:21:52.536766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:52.536569+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:52.536569+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.536569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:49.821834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.725275+0000","last_change":"2024-04-27T03:21:52.725328+0000","last_active":"2024-04-27T03:21:52.725275+0000","last_peered":"2024-04-27T03:21:52.725275+0000","last_clean":"2024-04-27T03:21:52.725275+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:21:52.725275+0000","last_undegraded":"2024-04-27T03:21:52.725275+0000","last_fullsized":"2024-04-27T03:21:52.725275+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:52.725071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:52.725071+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.725071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:49:42.890358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.749861+0000","last_change":"2024-04-27T03:21:35.749914+0000","last_active":"2024-04-27T03:21:35.749861+0000","last_peered":"2024-04-27T03:21:35.749861+0000","last_clean":"2024-04-27T03:21:35.749861+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:21:35.749861+0000","last_undegraded":"2024-04-27T03:21:35.749861+0000","last_fullsized":"2024-04-27T03:21:35.749861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:35.749664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:35.749664+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.749664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:49:21.865491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.185318+0000","last_change":"2024-04-27T03:21:19.185455+0000","last_active":"2024-04-27T03:21:19.185318+0000","last_peered":"2024-04-27T03:21:19.185318+0000","last_clean":"2024-04-27T03:21:19.185318+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:19.185318+0000","last_undegraded":"2024-04-27T03:21:19.185318+0000","last_fullsized":"2024-04-27T03:21:19.185318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.185100+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:12.343058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":250,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.823643+0000","last_change":"2024-04-27T03:21:24.823694+0000","last_active":"2024-04-27T03:21:24.823643+0000","last_peered":"2024-04-27T03:21:24.823643+0000","last_clean":"2024-04-27T03:21:24.823643+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:21:24.823643+0000","last_undegraded":"2024-04-27T03:21:24.823643+0000","last_fullsized":"2024-04-27T03:21:24.823643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.823449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:10:09.300634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.734550+0000","last_change":"2024-04-27T03:17:57.089220+0000","last_active":"2024-04-27T03:20:27.734550+0000","last_peered":"2024-04-27T03:20:27.734550+0000","last_clean":"2024-04-27T03:20:27.734550+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:20:27.734550+0000","last_undegraded":"2024-04-27T03:20:27.734550+0000","last_fullsized":"2024-04-27T03:20:27.734550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.964257+0000","last_change":"2024-04-27T03:18:34.799245+0000","last_active":"2024-04-27T03:21:00.964257+0000","last_peered":"2024-04-27T03:21:00.964257+0000","last_clean":"2024-04-27T03:21:00.964257+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:21:00.964257+0000","last_undegraded":"2024-04-27T03:21:00.964257+0000","last_fullsized":"2024-04-27T03:21:00.964257+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.115332+0000","last_change":"2024-04-27T03:17:47.012224+0000","last_active":"2024-04-27T03:21:01.115332+0000","last_peered":"2024-04-27T03:21:01.115332+0000","last_clean":"2024-04-27T03:21:01.115332+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:21:01.115332+0000","last_undegraded":"2024-04-27T03:21:01.115332+0000","last_fullsized":"2024-04-27T03:21:01.115332+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.096198+0000","last_change":"2024-04-27T03:21:12.096234+0000","last_active":"2024-04-27T03:21:12.096198+0000","last_peered":"2024-04-27T03:21:12.096198+0000","last_clean":"2024-04-27T03:21:12.096198+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:21:12.096198+0000","last_undegraded":"2024-04-27T03:21:12.096198+0000","last_fullsized":"2024-04-27T03:21:12.096198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.096079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:20.095840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'47","reported_seq":123,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.666937+0000","last_change":"2024-04-27T03:21:30.667010+0000","last_active":"2024-04-27T03:21:30.666937+0000","last_peered":"2024-04-27T03:21:30.666937+0000","last_clean":"2024-04-27T03:21:30.666937+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:30.666937+0000","last_undegraded":"2024-04-27T03:21:30.666937+0000","last_fullsized":"2024-04-27T03:21:30.666937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-04-27T03:21:30.666704+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-04-27T03:21:30.666704+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.666704+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:05.358118+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":67,"num_read_kb":70,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":662,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.199623+0000","last_change":"2024-04-27T03:21:20.199682+0000","last_active":"2024-04-27T03:21:20.199623+0000","last_peered":"2024-04-27T03:21:20.199623+0000","last_clean":"2024-04-27T03:21:20.199623+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:21:20.199623+0000","last_undegraded":"2024-04-27T03:21:20.199623+0000","last_fullsized":"2024-04-27T03:21:20.199623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.199413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:10:57.834556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.745960+0000","last_change":"2024-04-27T03:17:57.086967+0000","last_active":"2024-04-27T03:20:27.745960+0000","last_peered":"2024-04-27T03:20:27.745960+0000","last_clean":"2024-04-27T03:20:27.745960+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:20:27.745960+0000","last_undegraded":"2024-04-27T03:20:27.745960+0000","last_fullsized":"2024-04-27T03:20:27.745960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.247672+0000","last_change":"2024-04-27T03:17:49.014835+0000","last_active":"2024-04-27T03:21:01.247672+0000","last_peered":"2024-04-27T03:21:01.247672+0000","last_clean":"2024-04-27T03:21:01.247672+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:01.247672+0000","last_undegraded":"2024-04-27T03:21:01.247672+0000","last_fullsized":"2024-04-27T03:21:01.247672+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.731154+0000","last_change":"2024-04-27T03:17:47.026622+0000","last_active":"2024-04-27T03:20:12.731154+0000","last_peered":"2024-04-27T03:20:12.731154+0000","last_clean":"2024-04-27T03:20:12.731154+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:20:12.731154+0000","last_undegraded":"2024-04-27T03:20:12.731154+0000","last_fullsized":"2024-04-27T03:20:12.731154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.813369+0000","last_change":"2024-04-27T03:21:37.813424+0000","last_active":"2024-04-27T03:21:37.813369+0000","last_peered":"2024-04-27T03:21:37.813369+0000","last_clean":"2024-04-27T03:21:37.813369+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:21:37.813369+0000","last_undegraded":"2024-04-27T03:21:37.813369+0000","last_fullsized":"2024-04-27T03:21:37.813369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:37.813166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:37.813166+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.813166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:26.811307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'37","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.159109+0000","last_change":"2024-04-27T03:21:17.159218+0000","last_active":"2024-04-27T03:21:17.159109+0000","last_peered":"2024-04-27T03:21:17.159109+0000","last_clean":"2024-04-27T03:21:17.159109+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:21:17.159109+0000","last_undegraded":"2024-04-27T03:21:17.159109+0000","last_fullsized":"2024-04-27T03:21:17.159109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.158898+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:06.103710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.871801+0000","last_change":"2024-04-27T03:21:23.871852+0000","last_active":"2024-04-27T03:21:23.871801+0000","last_peered":"2024-04-27T03:21:23.871801+0000","last_clean":"2024-04-27T03:21:23.871801+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:21:23.871801+0000","last_undegraded":"2024-04-27T03:21:23.871801+0000","last_fullsized":"2024-04-27T03:21:23.871801+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.871608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:14.918299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'2","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.137057+0000","last_change":"2024-04-27T03:21:39.137118+0000","last_active":"2024-04-27T03:21:39.137057+0000","last_peered":"2024-04-27T03:21:39.137057+0000","last_clean":"2024-04-27T03:21:39.137057+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:21:39.137057+0000","last_undegraded":"2024-04-27T03:21:39.137057+0000","last_fullsized":"2024-04-27T03:21:39.137057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:39.136858+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:39.136858+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.136858+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:15.417970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.124071+0000","last_change":"2024-04-27T03:17:47.032082+0000","last_active":"2024-04-27T03:21:01.124071+0000","last_peered":"2024-04-27T03:21:01.124071+0000","last_clean":"2024-04-27T03:21:01.124071+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:21:01.124071+0000","last_undegraded":"2024-04-27T03:21:01.124071+0000","last_fullsized":"2024-04-27T03:21:01.124071+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'5","reported_seq":243,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.496951+0000","last_change":"2024-04-27T03:21:33.496951+0000","last_active":"2024-04-27T03:21:33.496951+0000","last_peered":"2024-04-27T03:21:33.496951+0000","last_clean":"2024-04-27T03:21:33.496951+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:21:33.496951+0000","last_undegraded":"2024-04-27T03:21:33.496951+0000","last_fullsized":"2024-04-27T03:21:33.496951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:21:33.496514+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:21:33.496514+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.496514+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:17:35.575680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":301,"num_read_kb":301,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'72","reported_seq":155,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.270317+0000","last_change":"2024-04-27T03:21:23.270391+0000","last_active":"2024-04-27T03:21:23.270317+0000","last_peered":"2024-04-27T03:21:23.270317+0000","last_clean":"2024-04-27T03:21:23.270317+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:21:23.270317+0000","last_undegraded":"2024-04-27T03:21:23.270317+0000","last_fullsized":"2024-04-27T03:21:23.270317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'72","last_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_deep_scrub":"31'72","last_deep_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.270178+0000","objects_scrubbed":6,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:28.956226+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":28,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.743462+0000","last_change":"2024-04-27T03:21:16.743518+0000","last_active":"2024-04-27T03:21:16.743462+0000","last_peered":"2024-04-27T03:21:16.743462+0000","last_clean":"2024-04-27T03:21:16.743462+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:21:16.743462+0000","last_undegraded":"2024-04-27T03:21:16.743462+0000","last_fullsized":"2024-04-27T03:21:16.743462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.743267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:34:18.284825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.724167+0000","last_change":"2024-04-27T03:21:48.724227+0000","last_active":"2024-04-27T03:21:48.724167+0000","last_peered":"2024-04-27T03:21:48.724167+0000","last_clean":"2024-04-27T03:21:48.724167+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:21:48.724167+0000","last_undegraded":"2024-04-27T03:21:48.724167+0000","last_fullsized":"2024-04-27T03:21:48.724167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:48.723971+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:48.723971+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.723971+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:58.946837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.540808+0000","last_change":"2024-04-27T03:21:46.540808+0000","last_active":"2024-04-27T03:21:46.540808+0000","last_peered":"2024-04-27T03:21:46.540808+0000","last_clean":"2024-04-27T03:21:46.540808+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:21:46.540808+0000","last_undegraded":"2024-04-27T03:21:46.540808+0000","last_fullsized":"2024-04-27T03:21:46.540808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:46.540543+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:46.540543+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.540543+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:16.927826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.730774+0000","last_change":"2024-04-27T03:17:47.021864+0000","last_active":"2024-04-27T03:20:12.730774+0000","last_peered":"2024-04-27T03:20:12.730774+0000","last_clean":"2024-04-27T03:20:12.730774+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:20:12.730774+0000","last_undegraded":"2024-04-27T03:20:12.730774+0000","last_fullsized":"2024-04-27T03:20:12.730774+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.193746+0000","last_change":"2024-04-27T03:21:20.193806+0000","last_active":"2024-04-27T03:21:20.193746+0000","last_peered":"2024-04-27T03:21:20.193746+0000","last_clean":"2024-04-27T03:21:20.193746+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:21:20.193746+0000","last_undegraded":"2024-04-27T03:21:20.193746+0000","last_fullsized":"2024-04-27T03:21:20.193746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.193546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:27:42.782149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'45","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.064594+0000","last_change":"2024-04-27T03:21:13.064640+0000","last_active":"2024-04-27T03:21:13.064594+0000","last_peered":"2024-04-27T03:21:13.064594+0000","last_clean":"2024-04-27T03:21:13.064594+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:21:13.064594+0000","last_undegraded":"2024-04-27T03:21:13.064594+0000","last_fullsized":"2024-04-27T03:21:13.064594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.064511+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:06:47.923690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.302195+0000","last_change":"2024-04-27T03:21:14.302256+0000","last_active":"2024-04-27T03:21:14.302195+0000","last_peered":"2024-04-27T03:21:14.302195+0000","last_clean":"2024-04-27T03:21:14.302195+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:21:14.302195+0000","last_undegraded":"2024-04-27T03:21:14.302195+0000","last_fullsized":"2024-04-27T03:21:14.302195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.302007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:00.292019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747121+0000","last_change":"2024-04-27T03:17:57.070067+0000","last_active":"2024-04-27T03:20:17.747121+0000","last_peered":"2024-04-27T03:20:17.747121+0000","last_clean":"2024-04-27T03:20:17.747121+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:20:17.747121+0000","last_undegraded":"2024-04-27T03:20:17.747121+0000","last_fullsized":"2024-04-27T03:20:17.747121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.744728+0000","last_change":"2024-04-27T03:17:49.004464+0000","last_active":"2024-04-27T03:20:22.744728+0000","last_peered":"2024-04-27T03:20:22.744728+0000","last_clean":"2024-04-27T03:20:22.744728+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:20:22.744728+0000","last_undegraded":"2024-04-27T03:20:22.744728+0000","last_fullsized":"2024-04-27T03:20:22.744728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747160+0000","last_change":"2024-04-27T03:17:47.027156+0000","last_active":"2024-04-27T03:20:17.747160+0000","last_peered":"2024-04-27T03:20:17.747160+0000","last_clean":"2024-04-27T03:20:17.747160+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:20:17.747160+0000","last_undegraded":"2024-04-27T03:20:17.747160+0000","last_fullsized":"2024-04-27T03:20:17.747160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.231515+0000","last_change":"2024-04-27T03:21:22.231634+0000","last_active":"2024-04-27T03:21:22.231515+0000","last_peered":"2024-04-27T03:21:22.231515+0000","last_clean":"2024-04-27T03:21:22.231515+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:21:22.231515+0000","last_undegraded":"2024-04-27T03:21:22.231515+0000","last_fullsized":"2024-04-27T03:21:22.231515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.231282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:38.450738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'47","reported_seq":113,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.572445+0000","last_change":"2024-04-27T03:21:16.572535+0000","last_active":"2024-04-27T03:21:16.572445+0000","last_peered":"2024-04-27T03:21:16.572445+0000","last_clean":"2024-04-27T03:21:16.572445+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:21:16.572445+0000","last_undegraded":"2024-04-27T03:21:16.572445+0000","last_fullsized":"2024-04-27T03:21:16.572445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.572262+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:05.517074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.304067+0000","last_change":"2024-04-27T03:21:28.304119+0000","last_active":"2024-04-27T03:21:28.304067+0000","last_peered":"2024-04-27T03:21:28.304067+0000","last_clean":"2024-04-27T03:21:28.304067+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:21:28.304067+0000","last_undegraded":"2024-04-27T03:21:28.304067+0000","last_fullsized":"2024-04-27T03:21:28.304067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:28.303861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:28.303861+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.303861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:26.889521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747057+0000","last_change":"2024-04-27T03:17:57.066218+0000","last_active":"2024-04-27T03:20:17.747057+0000","last_peered":"2024-04-27T03:20:17.747057+0000","last_clean":"2024-04-27T03:20:17.747057+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:20:17.747057+0000","last_undegraded":"2024-04-27T03:20:17.747057+0000","last_fullsized":"2024-04-27T03:20:17.747057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.698657+0000","last_change":"2024-04-27T03:21:35.698721+0000","last_active":"2024-04-27T03:21:35.698657+0000","last_peered":"2024-04-27T03:21:35.698657+0000","last_clean":"2024-04-27T03:21:35.698657+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:21:35.698657+0000","last_undegraded":"2024-04-27T03:21:35.698657+0000","last_fullsized":"2024-04-27T03:21:35.698657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:35.698441+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:35.698441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.698441+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:43.033392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.743118+0000","last_change":"2024-04-27T03:17:47.018817+0000","last_active":"2024-04-27T03:20:17.743118+0000","last_peered":"2024-04-27T03:20:17.743118+0000","last_clean":"2024-04-27T03:20:17.743118+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:20:17.743118+0000","last_undegraded":"2024-04-27T03:20:17.743118+0000","last_fullsized":"2024-04-27T03:20:17.743118+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.301687+0000","last_change":"2024-04-27T03:21:27.301740+0000","last_active":"2024-04-27T03:21:27.301687+0000","last_peered":"2024-04-27T03:21:27.301687+0000","last_clean":"2024-04-27T03:21:27.301687+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:21:27.301687+0000","last_undegraded":"2024-04-27T03:21:27.301687+0000","last_fullsized":"2024-04-27T03:21:27.301687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.301481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:33.928268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"31'22","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.677168+0000","last_change":"2024-04-27T03:21:31.677168+0000","last_active":"2024-04-27T03:21:31.677168+0000","last_peered":"2024-04-27T03:21:31.677168+0000","last_clean":"2024-04-27T03:21:31.677168+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:21:31.677168+0000","last_undegraded":"2024-04-27T03:21:31.677168+0000","last_fullsized":"2024-04-27T03:21:31.677168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'22","last_scrub_stamp":"2024-04-27T03:21:31.676892+0000","last_deep_scrub":"31'22","last_deep_scrub_stamp":"2024-04-27T03:21:31.676892+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.676892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:19.196626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'285","reported_seq":392,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.185100+0000","last_change":"2024-04-27T03:21:25.185168+0000","last_active":"2024-04-27T03:21:25.185100+0000","last_peered":"2024-04-27T03:21:25.185100+0000","last_clean":"2024-04-27T03:21:25.185100+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:25.185100+0000","last_undegraded":"2024-04-27T03:21:25.185100+0000","last_fullsized":"2024-04-27T03:21:25.185100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.184906+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:35.796002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.746971+0000","last_change":"2024-04-27T03:17:57.084816+0000","last_active":"2024-04-27T03:20:17.746971+0000","last_peered":"2024-04-27T03:20:17.746971+0000","last_clean":"2024-04-27T03:20:17.746971+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:20:17.746971+0000","last_undegraded":"2024-04-27T03:20:17.746971+0000","last_fullsized":"2024-04-27T03:20:17.746971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.686144+0000","last_change":"2024-04-27T03:21:32.686210+0000","last_active":"2024-04-27T03:21:32.686144+0000","last_peered":"2024-04-27T03:21:32.686144+0000","last_clean":"2024-04-27T03:21:32.686144+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:32.686144+0000","last_undegraded":"2024-04-27T03:21:32.686144+0000","last_fullsized":"2024-04-27T03:21:32.686144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:32.685934+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:32.685934+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.685934+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:18.675946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.498152+0000","last_change":"2024-04-27T03:21:51.498216+0000","last_active":"2024-04-27T03:21:51.498152+0000","last_peered":"2024-04-27T03:21:51.498152+0000","last_clean":"2024-04-27T03:21:51.498152+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:21:51.498152+0000","last_undegraded":"2024-04-27T03:21:51.498152+0000","last_fullsized":"2024-04-27T03:21:51.498152+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'4","last_scrub_stamp":"2024-04-27T03:21:51.497953+0000","last_deep_scrub":"27'4","last_deep_scrub_stamp":"2024-04-27T03:21:51.497953+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.497953+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:38.452327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.332423+0000","last_change":"2024-04-27T03:21:13.332476+0000","last_active":"2024-04-27T03:21:13.332423+0000","last_peered":"2024-04-27T03:21:13.332423+0000","last_clean":"2024-04-27T03:21:13.332423+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:21:13.332423+0000","last_undegraded":"2024-04-27T03:21:13.332423+0000","last_fullsized":"2024-04-27T03:21:13.332423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.332213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:31.014208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'73","reported_seq":159,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.145202+0000","last_change":"2024-04-27T03:21:32.145296+0000","last_active":"2024-04-27T03:21:32.145202+0000","last_peered":"2024-04-27T03:21:32.145202+0000","last_clean":"2024-04-27T03:21:32.145202+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:21:32.145202+0000","last_undegraded":"2024-04-27T03:21:32.145202+0000","last_fullsized":"2024-04-27T03:21:32.145202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'73","last_scrub_stamp":"2024-04-27T03:21:32.145018+0000","last_deep_scrub":"31'73","last_deep_scrub_stamp":"2024-04-27T03:21:32.145018+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.145018+0000","objects_scrubbed":11,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:06.836432+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":44,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.654794+0000","last_change":"2024-04-27T03:21:26.654845+0000","last_active":"2024-04-27T03:21:26.654794+0000","last_peered":"2024-04-27T03:21:26.654794+0000","last_clean":"2024-04-27T03:21:26.654794+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:21:26.654794+0000","last_undegraded":"2024-04-27T03:21:26.654794+0000","last_fullsized":"2024-04-27T03:21:26.654794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.654609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:01.305349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.819801+0000","last_change":"2024-04-27T03:17:57.066613+0000","last_active":"2024-04-27T03:21:00.819801+0000","last_peered":"2024-04-27T03:21:00.819801+0000","last_clean":"2024-04-27T03:21:00.819801+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:21:00.819801+0000","last_undegraded":"2024-04-27T03:21:00.819801+0000","last_fullsized":"2024-04-27T03:21:00.819801+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.195754+0000","last_change":"2024-04-27T03:21:44.195813+0000","last_active":"2024-04-27T03:21:44.195754+0000","last_peered":"2024-04-27T03:21:44.195754+0000","last_clean":"2024-04-27T03:21:44.195754+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:44.195754+0000","last_undegraded":"2024-04-27T03:21:44.195754+0000","last_fullsized":"2024-04-27T03:21:44.195754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:44.195561+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:44.195561+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.195561+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:31.149936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747314+0000","last_change":"2024-04-27T03:17:47.022941+0000","last_active":"2024-04-27T03:20:17.747314+0000","last_peered":"2024-04-27T03:20:17.747314+0000","last_clean":"2024-04-27T03:20:17.747314+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:20:17.747314+0000","last_undegraded":"2024-04-27T03:20:17.747314+0000","last_fullsized":"2024-04-27T03:20:17.747314+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.156966+0000","last_change":"2024-04-27T03:21:19.157032+0000","last_active":"2024-04-27T03:21:19.156966+0000","last_peered":"2024-04-27T03:21:19.156966+0000","last_clean":"2024-04-27T03:21:19.156966+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:21:19.156966+0000","last_undegraded":"2024-04-27T03:21:19.156966+0000","last_fullsized":"2024-04-27T03:21:19.156966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.156758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:49.521232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'45","reported_seq":110,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.820346+0000","last_change":"2024-04-27T03:21:27.820426+0000","last_active":"2024-04-27T03:21:27.820346+0000","last_peered":"2024-04-27T03:21:27.820346+0000","last_clean":"2024-04-27T03:21:27.820346+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:21:27.820346+0000","last_undegraded":"2024-04-27T03:21:27.820346+0000","last_fullsized":"2024-04-27T03:21:27.820346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:27.820155+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:27.820155+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.820155+0000","objects_scrubbed":5,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:39:02.307136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.175376+0000","last_change":"2024-04-27T03:21:24.175430+0000","last_active":"2024-04-27T03:21:24.175376+0000","last_peered":"2024-04-27T03:21:24.175376+0000","last_clean":"2024-04-27T03:21:24.175376+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:21:24.175376+0000","last_undegraded":"2024-04-27T03:21:24.175376+0000","last_fullsized":"2024-04-27T03:21:24.175376+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.175164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:23.877311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.480429+0000","last_change":"2024-04-27T03:17:57.087706+0000","last_active":"2024-04-27T03:21:01.480429+0000","last_peered":"2024-04-27T03:21:01.480429+0000","last_clean":"2024-04-27T03:21:01.480429+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:21:01.480429+0000","last_undegraded":"2024-04-27T03:21:01.480429+0000","last_fullsized":"2024-04-27T03:21:01.480429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.962890+0000","last_change":"2024-04-27T03:17:49.011902+0000","last_active":"2024-04-27T03:21:00.962890+0000","last_peered":"2024-04-27T03:21:00.962890+0000","last_clean":"2024-04-27T03:21:00.962890+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:21:00.962890+0000","last_undegraded":"2024-04-27T03:21:00.962890+0000","last_fullsized":"2024-04-27T03:21:00.962890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.747368+0000","last_change":"2024-04-27T03:17:47.016344+0000","last_active":"2024-04-27T03:21:12.747368+0000","last_peered":"2024-04-27T03:21:12.747368+0000","last_clean":"2024-04-27T03:21:12.747368+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:21:12.747368+0000","last_undegraded":"2024-04-27T03:21:12.747368+0000","last_fullsized":"2024-04-27T03:21:12.747368+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.336891+0000","last_change":"2024-04-27T03:21:29.336949+0000","last_active":"2024-04-27T03:21:29.336891+0000","last_peered":"2024-04-27T03:21:29.336891+0000","last_clean":"2024-04-27T03:21:29.336891+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:21:29.336891+0000","last_undegraded":"2024-04-27T03:21:29.336891+0000","last_fullsized":"2024-04-27T03:21:29.336891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:29.336698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:29.336698+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.336698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:15:11.724434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'37","reported_seq":100,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.538736+0000","last_change":"2024-04-27T03:21:14.538808+0000","last_active":"2024-04-27T03:21:14.538736+0000","last_peered":"2024-04-27T03:21:14.538736+0000","last_clean":"2024-04-27T03:21:14.538736+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:21:14.538736+0000","last_undegraded":"2024-04-27T03:21:14.538736+0000","last_fullsized":"2024-04-27T03:21:14.538736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.538542+0000","objects_scrubbed":3,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:32.436126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.189216+0000","last_change":"2024-04-27T03:21:41.189268+0000","last_active":"2024-04-27T03:21:41.189216+0000","last_peered":"2024-04-27T03:21:41.189216+0000","last_clean":"2024-04-27T03:21:41.189216+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:21:41.189216+0000","last_undegraded":"2024-04-27T03:21:41.189216+0000","last_fullsized":"2024-04-27T03:21:41.189216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:41.189019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:41.189019+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.189019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:28.342309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.744683+0000","last_change":"2024-04-27T03:17:57.086785+0000","last_active":"2024-04-27T03:20:22.744683+0000","last_peered":"2024-04-27T03:20:22.744683+0000","last_clean":"2024-04-27T03:20:22.744683+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:20:22.744683+0000","last_undegraded":"2024-04-27T03:20:22.744683+0000","last_fullsized":"2024-04-27T03:20:22.744683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'16","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.184766+0000","last_change":"2024-04-27T03:21:51.184831+0000","last_active":"2024-04-27T03:21:51.184766+0000","last_peered":"2024-04-27T03:21:51.184766+0000","last_clean":"2024-04-27T03:21:51.184766+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:51.184766+0000","last_undegraded":"2024-04-27T03:21:51.184766+0000","last_fullsized":"2024-04-27T03:21:51.184766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:51.184588+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:51.184588+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.184588+0000","objects_scrubbed":2,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:01.407454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:00.026513+0000","last_change":"2024-04-27T03:17:47.002631+0000","last_active":"2024-04-27T03:20:00.026513+0000","last_peered":"2024-04-27T03:20:00.026513+0000","last_clean":"2024-04-27T03:20:00.026513+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:20:00.026513+0000","last_undegraded":"2024-04-27T03:20:00.026513+0000","last_fullsized":"2024-04-27T03:20:00.026513+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'2","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:02.747502+0000","last_change":"2024-04-27T03:18:34.794992+0000","last_active":"2024-04-27T03:21:02.747502+0000","last_peered":"2024-04-27T03:21:02.747502+0000","last_clean":"2024-04-27T03:21:02.747502+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:21:02.747502+0000","last_undegraded":"2024-04-27T03:21:02.747502+0000","last_fullsized":"2024-04-27T03:21:02.747502+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:49.992337+0000","last_clean_scrub_stamp":"2024-04-27T03:17:49.992337+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'53","reported_seq":126,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.140107+0000","last_change":"2024-04-27T03:21:34.140183+0000","last_active":"2024-04-27T03:21:34.140107+0000","last_peered":"2024-04-27T03:21:34.140107+0000","last_clean":"2024-04-27T03:21:34.140107+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:34.140107+0000","last_undegraded":"2024-04-27T03:21:34.140107+0000","last_fullsized":"2024-04-27T03:21:34.140107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'53","last_scrub_stamp":"2024-04-27T03:21:34.139891+0000","last_deep_scrub":"31'53","last_deep_scrub_stamp":"2024-04-27T03:21:34.139891+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.139891+0000","objects_scrubbed":7,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:44.594383+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":52,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1138,"num_omap_keys":6,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.822080+0000","last_change":"2024-04-27T03:21:19.822219+0000","last_active":"2024-04-27T03:21:19.822080+0000","last_peered":"2024-04-27T03:21:19.822080+0000","last_clean":"2024-04-27T03:21:19.822080+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:21:19.822080+0000","last_undegraded":"2024-04-27T03:21:19.822080+0000","last_fullsized":"2024-04-27T03:21:19.822080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.821853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:18.101674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747256+0000","last_change":"2024-04-27T03:17:57.061982+0000","last_active":"2024-04-27T03:20:17.747256+0000","last_peered":"2024-04-27T03:20:17.747256+0000","last_clean":"2024-04-27T03:20:17.747256+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:20:17.747256+0000","last_undegraded":"2024-04-27T03:20:17.747256+0000","last_fullsized":"2024-04-27T03:20:17.747256+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.475609+0000","last_change":"2024-04-27T03:21:48.475670+0000","last_active":"2024-04-27T03:21:48.475609+0000","last_peered":"2024-04-27T03:21:48.475609+0000","last_clean":"2024-04-27T03:21:48.475609+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:21:48.475609+0000","last_undegraded":"2024-04-27T03:21:48.475609+0000","last_fullsized":"2024-04-27T03:21:48.475609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:48.475413+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.475413+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.475413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:40:05.549233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.748688+0000","last_change":"2024-04-27T03:21:46.748753+0000","last_active":"2024-04-27T03:21:46.748688+0000","last_peered":"2024-04-27T03:21:46.748688+0000","last_clean":"2024-04-27T03:21:46.748688+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:21:46.748688+0000","last_undegraded":"2024-04-27T03:21:46.748688+0000","last_fullsized":"2024-04-27T03:21:46.748688+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:46.748445+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:46.748445+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.748445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:45.334105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.624237+0000","last_change":"2024-04-27T03:21:41.624365+0000","last_active":"2024-04-27T03:21:41.624237+0000","last_peered":"2024-04-27T03:21:41.624237+0000","last_clean":"2024-04-27T03:21:41.624237+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:21:41.624237+0000","last_undegraded":"2024-04-27T03:21:41.624237+0000","last_fullsized":"2024-04-27T03:21:41.624237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:41.624007+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:41.624007+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.624007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:52.078499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'2","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.988329+0000","last_change":"2024-04-27T03:18:34.773740+0000","last_active":"2024-04-27T03:21:07.988329+0000","last_peered":"2024-04-27T03:21:07.988329+0000","last_clean":"2024-04-27T03:21:07.988329+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:21:07.988329+0000","last_undegraded":"2024-04-27T03:21:07.988329+0000","last_fullsized":"2024-04-27T03:21:07.988329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:47.744454+0000","last_change":"2024-04-27T03:18:34.784905+0000","last_active":"2024-04-27T03:20:47.744454+0000","last_peered":"2024-04-27T03:20:47.744454+0000","last_clean":"2024-04-27T03:20:47.744454+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:20:47.744454+0000","last_undegraded":"2024-04-27T03:20:47.744454+0000","last_fullsized":"2024-04-27T03:20:47.744454+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:52.736562+0000","last_change":"2024-04-27T03:17:47.036148+0000","last_active":"2024-04-27T03:19:52.736562+0000","last_peered":"2024-04-27T03:19:52.736562+0000","last_clean":"2024-04-27T03:19:52.736562+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:19:52.736562+0000","last_undegraded":"2024-04-27T03:19:52.736562+0000","last_fullsized":"2024-04-27T03:19:52.736562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.518423+0000","last_change":"2024-04-27T03:21:35.518519+0000","last_active":"2024-04-27T03:21:35.518423+0000","last_peered":"2024-04-27T03:21:35.518423+0000","last_clean":"2024-04-27T03:21:35.518423+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:21:35.518423+0000","last_undegraded":"2024-04-27T03:21:35.518423+0000","last_fullsized":"2024-04-27T03:21:35.518423+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'6","last_scrub_stamp":"2024-04-27T03:21:35.518220+0000","last_deep_scrub":"27'6","last_deep_scrub_stamp":"2024-04-27T03:21:35.518220+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.518220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:50.091666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.732926+0000","last_change":"2024-04-27T03:21:49.732999+0000","last_active":"2024-04-27T03:21:49.732926+0000","last_peered":"2024-04-27T03:21:49.732926+0000","last_clean":"2024-04-27T03:21:49.732926+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:21:49.732926+0000","last_undegraded":"2024-04-27T03:21:49.732926+0000","last_fullsized":"2024-04-27T03:21:49.732926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'4","last_scrub_stamp":"2024-04-27T03:21:49.732701+0000","last_deep_scrub":"29'4","last_deep_scrub_stamp":"2024-04-27T03:21:49.732701+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.732701+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:41.379770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.819779+0000","last_change":"2024-04-27T03:21:42.819841+0000","last_active":"2024-04-27T03:21:42.819779+0000","last_peered":"2024-04-27T03:21:42.819779+0000","last_clean":"2024-04-27T03:21:42.819779+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:42.819779+0000","last_undegraded":"2024-04-27T03:21:42.819779+0000","last_fullsized":"2024-04-27T03:21:42.819779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:42.819561+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:42.819561+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.819561+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:40.104826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737696+0000","last_change":"2024-04-27T03:17:47.039423+0000","last_active":"2024-04-27T03:19:57.737696+0000","last_peered":"2024-04-27T03:19:57.737696+0000","last_clean":"2024-04-27T03:19:57.737696+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:19:57.737696+0000","last_undegraded":"2024-04-27T03:19:57.737696+0000","last_fullsized":"2024-04-27T03:19:57.737696+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.726869+0000","last_change":"2024-04-27T03:17:49.006801+0000","last_active":"2024-04-27T03:19:57.726869+0000","last_peered":"2024-04-27T03:19:57.726869+0000","last_clean":"2024-04-27T03:19:57.726869+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:19:57.726869+0000","last_undegraded":"2024-04-27T03:19:57.726869+0000","last_fullsized":"2024-04-27T03:19:57.726869+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.514937+0000","last_change":"2024-04-27T03:21:47.515012+0000","last_active":"2024-04-27T03:21:47.514937+0000","last_peered":"2024-04-27T03:21:47.514937+0000","last_clean":"2024-04-27T03:21:47.514937+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:21:47.514937+0000","last_undegraded":"2024-04-27T03:21:47.514937+0000","last_fullsized":"2024-04-27T03:21:47.514937+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:47.514720+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:47.514720+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.514720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:36.512861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.711298+0000","last_change":"2024-04-27T03:21:50.711428+0000","last_active":"2024-04-27T03:21:50.711298+0000","last_peered":"2024-04-27T03:21:50.711298+0000","last_clean":"2024-04-27T03:21:50.711298+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:50.711298+0000","last_undegraded":"2024-04-27T03:21:50.711298+0000","last_fullsized":"2024-04-27T03:21:50.711298+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:50.710962+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:50.710962+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.710962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:53.828224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:07.987179+0000","last_change":"2024-04-27T03:17:47.031857+0000","last_active":"2024-04-27T03:21:07.987179+0000","last_peered":"2024-04-27T03:21:07.987179+0000","last_clean":"2024-04-27T03:21:07.987179+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:07.987179+0000","last_undegraded":"2024-04-27T03:21:07.987179+0000","last_fullsized":"2024-04-27T03:21:07.987179+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":87,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.226613+0000","last_change":"2024-04-27T03:21:46.226679+0000","last_active":"2024-04-27T03:21:46.226613+0000","last_peered":"2024-04-27T03:21:46.226613+0000","last_clean":"2024-04-27T03:21:46.226613+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:46.226613+0000","last_undegraded":"2024-04-27T03:21:46.226613+0000","last_fullsized":"2024-04-27T03:21:46.226613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:46.226388+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:46.226388+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.226388+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:18.698646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737807+0000","last_change":"2024-04-27T03:17:47.039005+0000","last_active":"2024-04-27T03:19:57.737807+0000","last_peered":"2024-04-27T03:19:57.737807+0000","last_clean":"2024-04-27T03:19:57.737807+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:19:57.737807+0000","last_undegraded":"2024-04-27T03:19:57.737807+0000","last_fullsized":"2024-04-27T03:19:57.737807+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.643560+0000","last_change":"2024-04-27T03:21:39.643685+0000","last_active":"2024-04-27T03:21:39.643560+0000","last_peered":"2024-04-27T03:21:39.643560+0000","last_clean":"2024-04-27T03:21:39.643560+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:39.643560+0000","last_undegraded":"2024-04-27T03:21:39.643560+0000","last_fullsized":"2024-04-27T03:21:39.643560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:39.643340+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:39.643340+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.643340+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:56.798600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:19:57.737509+0000","last_change":"2024-04-27T03:17:47.016024+0000","last_active":"2024-04-27T03:19:57.737509+0000","last_peered":"2024-04-27T03:19:57.737509+0000","last_clean":"2024-04-27T03:19:57.737509+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:19:57.737509+0000","last_undegraded":"2024-04-27T03:19:57.737509+0000","last_fullsized":"2024-04-27T03:19:57.737509+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.316963+0000","last_change":"2024-04-27T03:17:49.010865+0000","last_active":"2024-04-27T03:21:01.316963+0000","last_peered":"2024-04-27T03:21:01.316963+0000","last_clean":"2024-04-27T03:21:01.316963+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:21:01.316963+0000","last_undegraded":"2024-04-27T03:21:01.316963+0000","last_fullsized":"2024-04-27T03:21:01.316963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.738611+0000","last_change":"2024-04-27T03:17:47.019986+0000","last_active":"2024-04-27T03:20:02.738611+0000","last_peered":"2024-04-27T03:20:02.738611+0000","last_clean":"2024-04-27T03:20:02.738611+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:20:02.738611+0000","last_undegraded":"2024-04-27T03:20:02.738611+0000","last_fullsized":"2024-04-27T03:20:02.738611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.767018+0000","last_change":"2024-04-27T03:21:42.767075+0000","last_active":"2024-04-27T03:21:42.767018+0000","last_peered":"2024-04-27T03:21:42.767018+0000","last_clean":"2024-04-27T03:21:42.767018+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:42.767018+0000","last_undegraded":"2024-04-27T03:21:42.767018+0000","last_fullsized":"2024-04-27T03:21:42.767018+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:42.766832+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:42.766832+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.766832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:32:00.664417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743433+0000","last_change":"2024-04-27T03:17:47.023505+0000","last_active":"2024-04-27T03:20:02.743433+0000","last_peered":"2024-04-27T03:20:02.743433+0000","last_clean":"2024-04-27T03:20:02.743433+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:20:02.743433+0000","last_undegraded":"2024-04-27T03:20:02.743433+0000","last_fullsized":"2024-04-27T03:20:02.743433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729544+0000","last_change":"2024-04-27T03:17:49.009462+0000","last_active":"2024-04-27T03:20:07.729544+0000","last_peered":"2024-04-27T03:20:07.729544+0000","last_clean":"2024-04-27T03:20:07.729544+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:20:07.729544+0000","last_undegraded":"2024-04-27T03:20:07.729544+0000","last_fullsized":"2024-04-27T03:20:07.729544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.728069+0000","last_change":"2024-04-27T03:17:47.015754+0000","last_active":"2024-04-27T03:20:02.728069+0000","last_peered":"2024-04-27T03:20:02.728069+0000","last_clean":"2024-04-27T03:20:02.728069+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:20:02.728069+0000","last_undegraded":"2024-04-27T03:20:02.728069+0000","last_fullsized":"2024-04-27T03:20:02.728069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'8","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.217667+0000","last_change":"2024-04-27T03:21:47.217738+0000","last_active":"2024-04-27T03:21:47.217667+0000","last_peered":"2024-04-27T03:21:47.217667+0000","last_clean":"2024-04-27T03:21:47.217667+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:21:47.217667+0000","last_undegraded":"2024-04-27T03:21:47.217667+0000","last_fullsized":"2024-04-27T03:21:47.217667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:47.217463+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:47.217463+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.217463+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:30.144213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.738842+0000","last_change":"2024-04-27T03:17:47.038806+0000","last_active":"2024-04-27T03:20:02.738842+0000","last_peered":"2024-04-27T03:20:02.738842+0000","last_clean":"2024-04-27T03:20:02.738842+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:20:02.738842+0000","last_undegraded":"2024-04-27T03:20:02.738842+0000","last_fullsized":"2024-04-27T03:20:02.738842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.825968+0000","last_change":"2024-04-27T03:17:49.006535+0000","last_active":"2024-04-27T03:21:00.825968+0000","last_peered":"2024-04-27T03:21:00.825968+0000","last_clean":"2024-04-27T03:21:00.825968+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:00.825968+0000","last_undegraded":"2024-04-27T03:21:00.825968+0000","last_fullsized":"2024-04-27T03:21:00.825968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406762+0000","last_change":"2024-04-27T03:19:35.441020+0000","last_active":"2024-04-27T03:21:11.406762+0000","last_peered":"2024-04-27T03:21:11.406762+0000","last_clean":"2024-04-27T03:21:11.406762+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:21:11.406762+0000","last_undegraded":"2024-04-27T03:21:11.406762+0000","last_fullsized":"2024-04-27T03:21:11.406762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.754365+0000","last_change":"2024-04-27T03:21:47.754460+0000","last_active":"2024-04-27T03:21:47.754365+0000","last_peered":"2024-04-27T03:21:47.754365+0000","last_clean":"2024-04-27T03:21:47.754365+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:21:47.754365+0000","last_undegraded":"2024-04-27T03:21:47.754365+0000","last_fullsized":"2024-04-27T03:21:47.754365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:47.754158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:47.754158+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.754158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:25.426021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.774122+0000","last_change":"2024-04-27T03:21:51.774180+0000","last_active":"2024-04-27T03:21:51.774122+0000","last_peered":"2024-04-27T03:21:51.774122+0000","last_clean":"2024-04-27T03:21:51.774122+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:21:51.774122+0000","last_undegraded":"2024-04-27T03:21:51.774122+0000","last_fullsized":"2024-04-27T03:21:51.774122+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:51.773922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:51.773922+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.773922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:17:53.853087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.173234+0000","last_change":"2024-04-27T03:21:43.173295+0000","last_active":"2024-04-27T03:21:43.173234+0000","last_peered":"2024-04-27T03:21:43.173234+0000","last_clean":"2024-04-27T03:21:43.173234+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:43.173234+0000","last_undegraded":"2024-04-27T03:21:43.173234+0000","last_fullsized":"2024-04-27T03:21:43.173234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:43.173036+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:43.173036+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.173036+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:46.540282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740491+0000","last_change":"2024-04-27T03:17:47.026063+0000","last_active":"2024-04-27T03:20:07.740491+0000","last_peered":"2024-04-27T03:20:07.740491+0000","last_clean":"2024-04-27T03:20:07.740491+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:20:07.740491+0000","last_undegraded":"2024-04-27T03:20:07.740491+0000","last_fullsized":"2024-04-27T03:20:07.740491+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744497+0000","last_change":"2024-04-27T03:17:47.002843+0000","last_active":"2024-04-27T03:20:07.744497+0000","last_peered":"2024-04-27T03:20:07.744497+0000","last_clean":"2024-04-27T03:20:07.744497+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:20:07.744497+0000","last_undegraded":"2024-04-27T03:20:07.744497+0000","last_fullsized":"2024-04-27T03:20:07.744497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'12","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.724438+0000","last_change":"2024-04-27T03:21:31.724508+0000","last_active":"2024-04-27T03:21:31.724438+0000","last_peered":"2024-04-27T03:21:31.724438+0000","last_clean":"2024-04-27T03:21:31.724438+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:31.724438+0000","last_undegraded":"2024-04-27T03:21:31.724438+0000","last_fullsized":"2024-04-27T03:21:31.724438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:31.724252+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:31.724252+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.724252+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:49.406247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.729385+0000","last_change":"2024-04-27T03:17:47.022513+0000","last_active":"2024-04-27T03:20:07.729385+0000","last_peered":"2024-04-27T03:20:07.729385+0000","last_clean":"2024-04-27T03:20:07.729385+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:20:07.729385+0000","last_undegraded":"2024-04-27T03:20:07.729385+0000","last_fullsized":"2024-04-27T03:20:07.729385+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.602897+0000","last_change":"2024-04-27T03:21:38.603000+0000","last_active":"2024-04-27T03:21:38.602897+0000","last_peered":"2024-04-27T03:21:38.602897+0000","last_clean":"2024-04-27T03:21:38.602897+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:21:38.602897+0000","last_undegraded":"2024-04-27T03:21:38.602897+0000","last_fullsized":"2024-04-27T03:21:38.602897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:38.602665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:38.602665+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.602665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:00.698430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":180,"num_omap_keys":1,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2280,"ondisk_log_size":2280,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":11674,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1888,"num_read_kb":1653,"num_write":789,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2692,"num_omap_keys":14,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1740,"ondisk_log_size":1740,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":972,"num_read_kb":972,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":482,"ondisk_log_size":482,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":87,"num_write":316,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44072,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327768,"statfs":{"total":96636764160,"available":96591634432,"internally_reserved":0,"allocated":425984,"data_stored":148688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":983040,"data_stored":736592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705718,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":739335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39944,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331896,"statfs":{"total":96636764160,"available":96595861504,"internally_reserved":0,"allocated":458752,"data_stored":147935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:21:53.991 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.991 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.991 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.991 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.991 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.991 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.991 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.992 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.993 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.994 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.994 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.994 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.994 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.994 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.994 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.995 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.996 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.996 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.996 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.996 INFO:tasks.ceph:pgid 8.a last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.996 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.996 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.996 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.997 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.998 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.998 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.998 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.998 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.998 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.998 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.998 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:53.999 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.000 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.000 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.000 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.000 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.000 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.001 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.001 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.001 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.001 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.001 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.001 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.001 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-27T03:17:49.992337+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=49, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.002 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.003 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.004 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.004 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.004 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.004 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.004 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.004 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:21:54.004 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-27T03:22:14.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:22:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.125+0000 7f201251f640 1 Processor -- start 2024-04-27T03:22:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.125+0000 7f201251f640 1 -- start start 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f201251f640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f200c06f750 0x7f200c1335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f201251f640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f200c070470 con 0x7f200c06ec60 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f201251f640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7f200c0701a0 con 0x7f200c06f750 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f200c06f750 0x7f200c1335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200b7fe640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7f200c06f750 0x7f200c1335f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58734/0 (socket says 172.21.15.77:58734) 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200b7fe640 1 -- 172.21.15.77:0/2508638601 learned_addr learned my addr 172.21.15.77:0/2508638601 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200bfff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7f200c06ec60 0x7f200c06f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:43952/0 (socket says 172.21.15.77:43952) 2024-04-27T03:22:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200b7fe640 1 -- 172.21.15.77:0/2508638601 >> v1:172.21.15.77:6789/0 conn(0x7f200c06ec60 legacy=0x7f200c06f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:14.137 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200b7fe640 1 -- 172.21.15.77:0/2508638601 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c071b40 con 0x7f200c06f750 2024-04-27T03:22:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200b7fe640 1 --2- 172.21.15.77:0/2508638601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06f750 0x7f200c1335f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f20000097f0 tx=0x7f200001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861bd63902ad6c95 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200affd640 1 -- 172.21.15.77:0/2508638601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20000b0070 con 0x7f200c06f750 2024-04-27T03:22:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200affd640 1 -- 172.21.15.77:0/2508638601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20000ab530 con 0x7f200c06f750 2024-04-27T03:22:14.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.129+0000 7f200affd640 1 -- 172.21.15.77:0/2508638601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f20000ab830 con 0x7f200c06f750 2024-04-27T03:22:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/2508638601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06f750 msgr2=0x7f200c1335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 --2- 172.21.15.77:0/2508638601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06f750 0x7f200c1335f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f20000097f0 tx=0x7f200001c430 comp rx=0 tx=0).stop 2024-04-27T03:22:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/2508638601 shutdown_connections 2024-04-27T03:22:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 --2- 172.21.15.77:0/2508638601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06f750 0x7f200c1335f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/2508638601 >> 172.21.15.77:0/2508638601 conn(0x7f200c09dc80 msgr2=0x7f200c06d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:22:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/2508638601 shutdown_connections 2024-04-27T03:22:14.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/2508638601 wait complete. 2024-04-27T03:22:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 Processor -- start 2024-04-27T03:22:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- start start 2024-04-27T03:22:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06ec60 0x7f200c11df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7f200c1366e0 con 0x7f200c06ec60 2024-04-27T03:22:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f200bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06ec60 0x7f200c11df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f200bfff640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06ec60 0x7f200c11df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58748/0 (socket says 172.21.15.77:58748) 2024-04-27T03:22:14.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f200bfff640 1 -- 172.21.15.77:0/3268464414 learned_addr learned my addr 172.21.15.77:0/3268464414 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:22:14.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f200bfff640 1 -- 172.21.15.77:0/3268464414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200c120d00 con 0x7f200c06ec60 2024-04-27T03:22:14.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f200bfff640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06ec60 0x7f200c11df40 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00c3b0 tx=0x7f1ffc00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:14.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f2008ff9640 1 -- 172.21.15.77:0/3268464414 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ffc007410 con 0x7f200c06ec60 2024-04-27T03:22:14.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f2008ff9640 1 -- 172.21.15.77:0/3268464414 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ffc0075b0 con 0x7f200c06ec60 2024-04-27T03:22:14.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f200c120560 con 0x7f200c06ec60 2024-04-27T03:22:14.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f2008ff9640 1 -- 172.21.15.77:0/3268464414 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ffc004920 con 0x7f200c06ec60 2024-04-27T03:22:14.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f200c11e480 con 0x7f200c06ec60 2024-04-27T03:22:14.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f200c062e50 con 0x7f200c06ec60 2024-04-27T03:22:14.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f2008ff9640 1 -- 172.21.15.77:0/3268464414 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f1ffc01a020 con 0x7f200c06ec60 2024-04-27T03:22:14.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f2008ff9640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1ff8035db0 0x7f1ff8038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:14.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f2008ff9640 1 -- 172.21.15.77:0/3268464414 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7f1ffc048960 con 0x7f200c06ec60 2024-04-27T03:22:14.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.133+0000 7f200b7fe640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1ff8035db0 0x7f1ff8038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:14.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.137+0000 7f200b7fe640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1ff8035db0 0x7f1ff8038270 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f200c0505b0 tx=0x7f20000a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:14.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.137+0000 7f2008ff9640 1 -- 172.21.15.77:0/3268464414 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1ffc01b9c0 con 0x7f200c06ec60 2024-04-27T03:22:14.306 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.297+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f200c11f140 con 0x7f1ff8035db0 2024-04-27T03:22:14.350 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.341+0000 7f2008ff9640 1 -- 172.21.15.77:0/3268464414 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+664854 (secure 0 0 0) 0x7f200c11f140 con 0x7f1ff8035db0 2024-04-27T03:22:14.358 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:22:14.364 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1ff8035db0 msgr2=0x7f1ff8038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1ff8035db0 0x7f1ff8038270 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f200c0505b0 tx=0x7f20000a6000 comp rx=0 tx=0).stop 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06ec60 msgr2=0x7f200c11df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06ec60 0x7f200c11df40 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00c3b0 tx=0x7f1ffc00c870 comp rx=0 tx=0).stop 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 shutdown_connections 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7f1ff8035db0 0x7f1ff8038270 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 --2- 172.21.15.77:0/3268464414 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f200c06ec60 0x7f200c11df40 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 >> 172.21.15.77:0/3268464414 conn(0x7f200c09dc80 msgr2=0x7f200c116f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 shutdown_connections 2024-04-27T03:22:14.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:14.349+0000 7f201251f640 1 -- 172.21.15.77:0/3268464414 wait complete. 2024-04-27T03:22:14.373 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-04-27T03:22:13.768336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605383,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":5393,"num_read_kb":4942,"num_write":2371,"num_write_kb":61374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3041,"num_omap_keys":16,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4998,"ondisk_log_size":4998,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":140024,"kb_used_data":2840,"kb_used_omap":6,"kb_used_meta":137145,"kb_avail":377347336,"statfs":{"total":386547056640,"available":386403672064,"internally_reserved":0,"allocated":2908160,"data_stored":1772550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":140437270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":169,"num_omap_keys":1,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006359"},"pg_stats":[{"pgid":"3.3f","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.335160+0000","last_change":"2024-04-27T03:22:01.335229+0000","last_active":"2024-04-27T03:22:01.335160+0000","last_peered":"2024-04-27T03:22:01.335160+0000","last_clean":"2024-04-27T03:22:01.335160+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:22:01.335160+0000","last_undegraded":"2024-04-27T03:22:01.335160+0000","last_fullsized":"2024-04-27T03:22:01.335160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:01.334959+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:01.334959+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.334959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:56.778017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'6","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.158576+0000","last_change":"2024-04-27T03:21:42.158641+0000","last_active":"2024-04-27T03:21:42.158576+0000","last_peered":"2024-04-27T03:21:42.158576+0000","last_clean":"2024-04-27T03:21:42.158576+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:42.158576+0000","last_undegraded":"2024-04-27T03:21:42.158576+0000","last_fullsized":"2024-04-27T03:21:42.158576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:42.158374+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:42.158374+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.158374+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:53.231269+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.472960+0000","last_change":"2024-04-27T03:21:57.472960+0000","last_active":"2024-04-27T03:21:57.472960+0000","last_peered":"2024-04-27T03:21:57.472960+0000","last_clean":"2024-04-27T03:21:57.472960+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:21:57.472960+0000","last_undegraded":"2024-04-27T03:21:57.472960+0000","last_fullsized":"2024-04-27T03:21:57.472960+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:57.472693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:57.472693+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.472693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:35.144556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.789741+0000","last_change":"2024-04-27T03:21:43.789808+0000","last_active":"2024-04-27T03:21:43.789741+0000","last_peered":"2024-04-27T03:21:43.789741+0000","last_clean":"2024-04-27T03:21:43.789741+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:21:43.789741+0000","last_undegraded":"2024-04-27T03:21:43.789741+0000","last_fullsized":"2024-04-27T03:21:43.789741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'8","last_scrub_stamp":"2024-04-27T03:21:43.789520+0000","last_deep_scrub":"27'8","last_deep_scrub_stamp":"2024-04-27T03:21:43.789520+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.789520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:16.261778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.802731+0000","last_change":"2024-04-27T03:21:58.802787+0000","last_active":"2024-04-27T03:21:58.802731+0000","last_peered":"2024-04-27T03:21:58.802731+0000","last_clean":"2024-04-27T03:21:58.802731+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:21:58.802731+0000","last_undegraded":"2024-04-27T03:21:58.802731+0000","last_fullsized":"2024-04-27T03:21:58.802731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:58.802498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:58.802498+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.802498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:03.518528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.749840+0000","last_change":"2024-04-27T03:21:49.749992+0000","last_active":"2024-04-27T03:21:49.749840+0000","last_peered":"2024-04-27T03:21:49.749840+0000","last_clean":"2024-04-27T03:21:49.749840+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:21:49.749840+0000","last_undegraded":"2024-04-27T03:21:49.749840+0000","last_fullsized":"2024-04-27T03:21:49.749840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:49.749616+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:49.749616+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.749616+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:22.385004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.741829+0000","last_change":"2024-04-27T03:21:50.741891+0000","last_active":"2024-04-27T03:21:50.741829+0000","last_peered":"2024-04-27T03:21:50.741829+0000","last_clean":"2024-04-27T03:21:50.741829+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:21:50.741829+0000","last_undegraded":"2024-04-27T03:21:50.741829+0000","last_fullsized":"2024-04-27T03:21:50.741829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:50.741587+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:50.741587+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.741587+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:38.633527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":90,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.864487+0000","last_change":"2024-04-27T03:21:41.864610+0000","last_active":"2024-04-27T03:21:41.864487+0000","last_peered":"2024-04-27T03:21:41.864487+0000","last_clean":"2024-04-27T03:21:41.864487+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:41.864487+0000","last_undegraded":"2024-04-27T03:21:41.864487+0000","last_fullsized":"2024-04-27T03:21:41.864487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:41.864193+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:41.864193+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.864193+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:28.818568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.293028+0000","last_change":"2024-04-27T03:22:00.293160+0000","last_active":"2024-04-27T03:22:00.293028+0000","last_peered":"2024-04-27T03:22:00.293028+0000","last_clean":"2024-04-27T03:22:00.293028+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:22:00.293028+0000","last_undegraded":"2024-04-27T03:22:00.293028+0000","last_fullsized":"2024-04-27T03:22:00.293028+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:00.292799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:00.292799+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.292799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:45.148184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.698631+0000","last_change":"2024-04-27T03:21:58.698694+0000","last_active":"2024-04-27T03:21:58.698631+0000","last_peered":"2024-04-27T03:21:58.698631+0000","last_clean":"2024-04-27T03:21:58.698631+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:21:58.698631+0000","last_undegraded":"2024-04-27T03:21:58.698631+0000","last_fullsized":"2024-04-27T03:21:58.698631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:58.698432+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:58.698432+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.698432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:28:46.564342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:03.317458+0000","last_change":"2024-04-27T03:22:03.317531+0000","last_active":"2024-04-27T03:22:03.317458+0000","last_peered":"2024-04-27T03:22:03.317458+0000","last_clean":"2024-04-27T03:22:03.317458+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:22:03.317458+0000","last_undegraded":"2024-04-27T03:22:03.317458+0000","last_fullsized":"2024-04-27T03:22:03.317458+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:22:03.317238+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:22:03.317238+0000","last_clean_scrub_stamp":"2024-04-27T03:22:03.317238+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:29:09.833669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'12","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.217129+0000","last_change":"2024-04-27T03:21:50.217129+0000","last_active":"2024-04-27T03:21:50.217129+0000","last_peered":"2024-04-27T03:21:50.217129+0000","last_clean":"2024-04-27T03:21:50.217129+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:21:50.217129+0000","last_undegraded":"2024-04-27T03:21:50.217129+0000","last_fullsized":"2024-04-27T03:21:50.217129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:50.216879+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:50.216879+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.216879+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:27.888741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.759798+0000","last_change":"2024-04-27T03:22:01.759855+0000","last_active":"2024-04-27T03:22:01.759798+0000","last_peered":"2024-04-27T03:22:01.759798+0000","last_clean":"2024-04-27T03:22:01.759798+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:22:01.759798+0000","last_undegraded":"2024-04-27T03:22:01.759798+0000","last_fullsized":"2024-04-27T03:22:01.759798+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:01.759596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:01.759596+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.759596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:09:41.922637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.088312+0000","last_change":"2024-04-27T03:21:38.088312+0000","last_active":"2024-04-27T03:21:38.088312+0000","last_peered":"2024-04-27T03:21:38.088312+0000","last_clean":"2024-04-27T03:21:38.088312+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:38.088312+0000","last_undegraded":"2024-04-27T03:21:38.088312+0000","last_fullsized":"2024-04-27T03:21:38.088312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:38.088067+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:38.088067+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.088067+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:30.953622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:10.542137+0000","last_change":"2024-04-27T03:22:10.542203+0000","last_active":"2024-04-27T03:22:10.542137+0000","last_peered":"2024-04-27T03:22:10.542137+0000","last_clean":"2024-04-27T03:22:10.542137+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:22:10.542137+0000","last_undegraded":"2024-04-27T03:22:10.542137+0000","last_fullsized":"2024-04-27T03:22:10.542137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:22:10.541930+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:22:10.541930+0000","last_clean_scrub_stamp":"2024-04-27T03:22:10.541930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:40:12.968269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.734857+0000","last_change":"2024-04-27T03:21:37.734857+0000","last_active":"2024-04-27T03:21:37.734857+0000","last_peered":"2024-04-27T03:21:37.734857+0000","last_clean":"2024-04-27T03:21:37.734857+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:21:37.734857+0000","last_undegraded":"2024-04-27T03:21:37.734857+0000","last_fullsized":"2024-04-27T03:21:37.734857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:37.734535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:37.734535+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.734535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:00.453711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.163703+0000","last_change":"2024-04-27T03:21:40.163767+0000","last_active":"2024-04-27T03:21:40.163703+0000","last_peered":"2024-04-27T03:21:40.163703+0000","last_clean":"2024-04-27T03:21:40.163703+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:40.163703+0000","last_undegraded":"2024-04-27T03:21:40.163703+0000","last_fullsized":"2024-04-27T03:21:40.163703+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:40.163506+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:40.163506+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.163506+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:29.161646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.521461+0000","last_change":"2024-04-27T03:21:53.521581+0000","last_active":"2024-04-27T03:21:53.521461+0000","last_peered":"2024-04-27T03:21:53.521461+0000","last_clean":"2024-04-27T03:21:53.521461+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:21:53.521461+0000","last_undegraded":"2024-04-27T03:21:53.521461+0000","last_fullsized":"2024-04-27T03:21:53.521461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:53.521219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:53.521219+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.521219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:25.993477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.187547+0000","last_change":"2024-04-27T03:21:54.187602+0000","last_active":"2024-04-27T03:21:54.187547+0000","last_peered":"2024-04-27T03:21:54.187547+0000","last_clean":"2024-04-27T03:21:54.187547+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:21:54.187547+0000","last_undegraded":"2024-04-27T03:21:54.187547+0000","last_fullsized":"2024-04-27T03:21:54.187547+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:54.187345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:54.187345+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.187345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:02:44.457897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.771974+0000","last_change":"2024-04-27T03:21:48.772040+0000","last_active":"2024-04-27T03:21:48.771974+0000","last_peered":"2024-04-27T03:21:48.771974+0000","last_clean":"2024-04-27T03:21:48.771974+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:21:48.771974+0000","last_undegraded":"2024-04-27T03:21:48.771974+0000","last_fullsized":"2024-04-27T03:21:48.771974+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:48.771778+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.771778+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.771778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:10.421790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.825947+0000","last_change":"2024-04-27T03:17:49.011747+0000","last_active":"2024-04-27T03:21:00.825947+0000","last_peered":"2024-04-27T03:21:00.825947+0000","last_clean":"2024-04-27T03:21:00.825947+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:21:00.825947+0000","last_undegraded":"2024-04-27T03:21:00.825947+0000","last_fullsized":"2024-04-27T03:21:00.825947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.249433+0000","last_change":"2024-04-27T03:21:57.249502+0000","last_active":"2024-04-27T03:21:57.249433+0000","last_peered":"2024-04-27T03:21:57.249433+0000","last_clean":"2024-04-27T03:21:57.249433+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:21:57.249433+0000","last_undegraded":"2024-04-27T03:21:57.249433+0000","last_fullsized":"2024-04-27T03:21:57.249433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:57.249215+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:57.249215+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.249215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:06.922093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.710830+0000","last_change":"2024-04-27T03:21:33.710892+0000","last_active":"2024-04-27T03:21:33.710830+0000","last_peered":"2024-04-27T03:21:33.710830+0000","last_clean":"2024-04-27T03:21:33.710830+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:33.710830+0000","last_undegraded":"2024-04-27T03:21:33.710830+0000","last_fullsized":"2024-04-27T03:21:33.710830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:33.710637+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:33.710637+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.710637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:36.041679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:06.302523+0000","last_change":"2024-04-27T03:22:06.302586+0000","last_active":"2024-04-27T03:22:06.302523+0000","last_peered":"2024-04-27T03:22:06.302523+0000","last_clean":"2024-04-27T03:22:06.302523+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:22:06.302523+0000","last_undegraded":"2024-04-27T03:22:06.302523+0000","last_fullsized":"2024-04-27T03:22:06.302523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'4","last_scrub_stamp":"2024-04-27T03:22:06.302319+0000","last_deep_scrub":"27'4","last_deep_scrub_stamp":"2024-04-27T03:22:06.302319+0000","last_clean_scrub_stamp":"2024-04-27T03:22:06.302319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:21:01.297686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.716905+0000","last_change":"2024-04-27T03:21:34.716967+0000","last_active":"2024-04-27T03:21:34.716905+0000","last_peered":"2024-04-27T03:21:34.716905+0000","last_clean":"2024-04-27T03:21:34.716905+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:34.716905+0000","last_undegraded":"2024-04-27T03:21:34.716905+0000","last_fullsized":"2024-04-27T03:21:34.716905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:34.716714+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:34.716714+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.716714+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:06:56.440270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:05.682655+0000","last_change":"2024-04-27T03:22:05.682711+0000","last_active":"2024-04-27T03:22:05.682655+0000","last_peered":"2024-04-27T03:22:05.682655+0000","last_clean":"2024-04-27T03:22:05.682655+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:22:05.682655+0000","last_undegraded":"2024-04-27T03:22:05.682655+0000","last_fullsized":"2024-04-27T03:22:05.682655+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:05.682454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:05.682454+0000","last_clean_scrub_stamp":"2024-04-27T03:22:05.682454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:26:18.821680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.514246+0000","last_change":"2024-04-27T03:21:54.514398+0000","last_active":"2024-04-27T03:21:54.514246+0000","last_peered":"2024-04-27T03:21:54.514246+0000","last_clean":"2024-04-27T03:21:54.514246+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:54.514246+0000","last_undegraded":"2024-04-27T03:21:54.514246+0000","last_fullsized":"2024-04-27T03:21:54.514246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:54.514020+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:54.514020+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.514020+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:37.440770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:07.604658+0000","last_change":"2024-04-27T03:22:07.604716+0000","last_active":"2024-04-27T03:22:07.604658+0000","last_peered":"2024-04-27T03:22:07.604658+0000","last_clean":"2024-04-27T03:22:07.604658+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:22:07.604658+0000","last_undegraded":"2024-04-27T03:22:07.604658+0000","last_fullsized":"2024-04-27T03:22:07.604658+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:07.604461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:07.604461+0000","last_clean_scrub_stamp":"2024-04-27T03:22:07.604461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:52:58.924480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'37","reported_seq":106,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.340362+0000","last_change":"2024-04-27T03:17:57.088698+0000","last_active":"2024-04-27T03:21:01.340362+0000","last_peered":"2024-04-27T03:21:01.340362+0000","last_clean":"2024-04-27T03:21:01.340362+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:21:01.340362+0000","last_undegraded":"2024-04-27T03:21:01.340362+0000","last_fullsized":"2024-04-27T03:21:01.340362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.715048+0000","last_change":"2024-04-27T03:21:51.715106+0000","last_active":"2024-04-27T03:21:51.715048+0000","last_peered":"2024-04-27T03:21:51.715048+0000","last_clean":"2024-04-27T03:21:51.715048+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:51.715048+0000","last_undegraded":"2024-04-27T03:21:51.715048+0000","last_fullsized":"2024-04-27T03:21:51.715048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:51.714857+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:51.714857+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.714857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:02:41.985409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740181+0000","last_change":"2024-04-27T03:17:47.018824+0000","last_active":"2024-04-27T03:20:07.740181+0000","last_peered":"2024-04-27T03:20:07.740181+0000","last_clean":"2024-04-27T03:20:07.740181+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:20:07.740181+0000","last_undegraded":"2024-04-27T03:20:07.740181+0000","last_fullsized":"2024-04-27T03:20:07.740181+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.758982+0000","last_change":"2024-04-27T03:21:34.759034+0000","last_active":"2024-04-27T03:21:34.758982+0000","last_peered":"2024-04-27T03:21:34.758982+0000","last_clean":"2024-04-27T03:21:34.758982+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:21:34.758982+0000","last_undegraded":"2024-04-27T03:21:34.758982+0000","last_fullsized":"2024-04-27T03:21:34.758982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:34.758786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:34.758786+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.758786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:06.643052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'51","reported_seq":122,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.542440+0000","last_change":"2024-04-27T03:21:17.542518+0000","last_active":"2024-04-27T03:21:17.542440+0000","last_peered":"2024-04-27T03:21:17.542440+0000","last_clean":"2024-04-27T03:21:17.542440+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:21:17.542440+0000","last_undegraded":"2024-04-27T03:21:17.542440+0000","last_fullsized":"2024-04-27T03:21:17.542440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'51","last_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_deep_scrub":"31'51","last_deep_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.542266+0000","objects_scrubbed":6,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:24.169053+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.186893+0000","last_change":"2024-04-27T03:21:49.186953+0000","last_active":"2024-04-27T03:21:49.186893+0000","last_peered":"2024-04-27T03:21:49.186893+0000","last_clean":"2024-04-27T03:21:49.186893+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:21:49.186893+0000","last_undegraded":"2024-04-27T03:21:49.186893+0000","last_fullsized":"2024-04-27T03:21:49.186893+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:49.186698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:49.186698+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.186698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:31.658597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.158141+0000","last_change":"2024-04-27T03:21:53.158222+0000","last_active":"2024-04-27T03:21:53.158141+0000","last_peered":"2024-04-27T03:21:53.158141+0000","last_clean":"2024-04-27T03:21:53.158141+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:53.158141+0000","last_undegraded":"2024-04-27T03:21:53.158141+0000","last_fullsized":"2024-04-27T03:21:53.158141+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:53.157941+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:53.157941+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.157941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:56.275203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.784863+0000","last_change":"2024-04-27T03:21:46.784922+0000","last_active":"2024-04-27T03:21:46.784863+0000","last_peered":"2024-04-27T03:21:46.784863+0000","last_clean":"2024-04-27T03:21:46.784863+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:21:46.784863+0000","last_undegraded":"2024-04-27T03:21:46.784863+0000","last_fullsized":"2024-04-27T03:21:46.784863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:46.784639+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:46.784639+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.784639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:29.256538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.740666+0000","last_change":"2024-04-27T03:21:54.740727+0000","last_active":"2024-04-27T03:21:54.740666+0000","last_peered":"2024-04-27T03:21:54.740666+0000","last_clean":"2024-04-27T03:21:54.740666+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:21:54.740666+0000","last_undegraded":"2024-04-27T03:21:54.740666+0000","last_fullsized":"2024-04-27T03:21:54.740666+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:54.740455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:54.740455+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.740455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:29.391194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.767822+0000","last_change":"2024-04-27T03:21:38.767879+0000","last_active":"2024-04-27T03:21:38.767822+0000","last_peered":"2024-04-27T03:21:38.767822+0000","last_clean":"2024-04-27T03:21:38.767822+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:21:38.767822+0000","last_undegraded":"2024-04-27T03:21:38.767822+0000","last_fullsized":"2024-04-27T03:21:38.767822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:38.767589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:38.767589+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.767589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:55.841408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"31'16","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.287986+0000","last_change":"2024-04-27T03:21:22.288067+0000","last_active":"2024-04-27T03:21:22.287986+0000","last_peered":"2024-04-27T03:21:22.287986+0000","last_clean":"2024-04-27T03:21:22.287986+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:21:22.287986+0000","last_undegraded":"2024-04-27T03:21:22.287986+0000","last_fullsized":"2024-04-27T03:21:22.287986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.287788+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:31.752629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.205494+0000","last_change":"2024-04-27T03:21:23.205547+0000","last_active":"2024-04-27T03:21:23.205494+0000","last_peered":"2024-04-27T03:21:23.205494+0000","last_clean":"2024-04-27T03:21:23.205494+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:21:23.205494+0000","last_undegraded":"2024-04-27T03:21:23.205494+0000","last_fullsized":"2024-04-27T03:21:23.205494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.205266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:32.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.745803+0000","last_change":"2024-04-27T03:17:57.074831+0000","last_active":"2024-04-27T03:20:27.745803+0000","last_peered":"2024-04-27T03:20:27.745803+0000","last_clean":"2024-04-27T03:20:27.745803+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:20:27.745803+0000","last_undegraded":"2024-04-27T03:20:27.745803+0000","last_fullsized":"2024-04-27T03:20:27.745803+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'16","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.813339+0000","last_change":"2024-04-27T03:21:39.813406+0000","last_active":"2024-04-27T03:21:39.813339+0000","last_peered":"2024-04-27T03:21:39.813339+0000","last_clean":"2024-04-27T03:21:39.813339+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:39.813339+0000","last_undegraded":"2024-04-27T03:21:39.813339+0000","last_fullsized":"2024-04-27T03:21:39.813339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:39.813135+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:39.813135+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.813135+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:50.886030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:10.388133+0000","last_change":"2024-04-27T03:22:10.388195+0000","last_active":"2024-04-27T03:22:10.388133+0000","last_peered":"2024-04-27T03:22:10.388133+0000","last_clean":"2024-04-27T03:22:10.388133+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:22:10.388133+0000","last_undegraded":"2024-04-27T03:22:10.388133+0000","last_fullsized":"2024-04-27T03:22:10.388133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:10.387899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:10.387899+0000","last_clean_scrub_stamp":"2024-04-27T03:22:10.387899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:53:01.707918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.555607+0000","last_change":"2024-04-27T03:21:18.555657+0000","last_active":"2024-04-27T03:21:18.555607+0000","last_peered":"2024-04-27T03:21:18.555607+0000","last_clean":"2024-04-27T03:21:18.555607+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:21:18.555607+0000","last_undegraded":"2024-04-27T03:21:18.555607+0000","last_fullsized":"2024-04-27T03:21:18.555607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.555415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:17.141075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"31'29","reported_seq":84,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.786827+0000","last_change":"2024-04-27T03:21:28.786909+0000","last_active":"2024-04-27T03:21:28.786827+0000","last_peered":"2024-04-27T03:21:28.786827+0000","last_clean":"2024-04-27T03:21:28.786827+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:21:28.786827+0000","last_undegraded":"2024-04-27T03:21:28.786827+0000","last_fullsized":"2024-04-27T03:21:28.786827+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'29","last_scrub_stamp":"2024-04-27T03:21:28.786620+0000","last_deep_scrub":"31'29","last_deep_scrub_stamp":"2024-04-27T03:21:28.786620+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.786620+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:33.411918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'285","reported_seq":438,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.112035+0000","last_change":"2024-04-27T03:21:37.112095+0000","last_active":"2024-04-27T03:21:37.112035+0000","last_peered":"2024-04-27T03:21:37.112035+0000","last_clean":"2024-04-27T03:21:37.112035+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:37.112035+0000","last_undegraded":"2024-04-27T03:21:37.112035+0000","last_fullsized":"2024-04-27T03:21:37.112035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:37.111852+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:37.111852+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.111852+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:27.898262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.797908+0000","last_change":"2024-04-27T03:21:13.797972+0000","last_active":"2024-04-27T03:21:13.797908+0000","last_peered":"2024-04-27T03:21:13.797908+0000","last_clean":"2024-04-27T03:21:13.797908+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:21:13.797908+0000","last_undegraded":"2024-04-27T03:21:13.797908+0000","last_fullsized":"2024-04-27T03:21:13.797908+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.797701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:15.106812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.802531+0000","last_change":"2024-04-27T03:21:45.802590+0000","last_active":"2024-04-27T03:21:45.802531+0000","last_peered":"2024-04-27T03:21:45.802531+0000","last_clean":"2024-04-27T03:21:45.802531+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:45.802531+0000","last_undegraded":"2024-04-27T03:21:45.802531+0000","last_fullsized":"2024-04-27T03:21:45.802531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:45.802345+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:45.802345+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.802345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:23.495705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744463+0000","last_change":"2024-04-27T03:17:47.003169+0000","last_active":"2024-04-27T03:20:07.744463+0000","last_peered":"2024-04-27T03:20:07.744463+0000","last_clean":"2024-04-27T03:20:07.744463+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:20:07.744463+0000","last_undegraded":"2024-04-27T03:20:07.744463+0000","last_fullsized":"2024-04-27T03:20:07.744463+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.137288+0000","last_change":"2024-04-27T03:21:36.137341+0000","last_active":"2024-04-27T03:21:36.137288+0000","last_peered":"2024-04-27T03:21:36.137288+0000","last_clean":"2024-04-27T03:21:36.137288+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:21:36.137288+0000","last_undegraded":"2024-04-27T03:21:36.137288+0000","last_fullsized":"2024-04-27T03:21:36.137288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:36.137094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:36.137094+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.137094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:18.002317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'43","reported_seq":108,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.228263+0000","last_change":"2024-04-27T03:21:21.228344+0000","last_active":"2024-04-27T03:21:21.228263+0000","last_peered":"2024-04-27T03:21:21.228263+0000","last_clean":"2024-04-27T03:21:21.228263+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:21:21.228263+0000","last_undegraded":"2024-04-27T03:21:21.228263+0000","last_fullsized":"2024-04-27T03:21:21.228263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'43","last_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_deep_scrub":"31'43","last_deep_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.228072+0000","objects_scrubbed":5,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:05.540210+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'285","reported_seq":418,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.596919+0000","last_change":"2024-04-27T03:21:24.596980+0000","last_active":"2024-04-27T03:21:24.596919+0000","last_peered":"2024-04-27T03:21:24.596919+0000","last_clean":"2024-04-27T03:21:24.596919+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:24.596919+0000","last_undegraded":"2024-04-27T03:21:24.596919+0000","last_fullsized":"2024-04-27T03:21:24.596919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.596708+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:43.207566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'32","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.730071+0000","last_change":"2024-04-27T03:22:00.730133+0000","last_active":"2024-04-27T03:22:00.730071+0000","last_peered":"2024-04-27T03:22:00.730071+0000","last_clean":"2024-04-27T03:22:00.730071+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:22:00.730071+0000","last_undegraded":"2024-04-27T03:22:00.730071+0000","last_fullsized":"2024-04-27T03:22:00.730071+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'32","last_scrub_stamp":"2024-04-27T03:22:00.729884+0000","last_deep_scrub":"31'32","last_deep_scrub_stamp":"2024-04-27T03:22:00.729884+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.729884+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:20.945115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:04.277394+0000","last_change":"2024-04-27T03:22:04.277461+0000","last_active":"2024-04-27T03:22:04.277394+0000","last_peered":"2024-04-27T03:22:04.277394+0000","last_clean":"2024-04-27T03:22:04.277394+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:22:04.277394+0000","last_undegraded":"2024-04-27T03:22:04.277394+0000","last_fullsized":"2024-04-27T03:22:04.277394+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:04.277181+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:04.277181+0000","last_clean_scrub_stamp":"2024-04-27T03:22:04.277181+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:09:44.440223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.413670+0000","last_change":"2024-04-27T03:21:31.413670+0000","last_active":"2024-04-27T03:21:31.413670+0000","last_peered":"2024-04-27T03:21:31.413670+0000","last_clean":"2024-04-27T03:21:31.413670+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:21:31.413670+0000","last_undegraded":"2024-04-27T03:21:31.413670+0000","last_fullsized":"2024-04-27T03:21:31.413670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:31.413366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:31.413366+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.413366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:23.060435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'59","reported_seq":130,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.203825+0000","last_change":"2024-04-27T03:21:26.203825+0000","last_active":"2024-04-27T03:21:26.203825+0000","last_peered":"2024-04-27T03:21:26.203825+0000","last_clean":"2024-04-27T03:21:26.203825+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:21:26.203825+0000","last_undegraded":"2024-04-27T03:21:26.203825+0000","last_fullsized":"2024-04-27T03:21:26.203825+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'59","last_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_deep_scrub":"31'59","last_deep_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.203534+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:44.814392+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":47,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.260194+0000","last_change":"2024-04-27T03:21:15.260248+0000","last_active":"2024-04-27T03:21:15.260194+0000","last_peered":"2024-04-27T03:21:15.260194+0000","last_clean":"2024-04-27T03:21:15.260194+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:21:15.260194+0000","last_undegraded":"2024-04-27T03:21:15.260194+0000","last_fullsized":"2024-04-27T03:21:15.260194+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.259998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:59.062074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.825848+0000","last_change":"2024-04-27T03:21:25.825848+0000","last_active":"2024-04-27T03:21:25.825848+0000","last_peered":"2024-04-27T03:21:25.825848+0000","last_clean":"2024-04-27T03:21:25.825848+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:21:25.825848+0000","last_undegraded":"2024-04-27T03:21:25.825848+0000","last_fullsized":"2024-04-27T03:21:25.825848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'3","last_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_deep_scrub":"29'3","last_deep_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.825552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:43:06.575600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.537316+0000","last_change":"2024-04-27T03:21:50.537316+0000","last_active":"2024-04-27T03:21:50.537316+0000","last_peered":"2024-04-27T03:21:50.537316+0000","last_clean":"2024-04-27T03:21:50.537316+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:50.537316+0000","last_undegraded":"2024-04-27T03:21:50.537316+0000","last_fullsized":"2024-04-27T03:21:50.537316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:50.537066+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:50.537066+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.537066+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:53.904312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743585+0000","last_change":"2024-04-27T03:17:46.995080+0000","last_active":"2024-04-27T03:20:02.743585+0000","last_peered":"2024-04-27T03:20:02.743585+0000","last_clean":"2024-04-27T03:20:02.743585+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:20:02.743585+0000","last_undegraded":"2024-04-27T03:20:02.743585+0000","last_fullsized":"2024-04-27T03:20:02.743585+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.762950+0000","last_change":"2024-04-27T03:21:43.762950+0000","last_active":"2024-04-27T03:21:43.762950+0000","last_peered":"2024-04-27T03:21:43.762950+0000","last_clean":"2024-04-27T03:21:43.762950+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:21:43.762950+0000","last_undegraded":"2024-04-27T03:21:43.762950+0000","last_fullsized":"2024-04-27T03:21:43.762950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:43.762700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:43.762700+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.762700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:13.310122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'44","reported_seq":109,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.518797+0000","last_change":"2024-04-27T03:21:13.518875+0000","last_active":"2024-04-27T03:21:13.518797+0000","last_peered":"2024-04-27T03:21:13.518797+0000","last_clean":"2024-04-27T03:21:13.518797+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:21:13.518797+0000","last_undegraded":"2024-04-27T03:21:13.518797+0000","last_fullsized":"2024-04-27T03:21:13.518797+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'44","last_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_deep_scrub":"31'44","last_deep_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.518592+0000","objects_scrubbed":5,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:24:22.864770+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.708239+0000","last_change":"2024-04-27T03:21:32.708291+0000","last_active":"2024-04-27T03:21:32.708239+0000","last_peered":"2024-04-27T03:21:32.708239+0000","last_clean":"2024-04-27T03:21:32.708239+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:21:32.708239+0000","last_undegraded":"2024-04-27T03:21:32.708239+0000","last_fullsized":"2024-04-27T03:21:32.708239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:32.708044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:32.708044+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.708044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:43.162536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"31'6","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:11.267314+0000","last_change":"2024-04-27T03:22:11.267381+0000","last_active":"2024-04-27T03:22:11.267314+0000","last_peered":"2024-04-27T03:22:11.267314+0000","last_clean":"2024-04-27T03:22:11.267314+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:22:11.267314+0000","last_undegraded":"2024-04-27T03:22:11.267314+0000","last_fullsized":"2024-04-27T03:22:11.267314+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:11.267113+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:11.267113+0000","last_clean_scrub_stamp":"2024-04-27T03:22:11.267113+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:09:51.430154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":169,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.774871+0000","last_change":"2024-04-27T03:21:29.774989+0000","last_active":"2024-04-27T03:21:29.774871+0000","last_peered":"2024-04-27T03:21:29.774871+0000","last_clean":"2024-04-27T03:21:29.774871+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:29.774871+0000","last_undegraded":"2024-04-27T03:21:29.774871+0000","last_fullsized":"2024-04-27T03:21:29.774871+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-04-27T03:21:29.774563+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-04-27T03:21:29.774563+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.774563+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:41.424401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:09.395611+0000","last_change":"2024-04-27T03:22:09.395668+0000","last_active":"2024-04-27T03:22:09.395611+0000","last_peered":"2024-04-27T03:22:09.395611+0000","last_clean":"2024-04-27T03:22:09.395611+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:22:09.395611+0000","last_undegraded":"2024-04-27T03:22:09.395611+0000","last_fullsized":"2024-04-27T03:22:09.395611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:09.395415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:09.395415+0000","last_clean_scrub_stamp":"2024-04-27T03:22:09.395415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:52.077494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.794463+0000","last_change":"2024-04-27T03:21:45.794520+0000","last_active":"2024-04-27T03:21:45.794463+0000","last_peered":"2024-04-27T03:21:45.794463+0000","last_clean":"2024-04-27T03:21:45.794463+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:21:45.794463+0000","last_undegraded":"2024-04-27T03:21:45.794463+0000","last_fullsized":"2024-04-27T03:21:45.794463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:45.794255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:45.794255+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.794255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:52.421042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'51","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:08.091926+0000","last_change":"2024-04-27T03:18:34.800954+0000","last_active":"2024-04-27T03:21:08.091926+0000","last_peered":"2024-04-27T03:21:08.091926+0000","last_clean":"2024-04-27T03:21:08.091926+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:21:08.091926+0000","last_undegraded":"2024-04-27T03:21:08.091926+0000","last_fullsized":"2024-04-27T03:21:08.091926+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:52.005853+0000","last_clean_scrub_stamp":"2024-04-27T03:17:52.005853+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'285","reported_seq":393,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.866360+0000","last_change":"2024-04-27T03:21:20.866425+0000","last_active":"2024-04-27T03:21:20.866360+0000","last_peered":"2024-04-27T03:21:20.866360+0000","last_clean":"2024-04-27T03:21:20.866360+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:20.866360+0000","last_undegraded":"2024-04-27T03:21:20.866360+0000","last_fullsized":"2024-04-27T03:21:20.866360+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.866168+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:18.678233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.628232+0000","last_change":"2024-04-27T03:21:40.628282+0000","last_active":"2024-04-27T03:21:40.628232+0000","last_peered":"2024-04-27T03:21:40.628232+0000","last_clean":"2024-04-27T03:21:40.628232+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:21:40.628232+0000","last_undegraded":"2024-04-27T03:21:40.628232+0000","last_fullsized":"2024-04-27T03:21:40.628232+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:40.628044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:40.628044+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.628044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:28:28.493954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":97,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.749222+0000","last_change":"2024-04-27T03:21:40.749288+0000","last_active":"2024-04-27T03:21:40.749222+0000","last_peered":"2024-04-27T03:21:40.749222+0000","last_clean":"2024-04-27T03:21:40.749222+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:40.749222+0000","last_undegraded":"2024-04-27T03:21:40.749222+0000","last_fullsized":"2024-04-27T03:21:40.749222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:40.749042+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:40.749042+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.749042+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:50.213883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.743488+0000","last_change":"2024-04-27T03:18:34.772611+0000","last_active":"2024-04-27T03:20:02.743488+0000","last_peered":"2024-04-27T03:20:02.743488+0000","last_clean":"2024-04-27T03:20:02.743488+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:20:02.743488+0000","last_undegraded":"2024-04-27T03:20:02.743488+0000","last_fullsized":"2024-04-27T03:20:02.743488+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.756406+0000","last_change":"2024-04-27T03:21:30.756406+0000","last_active":"2024-04-27T03:21:30.756406+0000","last_peered":"2024-04-27T03:21:30.756406+0000","last_clean":"2024-04-27T03:21:30.756406+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:21:30.756406+0000","last_undegraded":"2024-04-27T03:21:30.756406+0000","last_fullsized":"2024-04-27T03:21:30.756406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:30.756077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:30.756077+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.756077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:41:43.084146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'77","reported_seq":167,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.632024+0000","last_change":"2024-04-27T03:21:27.632024+0000","last_active":"2024-04-27T03:21:27.632024+0000","last_peered":"2024-04-27T03:21:27.632024+0000","last_clean":"2024-04-27T03:21:27.632024+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:21:27.632024+0000","last_undegraded":"2024-04-27T03:21:27.632024+0000","last_fullsized":"2024-04-27T03:21:27.632024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'77","last_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_deep_scrub":"31'77","last_deep_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.631741+0000","objects_scrubbed":7,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:01.393467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":34,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.822541+0000","last_change":"2024-04-27T03:21:22.822651+0000","last_active":"2024-04-27T03:21:22.822541+0000","last_peered":"2024-04-27T03:21:22.822541+0000","last_clean":"2024-04-27T03:21:22.822541+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:21:22.822541+0000","last_undegraded":"2024-04-27T03:21:22.822541+0000","last_fullsized":"2024-04-27T03:21:22.822541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.822323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:20.663089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'21","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.464675+0000","last_change":"2024-04-27T03:21:32.464841+0000","last_active":"2024-04-27T03:21:32.464675+0000","last_peered":"2024-04-27T03:21:32.464675+0000","last_clean":"2024-04-27T03:21:32.464675+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:32.464675+0000","last_undegraded":"2024-04-27T03:21:32.464675+0000","last_fullsized":"2024-04-27T03:21:32.464675+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'21","last_scrub_stamp":"2024-04-27T03:21:32.464430+0000","last_deep_scrub":"31'21","last_deep_scrub_stamp":"2024-04-27T03:21:32.464430+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.464430+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:20.356371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.649742+0000","last_change":"2024-04-27T03:21:42.649809+0000","last_active":"2024-04-27T03:21:42.649742+0000","last_peered":"2024-04-27T03:21:42.649742+0000","last_clean":"2024-04-27T03:21:42.649742+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:42.649742+0000","last_undegraded":"2024-04-27T03:21:42.649742+0000","last_fullsized":"2024-04-27T03:21:42.649742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:42.649523+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:42.649523+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.649523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:58.314334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.709141+0000","last_change":"2024-04-27T03:21:56.709198+0000","last_active":"2024-04-27T03:21:56.709141+0000","last_peered":"2024-04-27T03:21:56.709141+0000","last_clean":"2024-04-27T03:21:56.709141+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:21:56.709141+0000","last_undegraded":"2024-04-27T03:21:56.709141+0000","last_fullsized":"2024-04-27T03:21:56.709141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:56.708942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:56.708942+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.708942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:18.804708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.740116+0000","last_change":"2024-04-27T03:21:39.740168+0000","last_active":"2024-04-27T03:21:39.740116+0000","last_peered":"2024-04-27T03:21:39.740116+0000","last_clean":"2024-04-27T03:21:39.740116+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:21:39.740116+0000","last_undegraded":"2024-04-27T03:21:39.740116+0000","last_fullsized":"2024-04-27T03:21:39.740116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:39.739923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:39.739923+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.739923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:52:10.953090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'48","reported_seq":114,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.092639+0000","last_change":"2024-04-27T03:21:30.092719+0000","last_active":"2024-04-27T03:21:30.092639+0000","last_peered":"2024-04-27T03:21:30.092639+0000","last_clean":"2024-04-27T03:21:30.092639+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:21:30.092639+0000","last_undegraded":"2024-04-27T03:21:30.092639+0000","last_fullsized":"2024-04-27T03:21:30.092639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'48","last_scrub_stamp":"2024-04-27T03:21:30.092441+0000","last_deep_scrub":"31'48","last_deep_scrub_stamp":"2024-04-27T03:21:30.092441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.092441+0000","objects_scrubbed":6,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:22.386772+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":63,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.585696+0000","last_change":"2024-04-27T03:21:23.585759+0000","last_active":"2024-04-27T03:21:23.585696+0000","last_peered":"2024-04-27T03:21:23.585696+0000","last_clean":"2024-04-27T03:21:23.585696+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:21:23.585696+0000","last_undegraded":"2024-04-27T03:21:23.585696+0000","last_fullsized":"2024-04-27T03:21:23.585696+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.585484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:34.196580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.749789+0000","last_change":"2024-04-27T03:17:57.080943+0000","last_active":"2024-04-27T03:21:12.749789+0000","last_peered":"2024-04-27T03:21:12.749789+0000","last_clean":"2024-04-27T03:21:12.749789+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:21:12.749789+0000","last_undegraded":"2024-04-27T03:21:12.749789+0000","last_fullsized":"2024-04-27T03:21:12.749789+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.193531+0000","last_change":"2024-04-27T03:21:48.193588+0000","last_active":"2024-04-27T03:21:48.193531+0000","last_peered":"2024-04-27T03:21:48.193531+0000","last_clean":"2024-04-27T03:21:48.193531+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:21:48.193531+0000","last_undegraded":"2024-04-27T03:21:48.193531+0000","last_fullsized":"2024-04-27T03:21:48.193531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:48.193325+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.193325+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.193325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:25.886685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:05.317248+0000","last_change":"2024-04-27T03:22:05.317248+0000","last_active":"2024-04-27T03:22:05.317248+0000","last_peered":"2024-04-27T03:22:05.317248+0000","last_clean":"2024-04-27T03:22:05.317248+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:22:05.317248+0000","last_undegraded":"2024-04-27T03:22:05.317248+0000","last_fullsized":"2024-04-27T03:22:05.317248+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:05.316823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:05.316823+0000","last_clean_scrub_stamp":"2024-04-27T03:22:05.316823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:59:18.115233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.324797+0000","last_change":"2024-04-27T03:21:12.324853+0000","last_active":"2024-04-27T03:21:12.324797+0000","last_peered":"2024-04-27T03:21:12.324797+0000","last_clean":"2024-04-27T03:21:12.324797+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:21:12.324797+0000","last_undegraded":"2024-04-27T03:21:12.324797+0000","last_fullsized":"2024-04-27T03:21:12.324797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.324599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:41:24.652668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'45","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.786964+0000","last_change":"2024-04-27T03:21:26.787057+0000","last_active":"2024-04-27T03:21:26.786964+0000","last_peered":"2024-04-27T03:21:26.786964+0000","last_clean":"2024-04-27T03:21:26.786964+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:21:26.786964+0000","last_undegraded":"2024-04-27T03:21:26.786964+0000","last_fullsized":"2024-04-27T03:21:26.786964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.786765+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:28.251865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'285","reported_seq":420,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.779843+0000","last_change":"2024-04-27T03:21:18.779964+0000","last_active":"2024-04-27T03:21:18.779843+0000","last_peered":"2024-04-27T03:21:18.779843+0000","last_clean":"2024-04-27T03:21:18.779843+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:18.779843+0000","last_undegraded":"2024-04-27T03:21:18.779843+0000","last_fullsized":"2024-04-27T03:21:18.779843+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.779620+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:58:08.955224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'13","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.591263+0000","last_change":"2024-04-27T03:21:44.591263+0000","last_active":"2024-04-27T03:21:44.591263+0000","last_peered":"2024-04-27T03:21:44.591263+0000","last_clean":"2024-04-27T03:21:44.591263+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:21:44.591263+0000","last_undegraded":"2024-04-27T03:21:44.591263+0000","last_fullsized":"2024-04-27T03:21:44.591263+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-04-27T03:21:44.590873+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-04-27T03:21:44.590873+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.590873+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:31.004553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":180,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.745548+0000","last_change":"2024-04-27T03:21:38.745694+0000","last_active":"2024-04-27T03:21:38.745548+0000","last_peered":"2024-04-27T03:21:38.745548+0000","last_clean":"2024-04-27T03:21:38.745548+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:21:38.745548+0000","last_undegraded":"2024-04-27T03:21:38.745548+0000","last_fullsized":"2024-04-27T03:21:38.745548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'8","last_scrub_stamp":"2024-04-27T03:21:38.745283+0000","last_deep_scrub":"27'8","last_deep_scrub_stamp":"2024-04-27T03:21:38.745283+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.745283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:11:16.380426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.772604+0000","last_change":"2024-04-27T03:21:59.772604+0000","last_active":"2024-04-27T03:21:59.772604+0000","last_peered":"2024-04-27T03:21:59.772604+0000","last_clean":"2024-04-27T03:21:59.772604+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:21:59.772604+0000","last_undegraded":"2024-04-27T03:21:59.772604+0000","last_fullsized":"2024-04-27T03:21:59.772604+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:59.772349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:59.772349+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.772349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:40:42.181739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'5","reported_seq":432,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.080708+0000","last_change":"2024-04-27T03:21:14.080869+0000","last_active":"2024-04-27T03:21:14.080708+0000","last_peered":"2024-04-27T03:21:14.080708+0000","last_clean":"2024-04-27T03:21:14.080708+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:21:14.080708+0000","last_undegraded":"2024-04-27T03:21:14.080708+0000","last_fullsized":"2024-04-27T03:21:14.080708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.080480+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:27.352786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":385,"num_read_kb":385,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"31'16","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.621515+0000","last_change":"2024-04-27T03:21:20.621586+0000","last_active":"2024-04-27T03:21:20.621515+0000","last_peered":"2024-04-27T03:21:20.621515+0000","last_clean":"2024-04-27T03:21:20.621515+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:21:20.621515+0000","last_undegraded":"2024-04-27T03:21:20.621515+0000","last_fullsized":"2024-04-27T03:21:20.621515+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.621330+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:04.933468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'285","reported_seq":476,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.100254+0000","last_change":"2024-04-27T03:21:33.100400+0000","last_active":"2024-04-27T03:21:33.100254+0000","last_peered":"2024-04-27T03:21:33.100254+0000","last_clean":"2024-04-27T03:21:33.100254+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:33.100254+0000","last_undegraded":"2024-04-27T03:21:33.100254+0000","last_fullsized":"2024-04-27T03:21:33.100254+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:33.100022+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:33.100022+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.100022+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:20.619756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.766241+0000","last_change":"2024-04-27T03:21:17.766241+0000","last_active":"2024-04-27T03:21:17.766241+0000","last_peered":"2024-04-27T03:21:17.766241+0000","last_clean":"2024-04-27T03:21:17.766241+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:21:17.766241+0000","last_undegraded":"2024-04-27T03:21:17.766241+0000","last_fullsized":"2024-04-27T03:21:17.766241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.765988+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:49.164397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.500158+0000","last_change":"2024-04-27T03:21:34.500219+0000","last_active":"2024-04-27T03:21:34.500158+0000","last_peered":"2024-04-27T03:21:34.500158+0000","last_clean":"2024-04-27T03:21:34.500158+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:21:34.500158+0000","last_undegraded":"2024-04-27T03:21:34.500158+0000","last_fullsized":"2024-04-27T03:21:34.500158+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:34.499952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:34.499952+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.499952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:49:24.665238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.747473+0000","last_change":"2024-04-27T03:17:47.031891+0000","last_active":"2024-04-27T03:21:12.747473+0000","last_peered":"2024-04-27T03:21:12.747473+0000","last_clean":"2024-04-27T03:21:12.747473+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:21:12.747473+0000","last_undegraded":"2024-04-27T03:21:12.747473+0000","last_fullsized":"2024-04-27T03:21:12.747473+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.628128+0000","last_change":"2024-04-27T03:21:21.628229+0000","last_active":"2024-04-27T03:21:21.628128+0000","last_peered":"2024-04-27T03:21:21.628128+0000","last_clean":"2024-04-27T03:21:21.628128+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:21:21.628128+0000","last_undegraded":"2024-04-27T03:21:21.628128+0000","last_fullsized":"2024-04-27T03:21:21.628128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.627898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:37.847354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'48","reported_seq":111,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.145209+0000","last_change":"2024-04-27T03:21:31.145327+0000","last_active":"2024-04-27T03:21:31.145209+0000","last_peered":"2024-04-27T03:21:31.145209+0000","last_clean":"2024-04-27T03:21:31.145209+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:21:31.145209+0000","last_undegraded":"2024-04-27T03:21:31.145209+0000","last_fullsized":"2024-04-27T03:21:31.145209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'48","last_scrub_stamp":"2024-04-27T03:21:31.145012+0000","last_deep_scrub":"31'48","last_deep_scrub_stamp":"2024-04-27T03:21:31.145012+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.145012+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:50:53.240778+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":37,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.235688+0000","last_change":"2024-04-27T03:21:45.235740+0000","last_active":"2024-04-27T03:21:45.235688+0000","last_peered":"2024-04-27T03:21:45.235688+0000","last_clean":"2024-04-27T03:21:45.235688+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:21:45.235688+0000","last_undegraded":"2024-04-27T03:21:45.235688+0000","last_fullsized":"2024-04-27T03:21:45.235688+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:45.235457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:45.235457+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.235457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:42.520721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'9","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.523804+0000","last_change":"2024-04-27T03:21:36.523804+0000","last_active":"2024-04-27T03:21:36.523804+0000","last_peered":"2024-04-27T03:21:36.523804+0000","last_clean":"2024-04-27T03:21:36.523804+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:36.523804+0000","last_undegraded":"2024-04-27T03:21:36.523804+0000","last_fullsized":"2024-04-27T03:21:36.523804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:36.523556+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:36.523556+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.523556+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:00.443161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.191335+0000","last_change":"2024-04-27T03:21:52.191335+0000","last_active":"2024-04-27T03:21:52.191335+0000","last_peered":"2024-04-27T03:21:52.191335+0000","last_clean":"2024-04-27T03:21:52.191335+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:52.191335+0000","last_undegraded":"2024-04-27T03:21:52.191335+0000","last_fullsized":"2024-04-27T03:21:52.191335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:52.191088+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:52.191088+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.191088+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:24.826476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"31'119","reported_seq":213,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:03.477400+0000","last_change":"2024-04-27T03:22:03.477578+0000","last_active":"2024-04-27T03:22:03.477400+0000","last_peered":"2024-04-27T03:22:03.477400+0000","last_clean":"2024-04-27T03:22:03.477400+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:22:03.477400+0000","last_undegraded":"2024-04-27T03:22:03.477400+0000","last_fullsized":"2024-04-27T03:22:03.477400+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'119","last_scrub_stamp":"2024-04-27T03:22:03.477178+0000","last_deep_scrub":"31'119","last_deep_scrub_stamp":"2024-04-27T03:22:03.477178+0000","last_clean_scrub_stamp":"2024-04-27T03:22:03.477178+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:08:33.020052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406527+0000","last_change":"2024-04-27T03:17:39.634806+0000","last_active":"2024-04-27T03:21:11.406527+0000","last_peered":"2024-04-27T03:21:11.406527+0000","last_clean":"2024-04-27T03:21:11.406527+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:21:11.406527+0000","last_undegraded":"2024-04-27T03:21:11.406527+0000","last_fullsized":"2024-04-27T03:21:11.406527+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.759769+0000","last_change":"2024-04-27T03:22:00.759847+0000","last_active":"2024-04-27T03:22:00.759769+0000","last_peered":"2024-04-27T03:22:00.759769+0000","last_clean":"2024-04-27T03:22:00.759769+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:22:00.759769+0000","last_undegraded":"2024-04-27T03:22:00.759769+0000","last_fullsized":"2024-04-27T03:22:00.759769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:00.759559+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:00.759559+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.759559+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:35:38.674495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.278617+0000","last_change":"2024-04-27T03:21:25.278617+0000","last_active":"2024-04-27T03:21:25.278617+0000","last_peered":"2024-04-27T03:21:25.278617+0000","last_clean":"2024-04-27T03:21:25.278617+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:21:25.278617+0000","last_undegraded":"2024-04-27T03:21:25.278617+0000","last_fullsized":"2024-04-27T03:21:25.278617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.278344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:54.825766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'94","reported_seq":195,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.155217+0000","last_change":"2024-04-27T03:21:28.155217+0000","last_active":"2024-04-27T03:21:28.155217+0000","last_peered":"2024-04-27T03:21:28.155217+0000","last_clean":"2024-04-27T03:21:28.155217+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:21:28.155217+0000","last_undegraded":"2024-04-27T03:21:28.155217+0000","last_fullsized":"2024-04-27T03:21:28.155217+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'94","last_scrub_stamp":"2024-04-27T03:21:28.154943+0000","last_deep_scrub":"31'94","last_deep_scrub_stamp":"2024-04-27T03:21:28.154943+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.154943+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:02.805683+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":81,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.635119+0000","last_change":"2024-04-27T03:21:25.635175+0000","last_active":"2024-04-27T03:21:25.635119+0000","last_peered":"2024-04-27T03:21:25.635119+0000","last_clean":"2024-04-27T03:21:25.635119+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:21:25.635119+0000","last_undegraded":"2024-04-27T03:21:25.635119+0000","last_fullsized":"2024-04-27T03:21:25.635119+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.634932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:40.208378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":176,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.825556+0000","last_change":"2024-04-27T03:21:21.825636+0000","last_active":"2024-04-27T03:21:21.825556+0000","last_peered":"2024-04-27T03:21:21.825556+0000","last_clean":"2024-04-27T03:21:21.825556+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:21.825556+0000","last_undegraded":"2024-04-27T03:21:21.825556+0000","last_fullsized":"2024-04-27T03:21:21.825556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'82","last_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_deep_scrub":"31'82","last_deep_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.825358+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:31:11.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:04.680259+0000","last_change":"2024-04-27T03:22:04.680259+0000","last_active":"2024-04-27T03:22:04.680259+0000","last_peered":"2024-04-27T03:22:04.680259+0000","last_clean":"2024-04-27T03:22:04.680259+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:22:04.680259+0000","last_undegraded":"2024-04-27T03:22:04.680259+0000","last_fullsized":"2024-04-27T03:22:04.680259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:22:04.680009+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:22:04.680009+0000","last_clean_scrub_stamp":"2024-04-27T03:22:04.680009+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:52.086768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406448+0000","last_change":"2024-04-27T03:17:39.637555+0000","last_active":"2024-04-27T03:21:11.406448+0000","last_peered":"2024-04-27T03:21:11.406448+0000","last_clean":"2024-04-27T03:21:11.406448+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:21:11.406448+0000","last_undegraded":"2024-04-27T03:21:11.406448+0000","last_fullsized":"2024-04-27T03:21:11.406448+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:08.381664+0000","last_change":"2024-04-27T03:22:08.381725+0000","last_active":"2024-04-27T03:22:08.381664+0000","last_peered":"2024-04-27T03:22:08.381664+0000","last_clean":"2024-04-27T03:22:08.381664+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:22:08.381664+0000","last_undegraded":"2024-04-27T03:22:08.381664+0000","last_fullsized":"2024-04-27T03:22:08.381664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:08.381446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:08.381446+0000","last_clean_scrub_stamp":"2024-04-27T03:22:08.381446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:13.097476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.174150+0000","last_change":"2024-04-27T03:21:18.174150+0000","last_active":"2024-04-27T03:21:18.174150+0000","last_peered":"2024-04-27T03:21:18.174150+0000","last_clean":"2024-04-27T03:21:18.174150+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:21:18.174150+0000","last_undegraded":"2024-04-27T03:21:18.174150+0000","last_fullsized":"2024-04-27T03:21:18.174150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.173908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:30.769576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'297","reported_seq":534,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.209173+0000","last_change":"2024-04-27T03:21:17.209173+0000","last_active":"2024-04-27T03:21:17.209173+0000","last_peered":"2024-04-27T03:21:17.209173+0000","last_clean":"2024-04-27T03:21:17.209173+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:17.209173+0000","last_undegraded":"2024-04-27T03:21:17.209173+0000","last_fullsized":"2024-04-27T03:21:17.209173+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'297","last_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_deep_scrub":"31'297","last_deep_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.208885+0000","objects_scrubbed":11,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:24.543836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":387,"num_read_kb":371,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.591170+0000","last_change":"2024-04-27T03:21:22.591170+0000","last_active":"2024-04-27T03:21:22.591170+0000","last_peered":"2024-04-27T03:21:22.591170+0000","last_clean":"2024-04-27T03:21:22.591170+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:21:22.591170+0000","last_undegraded":"2024-04-27T03:21:22.591170+0000","last_fullsized":"2024-04-27T03:21:22.591170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.590921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:10.482862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:12.258551+0000","last_change":"2024-04-27T03:22:12.258615+0000","last_active":"2024-04-27T03:22:12.258551+0000","last_peered":"2024-04-27T03:22:12.258551+0000","last_clean":"2024-04-27T03:22:12.258551+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:22:12.258551+0000","last_undegraded":"2024-04-27T03:22:12.258551+0000","last_fullsized":"2024-04-27T03:22:12.258551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'7","last_scrub_stamp":"2024-04-27T03:22:12.258329+0000","last_deep_scrub":"29'7","last_deep_scrub_stamp":"2024-04-27T03:22:12.258329+0000","last_clean_scrub_stamp":"2024-04-27T03:22:12.258329+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:48:04.136414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.558256+0000","last_change":"2024-04-27T03:21:37.558256+0000","last_active":"2024-04-27T03:21:37.558256+0000","last_peered":"2024-04-27T03:21:37.558256+0000","last_clean":"2024-04-27T03:21:37.558256+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:37.558256+0000","last_undegraded":"2024-04-27T03:21:37.558256+0000","last_fullsized":"2024-04-27T03:21:37.558256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:37.557861+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:37.557861+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.557861+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:29.852192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.792507+0000","last_change":"2024-04-27T03:21:47.792561+0000","last_active":"2024-04-27T03:21:47.792507+0000","last_peered":"2024-04-27T03:21:47.792507+0000","last_clean":"2024-04-27T03:21:47.792507+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:21:47.792507+0000","last_undegraded":"2024-04-27T03:21:47.792507+0000","last_fullsized":"2024-04-27T03:21:47.792507+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:47.792322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:47.792322+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.792322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:15:30.180059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:07.414650+0000","last_change":"2024-04-27T03:22:07.414713+0000","last_active":"2024-04-27T03:22:07.414650+0000","last_peered":"2024-04-27T03:22:07.414650+0000","last_clean":"2024-04-27T03:22:07.414650+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:22:07.414650+0000","last_undegraded":"2024-04-27T03:22:07.414650+0000","last_fullsized":"2024-04-27T03:22:07.414650+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:07.414454+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:07.414454+0000","last_clean_scrub_stamp":"2024-04-27T03:22:07.414454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:59.177596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.787965+0000","last_change":"2024-04-27T03:21:41.787965+0000","last_active":"2024-04-27T03:21:41.787965+0000","last_peered":"2024-04-27T03:21:41.787965+0000","last_clean":"2024-04-27T03:21:41.787965+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:21:41.787965+0000","last_undegraded":"2024-04-27T03:21:41.787965+0000","last_fullsized":"2024-04-27T03:21:41.787965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:41.787706+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:41.787706+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.787706+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:24:51.133885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'23","reported_seq":106,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.192744+0000","last_change":"2024-04-27T03:21:27.192744+0000","last_active":"2024-04-27T03:21:27.192744+0000","last_peered":"2024-04-27T03:21:27.192744+0000","last_clean":"2024-04-27T03:21:27.192744+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:27.192744+0000","last_undegraded":"2024-04-27T03:21:27.192744+0000","last_fullsized":"2024-04-27T03:21:27.192744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.192466+0000","objects_scrubbed":8,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:41.765912+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.232354+0000","last_change":"2024-04-27T03:21:16.232354+0000","last_active":"2024-04-27T03:21:16.232354+0000","last_peered":"2024-04-27T03:21:16.232354+0000","last_clean":"2024-04-27T03:21:16.232354+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:21:16.232354+0000","last_undegraded":"2024-04-27T03:21:16.232354+0000","last_fullsized":"2024-04-27T03:21:16.232354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.232084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:05.186600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.711510+0000","last_change":"2024-04-27T03:21:52.711568+0000","last_active":"2024-04-27T03:21:52.711510+0000","last_peered":"2024-04-27T03:21:52.711510+0000","last_clean":"2024-04-27T03:21:52.711510+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:52.711510+0000","last_undegraded":"2024-04-27T03:21:52.711510+0000","last_fullsized":"2024-04-27T03:21:52.711510+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:52.711315+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:52.711315+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.711315+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:37:09.185124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":77,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.564265+0000","last_change":"2024-04-27T03:21:45.564265+0000","last_active":"2024-04-27T03:21:45.564265+0000","last_peered":"2024-04-27T03:21:45.564265+0000","last_clean":"2024-04-27T03:21:45.564265+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:45.564265+0000","last_undegraded":"2024-04-27T03:21:45.564265+0000","last_fullsized":"2024-04-27T03:21:45.564265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:45.564008+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:45.564008+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.564008+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:34:05.340194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:12.360787+0000","last_change":"2024-04-27T03:22:12.360848+0000","last_active":"2024-04-27T03:22:12.360787+0000","last_peered":"2024-04-27T03:22:12.360787+0000","last_clean":"2024-04-27T03:22:12.360787+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:22:12.360787+0000","last_undegraded":"2024-04-27T03:22:12.360787+0000","last_fullsized":"2024-04-27T03:22:12.360787+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:12.360549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:12.360549+0000","last_clean_scrub_stamp":"2024-04-27T03:22:12.360549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.673649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.285465+0000","last_change":"2024-04-27T03:22:02.285533+0000","last_active":"2024-04-27T03:22:02.285465+0000","last_peered":"2024-04-27T03:22:02.285465+0000","last_clean":"2024-04-27T03:22:02.285465+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:22:02.285465+0000","last_undegraded":"2024-04-27T03:22:02.285465+0000","last_fullsized":"2024-04-27T03:22:02.285465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:02.285271+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:02.285271+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.285271+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:45.954095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.367056+0000","last_change":"2024-04-27T03:21:30.367056+0000","last_active":"2024-04-27T03:21:30.367056+0000","last_peered":"2024-04-27T03:21:30.367056+0000","last_clean":"2024-04-27T03:21:30.367056+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:21:30.367056+0000","last_undegraded":"2024-04-27T03:21:30.367056+0000","last_fullsized":"2024-04-27T03:21:30.367056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:30.366712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:30.366712+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.366712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:19:52.016724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"31'21","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.748898+0000","last_change":"2024-04-27T03:21:33.748898+0000","last_active":"2024-04-27T03:21:33.748898+0000","last_peered":"2024-04-27T03:21:33.748898+0000","last_clean":"2024-04-27T03:21:33.748898+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:33.748898+0000","last_undegraded":"2024-04-27T03:21:33.748898+0000","last_fullsized":"2024-04-27T03:21:33.748898+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'21","last_scrub_stamp":"2024-04-27T03:21:33.748366+0000","last_deep_scrub":"31'21","last_deep_scrub_stamp":"2024-04-27T03:21:33.748366+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.748366+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:49.413176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":350,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.816607+0000","last_change":"2024-04-27T03:21:40.816658+0000","last_active":"2024-04-27T03:21:40.816607+0000","last_peered":"2024-04-27T03:21:40.816607+0000","last_clean":"2024-04-27T03:21:40.816607+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:21:40.816607+0000","last_undegraded":"2024-04-27T03:21:40.816607+0000","last_fullsized":"2024-04-27T03:21:40.816607+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:40.816420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:40.816420+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.816420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:44.183666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.968086+0000","last_change":"2024-04-27T03:17:57.089242+0000","last_active":"2024-04-27T03:21:00.968086+0000","last_peered":"2024-04-27T03:21:00.968086+0000","last_clean":"2024-04-27T03:21:00.968086+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:21:00.968086+0000","last_undegraded":"2024-04-27T03:21:00.968086+0000","last_fullsized":"2024-04-27T03:21:00.968086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.437652+0000","last_change":"2024-04-27T03:21:58.437652+0000","last_active":"2024-04-27T03:21:58.437652+0000","last_peered":"2024-04-27T03:21:58.437652+0000","last_clean":"2024-04-27T03:21:58.437652+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:58.437652+0000","last_undegraded":"2024-04-27T03:21:58.437652+0000","last_fullsized":"2024-04-27T03:21:58.437652+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'20","last_scrub_stamp":"2024-04-27T03:21:58.437355+0000","last_deep_scrub":"31'20","last_deep_scrub_stamp":"2024-04-27T03:21:58.437355+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.437355+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:08.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.254082+0000","last_change":"2024-04-27T03:21:59.254142+0000","last_active":"2024-04-27T03:21:59.254082+0000","last_peered":"2024-04-27T03:21:59.254082+0000","last_clean":"2024-04-27T03:21:59.254082+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:21:59.254082+0000","last_undegraded":"2024-04-27T03:21:59.254082+0000","last_fullsized":"2024-04-27T03:21:59.254082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:59.253884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:59.253884+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.253884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:21.043164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:04.467449+0000","last_change":"2024-04-27T03:22:04.467585+0000","last_active":"2024-04-27T03:22:04.467449+0000","last_peered":"2024-04-27T03:22:04.467449+0000","last_clean":"2024-04-27T03:22:04.467449+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:22:04.467449+0000","last_undegraded":"2024-04-27T03:22:04.467449+0000","last_fullsized":"2024-04-27T03:22:04.467449+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:04.467218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:04.467218+0000","last_clean_scrub_stamp":"2024-04-27T03:22:04.467218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:14.140096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.757928+0000","last_change":"2024-04-27T03:21:55.757928+0000","last_active":"2024-04-27T03:21:55.757928+0000","last_peered":"2024-04-27T03:21:55.757928+0000","last_clean":"2024-04-27T03:21:55.757928+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:21:55.757928+0000","last_undegraded":"2024-04-27T03:21:55.757928+0000","last_fullsized":"2024-04-27T03:21:55.757928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:55.757617+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:55.757617+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.757617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:47:48.305961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'87","reported_seq":180,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.199826+0000","last_change":"2024-04-27T03:21:18.199826+0000","last_active":"2024-04-27T03:21:18.199826+0000","last_peered":"2024-04-27T03:21:18.199826+0000","last_clean":"2024-04-27T03:21:18.199826+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:21:18.199826+0000","last_undegraded":"2024-04-27T03:21:18.199826+0000","last_fullsized":"2024-04-27T03:21:18.199826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'87","last_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_deep_scrub":"31'87","last_deep_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.199538+0000","objects_scrubbed":11,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:49.872043+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":89,"num_read_kb":75,"num_write":42,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":292,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.101226+0000","last_change":"2024-04-27T03:21:29.101226+0000","last_active":"2024-04-27T03:21:29.101226+0000","last_peered":"2024-04-27T03:21:29.101226+0000","last_clean":"2024-04-27T03:21:29.101226+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:21:29.101226+0000","last_undegraded":"2024-04-27T03:21:29.101226+0000","last_fullsized":"2024-04-27T03:21:29.101226+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:29.100967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:29.100967+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.100967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:02.862694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.745180+0000","last_change":"2024-04-27T03:21:15.745242+0000","last_active":"2024-04-27T03:21:15.745180+0000","last_peered":"2024-04-27T03:21:15.745180+0000","last_clean":"2024-04-27T03:21:15.745180+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:21:15.745180+0000","last_undegraded":"2024-04-27T03:21:15.745180+0000","last_fullsized":"2024-04-27T03:21:15.745180+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.744992+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:00.286559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.623119+0000","last_change":"2024-04-27T03:21:43.623119+0000","last_active":"2024-04-27T03:21:43.623119+0000","last_peered":"2024-04-27T03:21:43.623119+0000","last_clean":"2024-04-27T03:21:43.623119+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:21:43.623119+0000","last_undegraded":"2024-04-27T03:21:43.623119+0000","last_fullsized":"2024-04-27T03:21:43.623119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:43.622863+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:43.622863+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.622863+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:25.488085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'6","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.510055+0000","last_change":"2024-04-27T03:21:49.510055+0000","last_active":"2024-04-27T03:21:49.510055+0000","last_peered":"2024-04-27T03:21:49.510055+0000","last_clean":"2024-04-27T03:21:49.510055+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:49.510055+0000","last_undegraded":"2024-04-27T03:21:49.510055+0000","last_fullsized":"2024-04-27T03:21:49.510055+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:49.509757+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:49.509757+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.509757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:00.582652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406515+0000","last_change":"2024-04-27T03:17:39.635877+0000","last_active":"2024-04-27T03:21:11.406515+0000","last_peered":"2024-04-27T03:21:11.406515+0000","last_clean":"2024-04-27T03:21:11.406515+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:21:11.406515+0000","last_undegraded":"2024-04-27T03:21:11.406515+0000","last_fullsized":"2024-04-27T03:21:11.406515+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:38.619764+0000","last_clean_scrub_stamp":"2024-04-27T03:17:38.619764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.128384+0000","last_change":"2024-04-27T03:21:16.128384+0000","last_active":"2024-04-27T03:21:16.128384+0000","last_peered":"2024-04-27T03:21:16.128384+0000","last_clean":"2024-04-27T03:21:16.128384+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:21:16.128384+0000","last_undegraded":"2024-04-27T03:21:16.128384+0000","last_fullsized":"2024-04-27T03:21:16.128384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.128135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:44.502862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"31'28","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.261734+0000","last_change":"2024-04-27T03:21:21.261734+0000","last_active":"2024-04-27T03:21:21.261734+0000","last_peered":"2024-04-27T03:21:21.261734+0000","last_clean":"2024-04-27T03:21:21.261734+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:21:21.261734+0000","last_undegraded":"2024-04-27T03:21:21.261734+0000","last_fullsized":"2024-04-27T03:21:21.261734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'28","last_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_deep_scrub":"31'28","last_deep_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.261420+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:52.474587+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'285","reported_seq":374,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.251366+0000","last_change":"2024-04-27T03:21:24.251366+0000","last_active":"2024-04-27T03:21:24.251366+0000","last_peered":"2024-04-27T03:21:24.251366+0000","last_clean":"2024-04-27T03:21:24.251366+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:24.251366+0000","last_undegraded":"2024-04-27T03:21:24.251366+0000","last_fullsized":"2024-04-27T03:21:24.251366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.250979+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:52.625705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.251899+0000","last_change":"2024-04-27T03:17:57.091210+0000","last_active":"2024-04-27T03:21:01.251899+0000","last_peered":"2024-04-27T03:21:01.251899+0000","last_clean":"2024-04-27T03:21:01.251899+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:21:01.251899+0000","last_undegraded":"2024-04-27T03:21:01.251899+0000","last_fullsized":"2024-04-27T03:21:01.251899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:06.636731+0000","last_change":"2024-04-27T03:22:06.636795+0000","last_active":"2024-04-27T03:22:06.636731+0000","last_peered":"2024-04-27T03:22:06.636731+0000","last_clean":"2024-04-27T03:22:06.636731+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:22:06.636731+0000","last_undegraded":"2024-04-27T03:22:06.636731+0000","last_fullsized":"2024-04-27T03:22:06.636731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:06.636531+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:06.636531+0000","last_clean_scrub_stamp":"2024-04-27T03:22:06.636531+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:49.318611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.761324+0000","last_change":"2024-04-27T03:21:56.761324+0000","last_active":"2024-04-27T03:21:56.761324+0000","last_peered":"2024-04-27T03:21:56.761324+0000","last_clean":"2024-04-27T03:21:56.761324+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:21:56.761324+0000","last_undegraded":"2024-04-27T03:21:56.761324+0000","last_fullsized":"2024-04-27T03:21:56.761324+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:56.761068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:56.761068+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.761068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:46.693610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.476853+0000","last_change":"2024-04-27T03:22:02.476905+0000","last_active":"2024-04-27T03:22:02.476853+0000","last_peered":"2024-04-27T03:22:02.476853+0000","last_clean":"2024-04-27T03:22:02.476853+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:22:02.476853+0000","last_undegraded":"2024-04-27T03:22:02.476853+0000","last_fullsized":"2024-04-27T03:22:02.476853+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:02.476650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:02.476650+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.476650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:37:18.950460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.178222+0000","last_change":"2024-04-27T03:21:35.178222+0000","last_active":"2024-04-27T03:21:35.178222+0000","last_peered":"2024-04-27T03:21:35.178222+0000","last_clean":"2024-04-27T03:21:35.178222+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:21:35.178222+0000","last_undegraded":"2024-04-27T03:21:35.178222+0000","last_fullsized":"2024-04-27T03:21:35.178222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:35.177942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:35.177942+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.177942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:50.842753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"31'22","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.688786+0000","last_change":"2024-04-27T03:21:29.688786+0000","last_active":"2024-04-27T03:21:29.688786+0000","last_peered":"2024-04-27T03:21:29.688786+0000","last_clean":"2024-04-27T03:21:29.688786+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:21:29.688786+0000","last_undegraded":"2024-04-27T03:21:29.688786+0000","last_fullsized":"2024-04-27T03:21:29.688786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'22","last_scrub_stamp":"2024-04-27T03:21:29.688480+0000","last_deep_scrub":"31'22","last_deep_scrub_stamp":"2024-04-27T03:21:29.688480+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.688480+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:50:51.784245+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'285","reported_seq":430,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.765171+0000","last_change":"2024-04-27T03:21:36.765171+0000","last_active":"2024-04-27T03:21:36.765171+0000","last_peered":"2024-04-27T03:21:36.765171+0000","last_clean":"2024-04-27T03:21:36.765171+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:36.765171+0000","last_undegraded":"2024-04-27T03:21:36.765171+0000","last_fullsized":"2024-04-27T03:21:36.765171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:36.764897+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:36.764897+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.764897+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:07.152179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:12.731000+0000","last_change":"2024-04-27T03:17:57.087004+0000","last_active":"2024-04-27T03:20:12.731000+0000","last_peered":"2024-04-27T03:20:12.731000+0000","last_clean":"2024-04-27T03:20:12.731000+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:20:12.731000+0000","last_undegraded":"2024-04-27T03:20:12.731000+0000","last_fullsized":"2024-04-27T03:20:12.731000+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":113,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.791930+0000","last_change":"2024-04-27T03:21:44.791930+0000","last_active":"2024-04-27T03:21:44.791930+0000","last_peered":"2024-04-27T03:21:44.791930+0000","last_clean":"2024-04-27T03:21:44.791930+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:21:44.791930+0000","last_undegraded":"2024-04-27T03:21:44.791930+0000","last_fullsized":"2024-04-27T03:21:44.791930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:44.791685+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:44.791685+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.791685+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:12.652507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:05.459487+0000","last_change":"2024-04-27T03:22:05.459487+0000","last_active":"2024-04-27T03:22:05.459487+0000","last_peered":"2024-04-27T03:22:05.459487+0000","last_clean":"2024-04-27T03:22:05.459487+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:22:05.459487+0000","last_undegraded":"2024-04-27T03:22:05.459487+0000","last_fullsized":"2024-04-27T03:22:05.459487+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:05.459218+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:05.459218+0000","last_clean_scrub_stamp":"2024-04-27T03:22:05.459218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:54.776239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:11.379145+0000","last_change":"2024-04-27T03:22:11.379264+0000","last_active":"2024-04-27T03:22:11.379145+0000","last_peered":"2024-04-27T03:22:11.379145+0000","last_clean":"2024-04-27T03:22:11.379145+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:22:11.379145+0000","last_undegraded":"2024-04-27T03:22:11.379145+0000","last_fullsized":"2024-04-27T03:22:11.379145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:11.378920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:11.378920+0000","last_clean_scrub_stamp":"2024-04-27T03:22:11.378920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:04:24.878661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.600253+0000","last_change":"2024-04-27T03:21:19.600253+0000","last_active":"2024-04-27T03:21:19.600253+0000","last_peered":"2024-04-27T03:21:19.600253+0000","last_clean":"2024-04-27T03:21:19.600253+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:21:19.600253+0000","last_undegraded":"2024-04-27T03:21:19.600253+0000","last_fullsized":"2024-04-27T03:21:19.600253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.600004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:53:52.295521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'108","reported_seq":212,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.121323+0000","last_change":"2024-04-27T03:21:15.121323+0000","last_active":"2024-04-27T03:21:15.121323+0000","last_peered":"2024-04-27T03:21:15.121323+0000","last_clean":"2024-04-27T03:21:15.121323+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:15.121323+0000","last_undegraded":"2024-04-27T03:21:15.121323+0000","last_fullsized":"2024-04-27T03:21:15.121323+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'108","last_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_deep_scrub":"31'108","last_deep_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.121034+0000","objects_scrubbed":9,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:20.807083+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4316,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":107,"num_read_kb":100,"num_write":44,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.753050+0000","last_change":"2024-04-27T03:21:44.753050+0000","last_active":"2024-04-27T03:21:44.753050+0000","last_peered":"2024-04-27T03:21:44.753050+0000","last_clean":"2024-04-27T03:21:44.753050+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:21:44.753050+0000","last_undegraded":"2024-04-27T03:21:44.753050+0000","last_fullsized":"2024-04-27T03:21:44.753050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:44.752732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:44.752732+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.752732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:27.679483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.824584+0000","last_change":"2024-04-27T03:21:12.824647+0000","last_active":"2024-04-27T03:21:12.824584+0000","last_peered":"2024-04-27T03:21:12.824584+0000","last_clean":"2024-04-27T03:21:12.824584+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:21:12.824584+0000","last_undegraded":"2024-04-27T03:21:12.824584+0000","last_fullsized":"2024-04-27T03:21:12.824584+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.824385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:28:25.571276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.779972+0000","last_change":"2024-04-27T03:21:14.779972+0000","last_active":"2024-04-27T03:21:14.779972+0000","last_peered":"2024-04-27T03:21:14.779972+0000","last_clean":"2024-04-27T03:21:14.779972+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:21:14.779972+0000","last_undegraded":"2024-04-27T03:21:14.779972+0000","last_fullsized":"2024-04-27T03:21:14.779972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.779705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:11:09.535832+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.696486+0000","last_change":"2024-04-27T03:21:36.696548+0000","last_active":"2024-04-27T03:21:36.696486+0000","last_peered":"2024-04-27T03:21:36.696486+0000","last_clean":"2024-04-27T03:21:36.696486+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:36.696486+0000","last_undegraded":"2024-04-27T03:21:36.696486+0000","last_fullsized":"2024-04-27T03:21:36.696486+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:36.696292+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:36.696292+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.696292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:12:08.368797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.229410+0000","last_change":"2024-04-27T03:21:56.229499+0000","last_active":"2024-04-27T03:21:56.229410+0000","last_peered":"2024-04-27T03:21:56.229410+0000","last_clean":"2024-04-27T03:21:56.229410+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:56.229410+0000","last_undegraded":"2024-04-27T03:21:56.229410+0000","last_fullsized":"2024-04-27T03:21:56.229410+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:56.229215+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:56.229215+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.229215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:37.561633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"31'2","reported_seq":234,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.286650+0000","last_change":"2024-04-27T03:21:26.286722+0000","last_active":"2024-04-27T03:21:26.286650+0000","last_peered":"2024-04-27T03:21:26.286650+0000","last_clean":"2024-04-27T03:21:26.286650+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:21:26.286650+0000","last_undegraded":"2024-04-27T03:21:26.286650+0000","last_fullsized":"2024-04-27T03:21:26.286650+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.286462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:36:54.147283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":285,"num_read_kb":285,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'46","reported_seq":112,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.542394+0000","last_change":"2024-04-27T03:21:15.542505+0000","last_active":"2024-04-27T03:21:15.542394+0000","last_peered":"2024-04-27T03:21:15.542394+0000","last_clean":"2024-04-27T03:21:15.542394+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:21:15.542394+0000","last_undegraded":"2024-04-27T03:21:15.542394+0000","last_fullsized":"2024-04-27T03:21:15.542394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'46","last_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_deep_scrub":"31'46","last_deep_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.542196+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:31.074928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.640360+0000","last_change":"2024-04-27T03:21:28.640422+0000","last_active":"2024-04-27T03:21:28.640360+0000","last_peered":"2024-04-27T03:21:28.640360+0000","last_clean":"2024-04-27T03:21:28.640360+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:21:28.640360+0000","last_undegraded":"2024-04-27T03:21:28.640360+0000","last_fullsized":"2024-04-27T03:21:28.640360+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:28.640146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:28.640146+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.640146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:20.934477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.750056+0000","last_change":"2024-04-27T03:17:57.087180+0000","last_active":"2024-04-27T03:21:12.750056+0000","last_peered":"2024-04-27T03:21:12.750056+0000","last_clean":"2024-04-27T03:21:12.750056+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:21:12.750056+0000","last_undegraded":"2024-04-27T03:21:12.750056+0000","last_fullsized":"2024-04-27T03:21:12.750056+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.536766+0000","last_change":"2024-04-27T03:21:52.536831+0000","last_active":"2024-04-27T03:21:52.536766+0000","last_peered":"2024-04-27T03:21:52.536766+0000","last_clean":"2024-04-27T03:21:52.536766+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:52.536766+0000","last_undegraded":"2024-04-27T03:21:52.536766+0000","last_fullsized":"2024-04-27T03:21:52.536766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:52.536569+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:52.536569+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.536569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:49.821834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.725275+0000","last_change":"2024-04-27T03:21:52.725328+0000","last_active":"2024-04-27T03:21:52.725275+0000","last_peered":"2024-04-27T03:21:52.725275+0000","last_clean":"2024-04-27T03:21:52.725275+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:21:52.725275+0000","last_undegraded":"2024-04-27T03:21:52.725275+0000","last_fullsized":"2024-04-27T03:21:52.725275+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:52.725071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:52.725071+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.725071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:49:42.890358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.749861+0000","last_change":"2024-04-27T03:21:35.749914+0000","last_active":"2024-04-27T03:21:35.749861+0000","last_peered":"2024-04-27T03:21:35.749861+0000","last_clean":"2024-04-27T03:21:35.749861+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:21:35.749861+0000","last_undegraded":"2024-04-27T03:21:35.749861+0000","last_fullsized":"2024-04-27T03:21:35.749861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:35.749664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:35.749664+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.749664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:49:21.865491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'9","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.185318+0000","last_change":"2024-04-27T03:21:19.185455+0000","last_active":"2024-04-27T03:21:19.185318+0000","last_peered":"2024-04-27T03:21:19.185318+0000","last_clean":"2024-04-27T03:21:19.185318+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:19.185318+0000","last_undegraded":"2024-04-27T03:21:19.185318+0000","last_fullsized":"2024-04-27T03:21:19.185318+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.185100+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:12.343058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":250,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.823643+0000","last_change":"2024-04-27T03:21:24.823694+0000","last_active":"2024-04-27T03:21:24.823643+0000","last_peered":"2024-04-27T03:21:24.823643+0000","last_clean":"2024-04-27T03:21:24.823643+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:21:24.823643+0000","last_undegraded":"2024-04-27T03:21:24.823643+0000","last_fullsized":"2024-04-27T03:21:24.823643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.823449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:10:09.300634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.734550+0000","last_change":"2024-04-27T03:17:57.089220+0000","last_active":"2024-04-27T03:20:27.734550+0000","last_peered":"2024-04-27T03:20:27.734550+0000","last_clean":"2024-04-27T03:20:27.734550+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:20:27.734550+0000","last_undegraded":"2024-04-27T03:20:27.734550+0000","last_fullsized":"2024-04-27T03:20:27.734550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.778594+0000","last_change":"2024-04-27T03:22:02.778656+0000","last_active":"2024-04-27T03:22:02.778594+0000","last_peered":"2024-04-27T03:22:02.778594+0000","last_clean":"2024-04-27T03:22:02.778594+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:22:02.778594+0000","last_undegraded":"2024-04-27T03:22:02.778594+0000","last_fullsized":"2024-04-27T03:22:02.778594+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:22:02.778374+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:22:02.778374+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.778374+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:49.192053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:08.355889+0000","last_change":"2024-04-27T03:22:08.355955+0000","last_active":"2024-04-27T03:22:08.355889+0000","last_peered":"2024-04-27T03:22:08.355889+0000","last_clean":"2024-04-27T03:22:08.355889+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:22:08.355889+0000","last_undegraded":"2024-04-27T03:22:08.355889+0000","last_fullsized":"2024-04-27T03:22:08.355889+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:08.355667+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:08.355667+0000","last_clean_scrub_stamp":"2024-04-27T03:22:08.355667+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:26:21.494893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.096198+0000","last_change":"2024-04-27T03:21:12.096234+0000","last_active":"2024-04-27T03:21:12.096198+0000","last_peered":"2024-04-27T03:21:12.096198+0000","last_clean":"2024-04-27T03:21:12.096198+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:21:12.096198+0000","last_undegraded":"2024-04-27T03:21:12.096198+0000","last_fullsized":"2024-04-27T03:21:12.096198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.096079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:20.095840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'47","reported_seq":123,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.666937+0000","last_change":"2024-04-27T03:21:30.667010+0000","last_active":"2024-04-27T03:21:30.666937+0000","last_peered":"2024-04-27T03:21:30.666937+0000","last_clean":"2024-04-27T03:21:30.666937+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:30.666937+0000","last_undegraded":"2024-04-27T03:21:30.666937+0000","last_fullsized":"2024-04-27T03:21:30.666937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-04-27T03:21:30.666704+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-04-27T03:21:30.666704+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.666704+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:05.358118+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":67,"num_read_kb":70,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":662,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.199623+0000","last_change":"2024-04-27T03:21:20.199682+0000","last_active":"2024-04-27T03:21:20.199623+0000","last_peered":"2024-04-27T03:21:20.199623+0000","last_clean":"2024-04-27T03:21:20.199623+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:21:20.199623+0000","last_undegraded":"2024-04-27T03:21:20.199623+0000","last_fullsized":"2024-04-27T03:21:20.199623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.199413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:10:57.834556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:27.745960+0000","last_change":"2024-04-27T03:17:57.086967+0000","last_active":"2024-04-27T03:20:27.745960+0000","last_peered":"2024-04-27T03:20:27.745960+0000","last_clean":"2024-04-27T03:20:27.745960+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:20:27.745960+0000","last_undegraded":"2024-04-27T03:20:27.745960+0000","last_fullsized":"2024-04-27T03:20:27.745960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.664640+0000","last_change":"2024-04-27T03:21:53.664699+0000","last_active":"2024-04-27T03:21:53.664640+0000","last_peered":"2024-04-27T03:21:53.664640+0000","last_clean":"2024-04-27T03:21:53.664640+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:53.664640+0000","last_undegraded":"2024-04-27T03:21:53.664640+0000","last_fullsized":"2024-04-27T03:21:53.664640+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:53.664441+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:53.664441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.664441+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:08:23.207315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.477017+0000","last_change":"2024-04-27T03:21:55.477076+0000","last_active":"2024-04-27T03:21:55.477017+0000","last_peered":"2024-04-27T03:21:55.477017+0000","last_clean":"2024-04-27T03:21:55.477017+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:21:55.477017+0000","last_undegraded":"2024-04-27T03:21:55.477017+0000","last_fullsized":"2024-04-27T03:21:55.477017+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:55.476808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:55.476808+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.476808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:33.170168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.813369+0000","last_change":"2024-04-27T03:21:37.813424+0000","last_active":"2024-04-27T03:21:37.813369+0000","last_peered":"2024-04-27T03:21:37.813369+0000","last_clean":"2024-04-27T03:21:37.813369+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:21:37.813369+0000","last_undegraded":"2024-04-27T03:21:37.813369+0000","last_fullsized":"2024-04-27T03:21:37.813369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:37.813166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:37.813166+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.813166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:26.811307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'37","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.159109+0000","last_change":"2024-04-27T03:21:17.159218+0000","last_active":"2024-04-27T03:21:17.159109+0000","last_peered":"2024-04-27T03:21:17.159109+0000","last_clean":"2024-04-27T03:21:17.159109+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:21:17.159109+0000","last_undegraded":"2024-04-27T03:21:17.159109+0000","last_fullsized":"2024-04-27T03:21:17.159109+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.158898+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:06.103710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.871801+0000","last_change":"2024-04-27T03:21:23.871852+0000","last_active":"2024-04-27T03:21:23.871801+0000","last_peered":"2024-04-27T03:21:23.871801+0000","last_clean":"2024-04-27T03:21:23.871801+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:21:23.871801+0000","last_undegraded":"2024-04-27T03:21:23.871801+0000","last_fullsized":"2024-04-27T03:21:23.871801+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.871608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:14.918299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'2","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.137118+0000","last_change":"2024-04-27T03:21:39.137118+0000","last_active":"2024-04-27T03:21:39.137118+0000","last_peered":"2024-04-27T03:21:39.137118+0000","last_clean":"2024-04-27T03:21:39.137118+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:21:39.137118+0000","last_undegraded":"2024-04-27T03:21:39.137118+0000","last_fullsized":"2024-04-27T03:21:39.137118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:39.136858+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:39.136858+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.136858+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:15.417970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:09.580393+0000","last_change":"2024-04-27T03:22:09.580463+0000","last_active":"2024-04-27T03:22:09.580393+0000","last_peered":"2024-04-27T03:22:09.580393+0000","last_clean":"2024-04-27T03:22:09.580393+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:22:09.580393+0000","last_undegraded":"2024-04-27T03:22:09.580393+0000","last_fullsized":"2024-04-27T03:22:09.580393+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:09.580198+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:09.580198+0000","last_clean_scrub_stamp":"2024-04-27T03:22:09.580198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:08.893299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'5","reported_seq":243,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.496951+0000","last_change":"2024-04-27T03:21:33.496951+0000","last_active":"2024-04-27T03:21:33.496951+0000","last_peered":"2024-04-27T03:21:33.496951+0000","last_clean":"2024-04-27T03:21:33.496951+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:21:33.496951+0000","last_undegraded":"2024-04-27T03:21:33.496951+0000","last_fullsized":"2024-04-27T03:21:33.496951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:21:33.496514+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:21:33.496514+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.496514+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:17:35.575680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":301,"num_read_kb":301,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'72","reported_seq":155,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.270317+0000","last_change":"2024-04-27T03:21:23.270391+0000","last_active":"2024-04-27T03:21:23.270317+0000","last_peered":"2024-04-27T03:21:23.270317+0000","last_clean":"2024-04-27T03:21:23.270317+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:21:23.270317+0000","last_undegraded":"2024-04-27T03:21:23.270317+0000","last_fullsized":"2024-04-27T03:21:23.270317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'72","last_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_deep_scrub":"31'72","last_deep_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.270178+0000","objects_scrubbed":6,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:28.956226+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":28,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.743462+0000","last_change":"2024-04-27T03:21:16.743518+0000","last_active":"2024-04-27T03:21:16.743462+0000","last_peered":"2024-04-27T03:21:16.743462+0000","last_clean":"2024-04-27T03:21:16.743462+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:21:16.743462+0000","last_undegraded":"2024-04-27T03:21:16.743462+0000","last_fullsized":"2024-04-27T03:21:16.743462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.743267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:34:18.284825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.724167+0000","last_change":"2024-04-27T03:21:48.724227+0000","last_active":"2024-04-27T03:21:48.724167+0000","last_peered":"2024-04-27T03:21:48.724167+0000","last_clean":"2024-04-27T03:21:48.724167+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:21:48.724167+0000","last_undegraded":"2024-04-27T03:21:48.724167+0000","last_fullsized":"2024-04-27T03:21:48.724167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:48.723971+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:48.723971+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.723971+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:58.946837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.540808+0000","last_change":"2024-04-27T03:21:46.540808+0000","last_active":"2024-04-27T03:21:46.540808+0000","last_peered":"2024-04-27T03:21:46.540808+0000","last_clean":"2024-04-27T03:21:46.540808+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:21:46.540808+0000","last_undegraded":"2024-04-27T03:21:46.540808+0000","last_fullsized":"2024-04-27T03:21:46.540808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:46.540543+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:46.540543+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.540543+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:16.927826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.429062+0000","last_change":"2024-04-27T03:22:01.429118+0000","last_active":"2024-04-27T03:22:01.429062+0000","last_peered":"2024-04-27T03:22:01.429062+0000","last_clean":"2024-04-27T03:22:01.429062+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:22:01.429062+0000","last_undegraded":"2024-04-27T03:22:01.429062+0000","last_fullsized":"2024-04-27T03:22:01.429062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:01.428879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:01.428879+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.428879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:02:51.699431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.193806+0000","last_change":"2024-04-27T03:21:20.193806+0000","last_active":"2024-04-27T03:21:20.193806+0000","last_peered":"2024-04-27T03:21:20.193806+0000","last_clean":"2024-04-27T03:21:20.193806+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:21:20.193806+0000","last_undegraded":"2024-04-27T03:21:20.193806+0000","last_fullsized":"2024-04-27T03:21:20.193806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.193546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:27:42.782149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'45","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.064594+0000","last_change":"2024-04-27T03:21:13.064640+0000","last_active":"2024-04-27T03:21:13.064594+0000","last_peered":"2024-04-27T03:21:13.064594+0000","last_clean":"2024-04-27T03:21:13.064594+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:21:13.064594+0000","last_undegraded":"2024-04-27T03:21:13.064594+0000","last_fullsized":"2024-04-27T03:21:13.064594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.064511+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:06:47.923690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.302195+0000","last_change":"2024-04-27T03:21:14.302256+0000","last_active":"2024-04-27T03:21:14.302195+0000","last_peered":"2024-04-27T03:21:14.302195+0000","last_clean":"2024-04-27T03:21:14.302195+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:21:14.302195+0000","last_undegraded":"2024-04-27T03:21:14.302195+0000","last_fullsized":"2024-04-27T03:21:14.302195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.302007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:00.292019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747121+0000","last_change":"2024-04-27T03:17:57.070067+0000","last_active":"2024-04-27T03:20:17.747121+0000","last_peered":"2024-04-27T03:20:17.747121+0000","last_clean":"2024-04-27T03:20:17.747121+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:20:17.747121+0000","last_undegraded":"2024-04-27T03:20:17.747121+0000","last_fullsized":"2024-04-27T03:20:17.747121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.744246+0000","last_change":"2024-04-27T03:22:02.744298+0000","last_active":"2024-04-27T03:22:02.744246+0000","last_peered":"2024-04-27T03:22:02.744246+0000","last_clean":"2024-04-27T03:22:02.744246+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:22:02.744246+0000","last_undegraded":"2024-04-27T03:22:02.744246+0000","last_fullsized":"2024-04-27T03:22:02.744246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:02.744043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:02.744043+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.744043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:54.622128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:07.343260+0000","last_change":"2024-04-27T03:22:07.343331+0000","last_active":"2024-04-27T03:22:07.343260+0000","last_peered":"2024-04-27T03:22:07.343260+0000","last_clean":"2024-04-27T03:22:07.343260+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:22:07.343260+0000","last_undegraded":"2024-04-27T03:22:07.343260+0000","last_fullsized":"2024-04-27T03:22:07.343260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:07.343061+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:07.343061+0000","last_clean_scrub_stamp":"2024-04-27T03:22:07.343061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:54.749820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.231515+0000","last_change":"2024-04-27T03:21:22.231634+0000","last_active":"2024-04-27T03:21:22.231515+0000","last_peered":"2024-04-27T03:21:22.231515+0000","last_clean":"2024-04-27T03:21:22.231515+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:21:22.231515+0000","last_undegraded":"2024-04-27T03:21:22.231515+0000","last_fullsized":"2024-04-27T03:21:22.231515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.231282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:38.450738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'47","reported_seq":114,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.572535+0000","last_change":"2024-04-27T03:21:16.572535+0000","last_active":"2024-04-27T03:21:16.572535+0000","last_peered":"2024-04-27T03:21:16.572535+0000","last_clean":"2024-04-27T03:21:16.572535+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:21:16.572535+0000","last_undegraded":"2024-04-27T03:21:16.572535+0000","last_fullsized":"2024-04-27T03:21:16.572535+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.572262+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:05.517074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.304119+0000","last_change":"2024-04-27T03:21:28.304119+0000","last_active":"2024-04-27T03:21:28.304119+0000","last_peered":"2024-04-27T03:21:28.304119+0000","last_clean":"2024-04-27T03:21:28.304119+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:21:28.304119+0000","last_undegraded":"2024-04-27T03:21:28.304119+0000","last_fullsized":"2024-04-27T03:21:28.304119+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:28.303861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:28.303861+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.303861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:26.889521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747057+0000","last_change":"2024-04-27T03:17:57.066218+0000","last_active":"2024-04-27T03:20:17.747057+0000","last_peered":"2024-04-27T03:20:17.747057+0000","last_clean":"2024-04-27T03:20:17.747057+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:20:17.747057+0000","last_undegraded":"2024-04-27T03:20:17.747057+0000","last_fullsized":"2024-04-27T03:20:17.747057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.698657+0000","last_change":"2024-04-27T03:21:35.698721+0000","last_active":"2024-04-27T03:21:35.698657+0000","last_peered":"2024-04-27T03:21:35.698657+0000","last_clean":"2024-04-27T03:21:35.698657+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:21:35.698657+0000","last_undegraded":"2024-04-27T03:21:35.698657+0000","last_fullsized":"2024-04-27T03:21:35.698657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:35.698441+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:35.698441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.698441+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:43.033392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.743118+0000","last_change":"2024-04-27T03:17:47.018817+0000","last_active":"2024-04-27T03:20:17.743118+0000","last_peered":"2024-04-27T03:20:17.743118+0000","last_clean":"2024-04-27T03:20:17.743118+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:20:17.743118+0000","last_undegraded":"2024-04-27T03:20:17.743118+0000","last_fullsized":"2024-04-27T03:20:17.743118+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.301687+0000","last_change":"2024-04-27T03:21:27.301740+0000","last_active":"2024-04-27T03:21:27.301687+0000","last_peered":"2024-04-27T03:21:27.301687+0000","last_clean":"2024-04-27T03:21:27.301687+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:21:27.301687+0000","last_undegraded":"2024-04-27T03:21:27.301687+0000","last_fullsized":"2024-04-27T03:21:27.301687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.301481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:33.928268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"31'22","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.677168+0000","last_change":"2024-04-27T03:21:31.677168+0000","last_active":"2024-04-27T03:21:31.677168+0000","last_peered":"2024-04-27T03:21:31.677168+0000","last_clean":"2024-04-27T03:21:31.677168+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:21:31.677168+0000","last_undegraded":"2024-04-27T03:21:31.677168+0000","last_fullsized":"2024-04-27T03:21:31.677168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'22","last_scrub_stamp":"2024-04-27T03:21:31.676892+0000","last_deep_scrub":"31'22","last_deep_scrub_stamp":"2024-04-27T03:21:31.676892+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.676892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:19.196626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'285","reported_seq":393,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.185168+0000","last_change":"2024-04-27T03:21:25.185168+0000","last_active":"2024-04-27T03:21:25.185168+0000","last_peered":"2024-04-27T03:21:25.185168+0000","last_clean":"2024-04-27T03:21:25.185168+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:25.185168+0000","last_undegraded":"2024-04-27T03:21:25.185168+0000","last_fullsized":"2024-04-27T03:21:25.185168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.184906+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:35.796002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":22,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.746971+0000","last_change":"2024-04-27T03:17:57.084816+0000","last_active":"2024-04-27T03:20:17.746971+0000","last_peered":"2024-04-27T03:20:17.746971+0000","last_clean":"2024-04-27T03:20:17.746971+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:20:17.746971+0000","last_undegraded":"2024-04-27T03:20:17.746971+0000","last_fullsized":"2024-04-27T03:20:17.746971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.686144+0000","last_change":"2024-04-27T03:21:32.686210+0000","last_active":"2024-04-27T03:21:32.686144+0000","last_peered":"2024-04-27T03:21:32.686144+0000","last_clean":"2024-04-27T03:21:32.686144+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:32.686144+0000","last_undegraded":"2024-04-27T03:21:32.686144+0000","last_fullsized":"2024-04-27T03:21:32.686144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:32.685934+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:32.685934+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.685934+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:18.675946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.498152+0000","last_change":"2024-04-27T03:21:51.498216+0000","last_active":"2024-04-27T03:21:51.498152+0000","last_peered":"2024-04-27T03:21:51.498152+0000","last_clean":"2024-04-27T03:21:51.498152+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:21:51.498152+0000","last_undegraded":"2024-04-27T03:21:51.498152+0000","last_fullsized":"2024-04-27T03:21:51.498152+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'4","last_scrub_stamp":"2024-04-27T03:21:51.497953+0000","last_deep_scrub":"27'4","last_deep_scrub_stamp":"2024-04-27T03:21:51.497953+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.497953+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:38.452327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.332423+0000","last_change":"2024-04-27T03:21:13.332476+0000","last_active":"2024-04-27T03:21:13.332423+0000","last_peered":"2024-04-27T03:21:13.332423+0000","last_clean":"2024-04-27T03:21:13.332423+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:21:13.332423+0000","last_undegraded":"2024-04-27T03:21:13.332423+0000","last_fullsized":"2024-04-27T03:21:13.332423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.332213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:31.014208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'73","reported_seq":159,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.145202+0000","last_change":"2024-04-27T03:21:32.145296+0000","last_active":"2024-04-27T03:21:32.145202+0000","last_peered":"2024-04-27T03:21:32.145202+0000","last_clean":"2024-04-27T03:21:32.145202+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:21:32.145202+0000","last_undegraded":"2024-04-27T03:21:32.145202+0000","last_fullsized":"2024-04-27T03:21:32.145202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'73","last_scrub_stamp":"2024-04-27T03:21:32.145018+0000","last_deep_scrub":"31'73","last_deep_scrub_stamp":"2024-04-27T03:21:32.145018+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.145018+0000","objects_scrubbed":11,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:06.836432+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":44,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.654794+0000","last_change":"2024-04-27T03:21:26.654845+0000","last_active":"2024-04-27T03:21:26.654794+0000","last_peered":"2024-04-27T03:21:26.654794+0000","last_clean":"2024-04-27T03:21:26.654794+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:21:26.654794+0000","last_undegraded":"2024-04-27T03:21:26.654794+0000","last_fullsized":"2024-04-27T03:21:26.654794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.654609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:01.305349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.819801+0000","last_change":"2024-04-27T03:17:57.066613+0000","last_active":"2024-04-27T03:21:00.819801+0000","last_peered":"2024-04-27T03:21:00.819801+0000","last_clean":"2024-04-27T03:21:00.819801+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:21:00.819801+0000","last_undegraded":"2024-04-27T03:21:00.819801+0000","last_fullsized":"2024-04-27T03:21:00.819801+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.195754+0000","last_change":"2024-04-27T03:21:44.195813+0000","last_active":"2024-04-27T03:21:44.195754+0000","last_peered":"2024-04-27T03:21:44.195754+0000","last_clean":"2024-04-27T03:21:44.195754+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:44.195754+0000","last_undegraded":"2024-04-27T03:21:44.195754+0000","last_fullsized":"2024-04-27T03:21:44.195754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:44.195561+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:44.195561+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.195561+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:31.149936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.232928+0000","last_change":"2024-04-27T03:21:58.232983+0000","last_active":"2024-04-27T03:21:58.232928+0000","last_peered":"2024-04-27T03:21:58.232928+0000","last_clean":"2024-04-27T03:21:58.232928+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:21:58.232928+0000","last_undegraded":"2024-04-27T03:21:58.232928+0000","last_fullsized":"2024-04-27T03:21:58.232928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:58.232714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:58.232714+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.232714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:14:05.190857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.156966+0000","last_change":"2024-04-27T03:21:19.157032+0000","last_active":"2024-04-27T03:21:19.156966+0000","last_peered":"2024-04-27T03:21:19.156966+0000","last_clean":"2024-04-27T03:21:19.156966+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:21:19.156966+0000","last_undegraded":"2024-04-27T03:21:19.156966+0000","last_fullsized":"2024-04-27T03:21:19.156966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.156758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:49.521232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'45","reported_seq":110,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.820346+0000","last_change":"2024-04-27T03:21:27.820426+0000","last_active":"2024-04-27T03:21:27.820346+0000","last_peered":"2024-04-27T03:21:27.820346+0000","last_clean":"2024-04-27T03:21:27.820346+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:21:27.820346+0000","last_undegraded":"2024-04-27T03:21:27.820346+0000","last_fullsized":"2024-04-27T03:21:27.820346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:27.820155+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:27.820155+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.820155+0000","objects_scrubbed":5,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:39:02.307136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.175376+0000","last_change":"2024-04-27T03:21:24.175430+0000","last_active":"2024-04-27T03:21:24.175376+0000","last_peered":"2024-04-27T03:21:24.175376+0000","last_clean":"2024-04-27T03:21:24.175376+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:21:24.175376+0000","last_undegraded":"2024-04-27T03:21:24.175376+0000","last_fullsized":"2024-04-27T03:21:24.175376+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.175164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:23.877311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'2","reported_seq":24,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:01.480429+0000","last_change":"2024-04-27T03:17:57.087706+0000","last_active":"2024-04-27T03:21:01.480429+0000","last_peered":"2024-04-27T03:21:01.480429+0000","last_clean":"2024-04-27T03:21:01.480429+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:21:01.480429+0000","last_undegraded":"2024-04-27T03:21:01.480429+0000","last_fullsized":"2024-04-27T03:21:01.480429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:00.962890+0000","last_change":"2024-04-27T03:17:49.011902+0000","last_active":"2024-04-27T03:21:00.962890+0000","last_peered":"2024-04-27T03:21:00.962890+0000","last_clean":"2024-04-27T03:21:00.962890+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:21:00.962890+0000","last_undegraded":"2024-04-27T03:21:00.962890+0000","last_fullsized":"2024-04-27T03:21:00.962890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:47.975250+0000","last_clean_scrub_stamp":"2024-04-27T03:17:47.975250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:12.573758+0000","last_change":"2024-04-27T03:22:12.573823+0000","last_active":"2024-04-27T03:22:12.573758+0000","last_peered":"2024-04-27T03:22:12.573758+0000","last_clean":"2024-04-27T03:22:12.573758+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:22:12.573758+0000","last_undegraded":"2024-04-27T03:22:12.573758+0000","last_fullsized":"2024-04-27T03:22:12.573758+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:12.573531+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:12.573531+0000","last_clean_scrub_stamp":"2024-04-27T03:22:12.573531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:10:56.304235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.336891+0000","last_change":"2024-04-27T03:21:29.336949+0000","last_active":"2024-04-27T03:21:29.336891+0000","last_peered":"2024-04-27T03:21:29.336891+0000","last_clean":"2024-04-27T03:21:29.336891+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:21:29.336891+0000","last_undegraded":"2024-04-27T03:21:29.336891+0000","last_fullsized":"2024-04-27T03:21:29.336891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:29.336698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:29.336698+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.336698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:15:11.724434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'37","reported_seq":100,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.538736+0000","last_change":"2024-04-27T03:21:14.538808+0000","last_active":"2024-04-27T03:21:14.538736+0000","last_peered":"2024-04-27T03:21:14.538736+0000","last_clean":"2024-04-27T03:21:14.538736+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:21:14.538736+0000","last_undegraded":"2024-04-27T03:21:14.538736+0000","last_fullsized":"2024-04-27T03:21:14.538736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.538542+0000","objects_scrubbed":3,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:32.436126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.189216+0000","last_change":"2024-04-27T03:21:41.189268+0000","last_active":"2024-04-27T03:21:41.189216+0000","last_peered":"2024-04-27T03:21:41.189216+0000","last_clean":"2024-04-27T03:21:41.189216+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:21:41.189216+0000","last_undegraded":"2024-04-27T03:21:41.189216+0000","last_fullsized":"2024-04-27T03:21:41.189216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:41.189019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:41.189019+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.189019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:28.342309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":26,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:22.744683+0000","last_change":"2024-04-27T03:17:57.086785+0000","last_active":"2024-04-27T03:20:22.744683+0000","last_peered":"2024-04-27T03:20:22.744683+0000","last_clean":"2024-04-27T03:20:22.744683+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:20:22.744683+0000","last_undegraded":"2024-04-27T03:20:22.744683+0000","last_fullsized":"2024-04-27T03:20:22.744683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'16","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.184766+0000","last_change":"2024-04-27T03:21:51.184831+0000","last_active":"2024-04-27T03:21:51.184766+0000","last_peered":"2024-04-27T03:21:51.184766+0000","last_clean":"2024-04-27T03:21:51.184766+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:51.184766+0000","last_undegraded":"2024-04-27T03:21:51.184766+0000","last_fullsized":"2024-04-27T03:21:51.184766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:51.184588+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:51.184588+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.184588+0000","objects_scrubbed":2,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:01.407454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:06.456721+0000","last_change":"2024-04-27T03:22:06.456783+0000","last_active":"2024-04-27T03:22:06.456721+0000","last_peered":"2024-04-27T03:22:06.456721+0000","last_clean":"2024-04-27T03:22:06.456721+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:22:06.456721+0000","last_undegraded":"2024-04-27T03:22:06.456721+0000","last_fullsized":"2024-04-27T03:22:06.456721+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:06.456530+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:06.456530+0000","last_clean_scrub_stamp":"2024-04-27T03:22:06.456530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:56.389072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'3","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.767122+0000","last_change":"2024-04-27T03:22:01.767186+0000","last_active":"2024-04-27T03:22:01.767122+0000","last_peered":"2024-04-27T03:22:01.767122+0000","last_clean":"2024-04-27T03:22:01.767122+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:22:01.767122+0000","last_undegraded":"2024-04-27T03:22:01.767122+0000","last_fullsized":"2024-04-27T03:22:01.767122+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-04-27T03:22:01.766938+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-04-27T03:22:01.766938+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.766938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:43.632161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'53","reported_seq":126,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.140107+0000","last_change":"2024-04-27T03:21:34.140183+0000","last_active":"2024-04-27T03:21:34.140107+0000","last_peered":"2024-04-27T03:21:34.140107+0000","last_clean":"2024-04-27T03:21:34.140107+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:34.140107+0000","last_undegraded":"2024-04-27T03:21:34.140107+0000","last_fullsized":"2024-04-27T03:21:34.140107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'53","last_scrub_stamp":"2024-04-27T03:21:34.139891+0000","last_deep_scrub":"31'53","last_deep_scrub_stamp":"2024-04-27T03:21:34.139891+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.139891+0000","objects_scrubbed":7,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:44.594383+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":52,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1138,"num_omap_keys":6,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.822080+0000","last_change":"2024-04-27T03:21:19.822219+0000","last_active":"2024-04-27T03:21:19.822080+0000","last_peered":"2024-04-27T03:21:19.822080+0000","last_clean":"2024-04-27T03:21:19.822080+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:21:19.822080+0000","last_undegraded":"2024-04-27T03:21:19.822080+0000","last_fullsized":"2024-04-27T03:21:19.822080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.821853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:18.101674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:17.747256+0000","last_change":"2024-04-27T03:17:57.061982+0000","last_active":"2024-04-27T03:20:17.747256+0000","last_peered":"2024-04-27T03:20:17.747256+0000","last_clean":"2024-04-27T03:20:17.747256+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:20:17.747256+0000","last_undegraded":"2024-04-27T03:20:17.747256+0000","last_fullsized":"2024-04-27T03:20:17.747256+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:56.032783+0000","last_clean_scrub_stamp":"2024-04-27T03:17:56.032783+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.475670+0000","last_change":"2024-04-27T03:21:48.475670+0000","last_active":"2024-04-27T03:21:48.475670+0000","last_peered":"2024-04-27T03:21:48.475670+0000","last_clean":"2024-04-27T03:21:48.475670+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:21:48.475670+0000","last_undegraded":"2024-04-27T03:21:48.475670+0000","last_fullsized":"2024-04-27T03:21:48.475670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:48.475413+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.475413+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.475413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:40:05.549233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.748753+0000","last_change":"2024-04-27T03:21:46.748753+0000","last_active":"2024-04-27T03:21:46.748753+0000","last_peered":"2024-04-27T03:21:46.748753+0000","last_clean":"2024-04-27T03:21:46.748753+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:21:46.748753+0000","last_undegraded":"2024-04-27T03:21:46.748753+0000","last_fullsized":"2024-04-27T03:21:46.748753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:46.748445+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:46.748445+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.748445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:45.334105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.624365+0000","last_change":"2024-04-27T03:21:41.624365+0000","last_active":"2024-04-27T03:21:41.624365+0000","last_peered":"2024-04-27T03:21:41.624365+0000","last_clean":"2024-04-27T03:21:41.624365+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:21:41.624365+0000","last_undegraded":"2024-04-27T03:21:41.624365+0000","last_fullsized":"2024-04-27T03:21:41.624365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:41.624007+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:41.624007+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.624007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:52.078499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'2","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.694164+0000","last_change":"2024-04-27T03:21:57.694164+0000","last_active":"2024-04-27T03:21:57.694164+0000","last_peered":"2024-04-27T03:21:57.694164+0000","last_clean":"2024-04-27T03:21:57.694164+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:21:57.694164+0000","last_undegraded":"2024-04-27T03:21:57.694164+0000","last_fullsized":"2024-04-27T03:21:57.694164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:57.693878+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:57.693878+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.693878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:14.849137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.706156+0000","last_change":"2024-04-27T03:21:59.706276+0000","last_active":"2024-04-27T03:21:59.706156+0000","last_peered":"2024-04-27T03:21:59.706156+0000","last_clean":"2024-04-27T03:21:59.706156+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:21:59.706156+0000","last_undegraded":"2024-04-27T03:21:59.706156+0000","last_fullsized":"2024-04-27T03:21:59.706156+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:59.705879+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:59.705879+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.705879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:41:32.982953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:03.726162+0000","last_change":"2024-04-27T03:22:03.726223+0000","last_active":"2024-04-27T03:22:03.726162+0000","last_peered":"2024-04-27T03:22:03.726162+0000","last_clean":"2024-04-27T03:22:03.726162+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:22:03.726162+0000","last_undegraded":"2024-04-27T03:22:03.726162+0000","last_fullsized":"2024-04-27T03:22:03.726162+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:03.725962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:03.725962+0000","last_clean_scrub_stamp":"2024-04-27T03:22:03.725962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:20:58.721329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.518519+0000","last_change":"2024-04-27T03:21:35.518519+0000","last_active":"2024-04-27T03:21:35.518519+0000","last_peered":"2024-04-27T03:21:35.518519+0000","last_clean":"2024-04-27T03:21:35.518519+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:21:35.518519+0000","last_undegraded":"2024-04-27T03:21:35.518519+0000","last_fullsized":"2024-04-27T03:21:35.518519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'6","last_scrub_stamp":"2024-04-27T03:21:35.518220+0000","last_deep_scrub":"27'6","last_deep_scrub_stamp":"2024-04-27T03:21:35.518220+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.518220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:50.091666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.732999+0000","last_change":"2024-04-27T03:21:49.732999+0000","last_active":"2024-04-27T03:21:49.732999+0000","last_peered":"2024-04-27T03:21:49.732999+0000","last_clean":"2024-04-27T03:21:49.732999+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:21:49.732999+0000","last_undegraded":"2024-04-27T03:21:49.732999+0000","last_fullsized":"2024-04-27T03:21:49.732999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'4","last_scrub_stamp":"2024-04-27T03:21:49.732701+0000","last_deep_scrub":"29'4","last_deep_scrub_stamp":"2024-04-27T03:21:49.732701+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.732701+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:41.379770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.819841+0000","last_change":"2024-04-27T03:21:42.819841+0000","last_active":"2024-04-27T03:21:42.819841+0000","last_peered":"2024-04-27T03:21:42.819841+0000","last_clean":"2024-04-27T03:21:42.819841+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:42.819841+0000","last_undegraded":"2024-04-27T03:21:42.819841+0000","last_fullsized":"2024-04-27T03:21:42.819841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:42.819561+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:42.819561+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.819561+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:40.104826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.709102+0000","last_change":"2024-04-27T03:21:54.709102+0000","last_active":"2024-04-27T03:21:54.709102+0000","last_peered":"2024-04-27T03:21:54.709102+0000","last_clean":"2024-04-27T03:21:54.709102+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:21:54.709102+0000","last_undegraded":"2024-04-27T03:21:54.709102+0000","last_fullsized":"2024-04-27T03:21:54.709102+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:54.708841+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:54.708841+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.708841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:04.381720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.443313+0000","last_change":"2024-04-27T03:22:00.443313+0000","last_active":"2024-04-27T03:22:00.443313+0000","last_peered":"2024-04-27T03:22:00.443313+0000","last_clean":"2024-04-27T03:22:00.443313+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:22:00.443313+0000","last_undegraded":"2024-04-27T03:22:00.443313+0000","last_fullsized":"2024-04-27T03:22:00.443313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:22:00.442911+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:22:00.442911+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.442911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:03.560173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.515012+0000","last_change":"2024-04-27T03:21:47.515012+0000","last_active":"2024-04-27T03:21:47.515012+0000","last_peered":"2024-04-27T03:21:47.515012+0000","last_clean":"2024-04-27T03:21:47.515012+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:21:47.515012+0000","last_undegraded":"2024-04-27T03:21:47.515012+0000","last_fullsized":"2024-04-27T03:21:47.515012+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:47.514720+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:47.514720+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.514720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:36.512861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.711428+0000","last_change":"2024-04-27T03:21:50.711428+0000","last_active":"2024-04-27T03:21:50.711428+0000","last_peered":"2024-04-27T03:21:50.711428+0000","last_clean":"2024-04-27T03:21:50.711428+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:50.711428+0000","last_undegraded":"2024-04-27T03:21:50.711428+0000","last_fullsized":"2024-04-27T03:21:50.711428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:50.710962+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:50.710962+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.710962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:53.828224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'4","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.712671+0000","last_change":"2024-04-27T03:21:53.712671+0000","last_active":"2024-04-27T03:21:53.712671+0000","last_peered":"2024-04-27T03:21:53.712671+0000","last_clean":"2024-04-27T03:21:53.712671+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:53.712671+0000","last_undegraded":"2024-04-27T03:21:53.712671+0000","last_fullsized":"2024-04-27T03:21:53.712671+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:53.712385+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:53.712385+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.712385+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:15.090838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.226679+0000","last_change":"2024-04-27T03:21:46.226679+0000","last_active":"2024-04-27T03:21:46.226679+0000","last_peered":"2024-04-27T03:21:46.226679+0000","last_clean":"2024-04-27T03:21:46.226679+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:46.226679+0000","last_undegraded":"2024-04-27T03:21:46.226679+0000","last_fullsized":"2024-04-27T03:21:46.226679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:46.226388+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:46.226388+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.226388+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:18.698646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.730514+0000","last_change":"2024-04-27T03:21:55.730514+0000","last_active":"2024-04-27T03:21:55.730514+0000","last_peered":"2024-04-27T03:21:55.730514+0000","last_clean":"2024-04-27T03:21:55.730514+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:21:55.730514+0000","last_undegraded":"2024-04-27T03:21:55.730514+0000","last_fullsized":"2024-04-27T03:21:55.730514+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:55.730240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:55.730240+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.730240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:45.047262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.643560+0000","last_change":"2024-04-27T03:21:39.643685+0000","last_active":"2024-04-27T03:21:39.643560+0000","last_peered":"2024-04-27T03:21:39.643560+0000","last_clean":"2024-04-27T03:21:39.643560+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:39.643560+0000","last_undegraded":"2024-04-27T03:21:39.643560+0000","last_fullsized":"2024-04-27T03:21:39.643560+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:39.643340+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:39.643340+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.643340+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:56.798600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:11.552408+0000","last_change":"2024-04-27T03:22:11.552473+0000","last_active":"2024-04-27T03:22:11.552408+0000","last_peered":"2024-04-27T03:22:11.552408+0000","last_clean":"2024-04-27T03:22:11.552408+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:22:11.552408+0000","last_undegraded":"2024-04-27T03:22:11.552408+0000","last_fullsized":"2024-04-27T03:22:11.552408+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:11.552206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:11.552206+0000","last_clean_scrub_stamp":"2024-04-27T03:22:11.552206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:59.839853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":84,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:08.617520+0000","last_change":"2024-04-27T03:22:08.617583+0000","last_active":"2024-04-27T03:22:08.617520+0000","last_peered":"2024-04-27T03:22:08.617520+0000","last_clean":"2024-04-27T03:22:08.617520+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:22:08.617520+0000","last_undegraded":"2024-04-27T03:22:08.617520+0000","last_fullsized":"2024-04-27T03:22:08.617520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:22:08.617332+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:22:08.617332+0000","last_clean_scrub_stamp":"2024-04-27T03:22:08.617332+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:04:22.117072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:02.738611+0000","last_change":"2024-04-27T03:17:47.019986+0000","last_active":"2024-04-27T03:20:02.738611+0000","last_peered":"2024-04-27T03:20:02.738611+0000","last_clean":"2024-04-27T03:20:02.738611+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:20:02.738611+0000","last_undegraded":"2024-04-27T03:20:02.738611+0000","last_fullsized":"2024-04-27T03:20:02.738611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.767018+0000","last_change":"2024-04-27T03:21:42.767075+0000","last_active":"2024-04-27T03:21:42.767018+0000","last_peered":"2024-04-27T03:21:42.767018+0000","last_clean":"2024-04-27T03:21:42.767018+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:42.767018+0000","last_undegraded":"2024-04-27T03:21:42.767018+0000","last_fullsized":"2024-04-27T03:21:42.767018+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:42.766832+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:42.766832+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.766832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:32:00.664417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.196097+0000","last_change":"2024-04-27T03:21:55.196227+0000","last_active":"2024-04-27T03:21:55.196097+0000","last_peered":"2024-04-27T03:21:55.196097+0000","last_clean":"2024-04-27T03:21:55.196097+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:21:55.196097+0000","last_undegraded":"2024-04-27T03:21:55.196097+0000","last_fullsized":"2024-04-27T03:21:55.196097+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:55.195864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:55.195864+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.195864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:37:11.669674+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:10.301220+0000","last_change":"2024-04-27T03:22:10.301346+0000","last_active":"2024-04-27T03:22:10.301220+0000","last_peered":"2024-04-27T03:22:10.301220+0000","last_clean":"2024-04-27T03:22:10.301220+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:22:10.301220+0000","last_undegraded":"2024-04-27T03:22:10.301220+0000","last_fullsized":"2024-04-27T03:22:10.301220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:10.300994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:10.300994+0000","last_clean_scrub_stamp":"2024-04-27T03:22:10.300994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:35:48.215930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:09.338894+0000","last_change":"2024-04-27T03:22:09.339067+0000","last_active":"2024-04-27T03:22:09.338894+0000","last_peered":"2024-04-27T03:22:09.338894+0000","last_clean":"2024-04-27T03:22:09.338894+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:22:09.338894+0000","last_undegraded":"2024-04-27T03:22:09.338894+0000","last_fullsized":"2024-04-27T03:22:09.338894+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:09.338670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:09.338670+0000","last_clean_scrub_stamp":"2024-04-27T03:22:09.338670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:40:51.748061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'8","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.217667+0000","last_change":"2024-04-27T03:21:47.217738+0000","last_active":"2024-04-27T03:21:47.217667+0000","last_peered":"2024-04-27T03:21:47.217667+0000","last_clean":"2024-04-27T03:21:47.217667+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:21:47.217667+0000","last_undegraded":"2024-04-27T03:21:47.217667+0000","last_fullsized":"2024-04-27T03:21:47.217667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:47.217463+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:47.217463+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.217463+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:30.144213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.799808+0000","last_change":"2024-04-27T03:21:57.799873+0000","last_active":"2024-04-27T03:21:57.799808+0000","last_peered":"2024-04-27T03:21:57.799808+0000","last_clean":"2024-04-27T03:21:57.799808+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:21:57.799808+0000","last_undegraded":"2024-04-27T03:21:57.799808+0000","last_fullsized":"2024-04-27T03:21:57.799808+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:57.799593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:57.799593+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.799593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:49.562735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.509870+0000","last_change":"2024-04-27T03:21:56.509929+0000","last_active":"2024-04-27T03:21:56.509870+0000","last_peered":"2024-04-27T03:21:56.509870+0000","last_clean":"2024-04-27T03:21:56.509870+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:56.509870+0000","last_undegraded":"2024-04-27T03:21:56.509870+0000","last_fullsized":"2024-04-27T03:21:56.509870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:56.509666+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:56.509666+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.509666+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:38.981565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:11.406762+0000","last_change":"2024-04-27T03:19:35.441020+0000","last_active":"2024-04-27T03:21:11.406762+0000","last_peered":"2024-04-27T03:21:11.406762+0000","last_clean":"2024-04-27T03:21:11.406762+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:21:11.406762+0000","last_undegraded":"2024-04-27T03:21:11.406762+0000","last_fullsized":"2024-04-27T03:21:11.406762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.754365+0000","last_change":"2024-04-27T03:21:47.754460+0000","last_active":"2024-04-27T03:21:47.754365+0000","last_peered":"2024-04-27T03:21:47.754365+0000","last_clean":"2024-04-27T03:21:47.754365+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:21:47.754365+0000","last_undegraded":"2024-04-27T03:21:47.754365+0000","last_fullsized":"2024-04-27T03:21:47.754365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:47.754158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:47.754158+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.754158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:25.426021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.774122+0000","last_change":"2024-04-27T03:21:51.774180+0000","last_active":"2024-04-27T03:21:51.774122+0000","last_peered":"2024-04-27T03:21:51.774122+0000","last_clean":"2024-04-27T03:21:51.774122+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:21:51.774122+0000","last_undegraded":"2024-04-27T03:21:51.774122+0000","last_fullsized":"2024-04-27T03:21:51.774122+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:51.773922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:51.773922+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.773922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:17:53.853087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.173295+0000","last_change":"2024-04-27T03:21:43.173295+0000","last_active":"2024-04-27T03:21:43.173295+0000","last_peered":"2024-04-27T03:21:43.173295+0000","last_clean":"2024-04-27T03:21:43.173295+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:43.173295+0000","last_undegraded":"2024-04-27T03:21:43.173295+0000","last_fullsized":"2024-04-27T03:21:43.173295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:43.173036+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:43.173036+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.173036+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:46.540282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.740491+0000","last_change":"2024-04-27T03:17:47.026063+0000","last_active":"2024-04-27T03:20:07.740491+0000","last_peered":"2024-04-27T03:20:07.740491+0000","last_clean":"2024-04-27T03:20:07.740491+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:20:07.740491+0000","last_undegraded":"2024-04-27T03:20:07.740491+0000","last_fullsized":"2024-04-27T03:20:07.740491+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:20:07.744497+0000","last_change":"2024-04-27T03:17:47.002843+0000","last_active":"2024-04-27T03:20:07.744497+0000","last_peered":"2024-04-27T03:20:07.744497+0000","last_clean":"2024-04-27T03:20:07.744497+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:20:07.744497+0000","last_undegraded":"2024-04-27T03:20:07.744497+0000","last_fullsized":"2024-04-27T03:20:07.744497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:17:45.958767+0000","last_clean_scrub_stamp":"2024-04-27T03:17:45.958767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'12","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.724438+0000","last_change":"2024-04-27T03:21:31.724508+0000","last_active":"2024-04-27T03:21:31.724438+0000","last_peered":"2024-04-27T03:21:31.724438+0000","last_clean":"2024-04-27T03:21:31.724438+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:31.724438+0000","last_undegraded":"2024-04-27T03:21:31.724438+0000","last_fullsized":"2024-04-27T03:21:31.724438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:31.724252+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:31.724252+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.724252+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:49.406247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.430579+0000","last_change":"2024-04-27T03:21:59.430741+0000","last_active":"2024-04-27T03:21:59.430579+0000","last_peered":"2024-04-27T03:21:59.430579+0000","last_clean":"2024-04-27T03:21:59.430579+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:21:59.430579+0000","last_undegraded":"2024-04-27T03:21:59.430579+0000","last_fullsized":"2024-04-27T03:21:59.430579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:59.430346+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:59.430346+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.430346+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:32.065734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.602897+0000","last_change":"2024-04-27T03:21:38.603000+0000","last_active":"2024-04-27T03:21:38.602897+0000","last_peered":"2024-04-27T03:21:38.602897+0000","last_clean":"2024-04-27T03:21:38.602897+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:21:38.602897+0000","last_undegraded":"2024-04-27T03:21:38.602897+0000","last_fullsized":"2024-04-27T03:21:38.602897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:38.602665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:38.602665+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.602665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:00.698430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":349,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2280,"ondisk_log_size":2280,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":11674,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1888,"num_read_kb":1653,"num_write":789,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2692,"num_omap_keys":14,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1740,"ondisk_log_size":1740,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":972,"num_read_kb":972,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":483,"ondisk_log_size":483,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":87,"num_write":316,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44072,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327768,"statfs":{"total":96636764160,"available":96591634432,"internally_reserved":0,"allocated":425984,"data_stored":148688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":983040,"data_stored":736592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705722,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":739335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39944,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331896,"statfs":{"total":96636764160,"available":96595861504,"internally_reserved":0,"allocated":458752,"data_stored":147935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:22:14.385 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.385 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.386 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.386 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.386 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.386 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.386 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-04-27T03:17:52.005853+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=52, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.387 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.387 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.387 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.387 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.387 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.388 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.388 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-27T03:17:38.619764+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=38, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.388 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.388 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.389 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.389 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.389 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.389 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.389 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.390 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.390 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.390 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.390 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.390 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-27T03:17:47.975250+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.390 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.391 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-04-27T03:17:56.032783+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=56, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.391 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.391 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.391 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.391 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-27T03:17:45.958767+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=17, tm_sec=45, tm_wday=5, tm_yday=118, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=27, tm_hour=3, tm_min=21, tm_sec=9, tm_wday=5, tm_yday=118, tm_isdst=0) 2024-04-27T03:22:14.391 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-27T03:22:34.392 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-27T03:22:34.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.512+0000 7fc3f71f4640 1 Processor -- start 2024-04-27T03:22:34.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.512+0000 7fc3f71f4640 1 -- start start 2024-04-27T03:22:34.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3f012b660 0x7fc3f012ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f00722e0 con 0x7fc3f006f690 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f0071ca0 con 0x7fc3f012b660 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f4f90640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3f012b660 0x7fc3f012ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f4f90640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7fc3f012b660 0x7fc3f012ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47348/0 (socket says 172.21.15.77:47348) 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3effff640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7fc3f006f690 0x7fc3f006faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55124/0 (socket says 172.21.15.77:55124) 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f4f90640 1 -- 172.21.15.77:0/1848736129 learned_addr learned my addr 172.21.15.77:0/1848736129 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f4f90640 1 -- 172.21.15.77:0/1848736129 >> v1:172.21.15.77:6789/0 conn(0x7fc3f006f690 legacy=0x7fc3f006faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f4f90640 1 -- 172.21.15.77:0/1848736129 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f012bf80 con 0x7fc3f012b660 2024-04-27T03:22:34.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3ef7fe640 1 -- 172.21.15.77:0/1848736129 <== mon.0 v1:172.21.15.77:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1839315210 0 0) 0x7fc3f00722e0 con 0x7fc3f006f690 2024-04-27T03:22:34.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f4f90640 1 --2- 172.21.15.77:0/1848736129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f012b660 0x7fc3f012ba40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80097f0 tx=0x7fc3d801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befaeee89ccd1b71 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:34.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3ef7fe640 1 -- 172.21.15.77:0/1848736129 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d80b0070 con 0x7fc3f012b660 2024-04-27T03:22:34.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3ef7fe640 1 -- 172.21.15.77:0/1848736129 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d80ab530 con 0x7fc3f012b660 2024-04-27T03:22:34.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3ef7fe640 1 -- 172.21.15.77:0/1848736129 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d80ab830 con 0x7fc3f012b660 2024-04-27T03:22:34.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1848736129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f012b660 msgr2=0x7fc3f012ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:34.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 --2- 172.21.15.77:0/1848736129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f012b660 0x7fc3f012ba40 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80097f0 tx=0x7fc3d801c3d0 comp rx=0 tx=0).stop 2024-04-27T03:22:34.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1848736129 shutdown_connections 2024-04-27T03:22:34.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 --2- 172.21.15.77:0/1848736129 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f012b660 0x7fc3f012ba40 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:34.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1848736129 >> 172.21.15.77:0/1848736129 conn(0x7fc3f00a0be0 msgr2=0x7fc3f00a0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:22:34.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1848736129 shutdown_connections 2024-04-27T03:22:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.516+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1848736129 wait complete. 2024-04-27T03:22:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f71f4640 1 Processor -- start 2024-04-27T03:22:34.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f71f4640 1 -- start start 2024-04-27T03:22:34.529 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f71f4640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f006f690 0x7fc3f0127870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f71f4640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f0137110 con 0x7fc3f006f690 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f4f90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f006f690 0x7fc3f0127870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f4f90640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f006f690 0x7fc3f0127870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47350/0 (socket says 172.21.15.77:47350) 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f4f90640 1 -- 172.21.15.77:0/1922927454 learned_addr learned my addr 172.21.15.77:0/1922927454 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f4f90640 1 -- 172.21.15.77:0/1922927454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f0119a00 con 0x7fc3f006f690 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f4f90640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f006f690 0x7fc3f0127870 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8019040 tx=0x7fc3d801ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3ed7fa640 1 -- 172.21.15.77:0/1922927454 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d80ba070 con 0x7fc3f006f690 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3ed7fa640 1 -- 172.21.15.77:0/1922927454 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d8004020 con 0x7fc3f006f690 2024-04-27T03:22:34.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f0119560 con 0x7fc3f006f690 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3ed7fa640 1 -- 172.21.15.77:0/1922927454 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d80b0040 con 0x7fc3f006f690 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f0127db0 con 0x7fc3f006f690 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3ed7fa640 1 -- 172.21.15.77:0/1922927454 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc3d8004300 con 0x7fc3f006f690 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f011b040 con 0x7fc3f006f690 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3ed7fa640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3d0035db0 0x7fc3d0038270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3ed7fa640 1 -- 172.21.15.77:0/1922927454 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7fc3d80e3690 con 0x7fc3f006f690 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.520+0000 7fc3effff640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3d0035db0 0x7fc3d0038270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:34.531 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.524+0000 7fc3effff640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3d0035db0 0x7fc3d0038270 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0000f90 tx=0x7fc3e0007dd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:34.534 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.524+0000 7fc3ed7fa640 1 -- 172.21.15.77:0/1922927454 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc3d80160b0 con 0x7fc3f006f690 2024-04-27T03:22:34.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.684+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 --> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3f0062e50 con 0x7fc3d0035db0 2024-04-27T03:22:34.736 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.728+0000 7fc3ed7fa640 1 -- 172.21.15.77:0/1922927454 <== mgr.4100 v2:172.21.15.110:6824/2492415371 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+666079 (secure 0 0 0) 0x7fc3f0062e50 con 0x7fc3d0035db0 2024-04-27T03:22:34.740 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-04-27T03:22:34.748 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-04-27T03:22:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3d0035db0 msgr2=0x7fc3d0038270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3d0035db0 0x7fc3d0038270 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0000f90 tx=0x7fc3e0007dd0 comp rx=0 tx=0).stop 2024-04-27T03:22:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f006f690 msgr2=0x7fc3f0127870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f006f690 0x7fc3f0127870 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8019040 tx=0x7fc3d801ac70 comp rx=0 tx=0).stop 2024-04-27T03:22:34.749 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 shutdown_connections 2024-04-27T03:22:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7fc3d0035db0 0x7fc3d0038270 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 --2- 172.21.15.77:0/1922927454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3f006f690 0x7fc3f0127870 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 >> 172.21.15.77:0/1922927454 conn(0x7fc3f00a0be0 msgr2=0x7fc3f006e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:22:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 shutdown_connections 2024-04-27T03:22:34.750 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.736+0000 7fc3f71f4640 1 -- 172.21.15.77:0/1922927454 wait complete. 2024-04-27T03:22:34.767 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-04-27T03:22:33.779582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":605383,"num_objects":252,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":5393,"num_read_kb":4942,"num_write":2371,"num_write_kb":61374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3041,"num_omap_keys":16,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5010,"ondisk_log_size":5010,"up":594,"acting":594,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":594,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":140024,"kb_used_data":2840,"kb_used_omap":6,"kb_used_meta":137145,"kb_avail":377347336,"statfs":{"total":386547056640,"available":386403672064,"internally_reserved":0,"allocated":2908160,"data_stored":1772550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6378,"internal_metadata":140437270},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006166"},"pg_stats":[{"pgid":"3.3f","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.335160+0000","last_change":"2024-04-27T03:22:01.335229+0000","last_active":"2024-04-27T03:22:01.335160+0000","last_peered":"2024-04-27T03:22:01.335160+0000","last_clean":"2024-04-27T03:22:01.335160+0000","last_became_active":"2024-04-27T03:17:47.024691+0000","last_became_peered":"2024-04-27T03:17:47.024691+0000","last_unstale":"2024-04-27T03:22:01.335160+0000","last_undegraded":"2024-04-27T03:22:01.335160+0000","last_fullsized":"2024-04-27T03:22:01.335160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:01.334959+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:01.334959+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.334959+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:56.778017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.38","version":"31'6","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.158576+0000","last_change":"2024-04-27T03:21:42.158641+0000","last_active":"2024-04-27T03:21:42.158576+0000","last_peered":"2024-04-27T03:21:42.158576+0000","last_clean":"2024-04-27T03:21:42.158576+0000","last_became_active":"2024-04-27T03:17:49.016585+0000","last_became_peered":"2024-04-27T03:17:49.016585+0000","last_unstale":"2024-04-27T03:21:42.158576+0000","last_undegraded":"2024-04-27T03:21:42.158576+0000","last_fullsized":"2024-04-27T03:21:42.158576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:42.158374+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:42.158374+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.158374+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:53.231269+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.472960+0000","last_change":"2024-04-27T03:21:57.472960+0000","last_active":"2024-04-27T03:21:57.472960+0000","last_peered":"2024-04-27T03:21:57.472960+0000","last_clean":"2024-04-27T03:21:57.472960+0000","last_became_active":"2024-04-27T03:17:47.023006+0000","last_became_peered":"2024-04-27T03:17:47.023006+0000","last_unstale":"2024-04-27T03:21:57.472960+0000","last_undegraded":"2024-04-27T03:21:57.472960+0000","last_fullsized":"2024-04-27T03:21:57.472960+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:57.472693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:57.472693+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.472693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:35.144556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.39","version":"27'8","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.789741+0000","last_change":"2024-04-27T03:21:43.789808+0000","last_active":"2024-04-27T03:21:43.789741+0000","last_peered":"2024-04-27T03:21:43.789741+0000","last_clean":"2024-04-27T03:21:43.789741+0000","last_became_active":"2024-04-27T03:17:49.017685+0000","last_became_peered":"2024-04-27T03:17:49.017685+0000","last_unstale":"2024-04-27T03:21:43.789741+0000","last_undegraded":"2024-04-27T03:21:43.789741+0000","last_fullsized":"2024-04-27T03:21:43.789741+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'8","last_scrub_stamp":"2024-04-27T03:21:43.789520+0000","last_deep_scrub":"27'8","last_deep_scrub_stamp":"2024-04-27T03:21:43.789520+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.789520+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:16.261778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.802731+0000","last_change":"2024-04-27T03:21:58.802787+0000","last_active":"2024-04-27T03:21:58.802731+0000","last_peered":"2024-04-27T03:21:58.802731+0000","last_clean":"2024-04-27T03:21:58.802731+0000","last_became_active":"2024-04-27T03:17:47.039058+0000","last_became_peered":"2024-04-27T03:17:47.039058+0000","last_unstale":"2024-04-27T03:21:58.802731+0000","last_undegraded":"2024-04-27T03:21:58.802731+0000","last_fullsized":"2024-04-27T03:21:58.802731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:58.802498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:58.802498+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.802498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:03.518528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"31'10","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.749840+0000","last_change":"2024-04-27T03:21:49.749992+0000","last_active":"2024-04-27T03:21:49.749840+0000","last_peered":"2024-04-27T03:21:49.749840+0000","last_clean":"2024-04-27T03:21:49.749840+0000","last_became_active":"2024-04-27T03:17:49.015836+0000","last_became_peered":"2024-04-27T03:17:49.015836+0000","last_unstale":"2024-04-27T03:21:49.749840+0000","last_undegraded":"2024-04-27T03:21:49.749840+0000","last_fullsized":"2024-04-27T03:21:49.749840+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:49.749616+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:49.749616+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.749616+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:22.385004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":30,"num_read_kb":26,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3c","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.741829+0000","last_change":"2024-04-27T03:21:50.741891+0000","last_active":"2024-04-27T03:21:50.741829+0000","last_peered":"2024-04-27T03:21:50.741829+0000","last_clean":"2024-04-27T03:21:50.741829+0000","last_became_active":"2024-04-27T03:17:47.021074+0000","last_became_peered":"2024-04-27T03:17:47.021074+0000","last_unstale":"2024-04-27T03:21:50.741829+0000","last_undegraded":"2024-04-27T03:21:50.741829+0000","last_fullsized":"2024-04-27T03:21:50.741829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:50.741587+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:50.741587+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.741587+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:38.633527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3b","version":"31'16","reported_seq":90,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.864487+0000","last_change":"2024-04-27T03:21:41.864610+0000","last_active":"2024-04-27T03:21:41.864487+0000","last_peered":"2024-04-27T03:21:41.864487+0000","last_clean":"2024-04-27T03:21:41.864487+0000","last_became_active":"2024-04-27T03:17:49.018079+0000","last_became_peered":"2024-04-27T03:17:49.018079+0000","last_unstale":"2024-04-27T03:21:41.864487+0000","last_undegraded":"2024-04-27T03:21:41.864487+0000","last_fullsized":"2024-04-27T03:21:41.864487+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:41.864193+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:41.864193+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.864193+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:28.818568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.293028+0000","last_change":"2024-04-27T03:22:00.293160+0000","last_active":"2024-04-27T03:22:00.293028+0000","last_peered":"2024-04-27T03:22:00.293028+0000","last_clean":"2024-04-27T03:22:00.293028+0000","last_became_active":"2024-04-27T03:17:47.014821+0000","last_became_peered":"2024-04-27T03:17:47.014821+0000","last_unstale":"2024-04-27T03:22:00.293028+0000","last_undegraded":"2024-04-27T03:22:00.293028+0000","last_fullsized":"2024-04-27T03:22:00.293028+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:00.292799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:00.292799+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.292799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:39:45.148184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3c","version":"27'2","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.698631+0000","last_change":"2024-04-27T03:21:58.698694+0000","last_active":"2024-04-27T03:21:58.698631+0000","last_peered":"2024-04-27T03:21:58.698631+0000","last_clean":"2024-04-27T03:21:58.698631+0000","last_became_active":"2024-04-27T03:18:34.765914+0000","last_became_peered":"2024-04-27T03:18:34.765914+0000","last_unstale":"2024-04-27T03:21:58.698631+0000","last_undegraded":"2024-04-27T03:21:58.698631+0000","last_fullsized":"2024-04-27T03:21:58.698631+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:58.698432+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:58.698432+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.698432+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:28:46.564342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3a","version":"31'5","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:03.317458+0000","last_change":"2024-04-27T03:22:03.317531+0000","last_active":"2024-04-27T03:22:03.317458+0000","last_peered":"2024-04-27T03:22:03.317458+0000","last_clean":"2024-04-27T03:22:03.317458+0000","last_became_active":"2024-04-27T03:17:47.021714+0000","last_became_peered":"2024-04-27T03:17:47.021714+0000","last_unstale":"2024-04-27T03:22:03.317458+0000","last_undegraded":"2024-04-27T03:22:03.317458+0000","last_fullsized":"2024-04-27T03:22:03.317458+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:22:03.317238+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:22:03.317238+0000","last_clean_scrub_stamp":"2024-04-27T03:22:03.317238+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:29:09.833669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3d","version":"31'12","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.217129+0000","last_change":"2024-04-27T03:21:50.217129+0000","last_active":"2024-04-27T03:21:50.217129+0000","last_peered":"2024-04-27T03:21:50.217129+0000","last_clean":"2024-04-27T03:21:50.217129+0000","last_became_active":"2024-04-27T03:17:49.013461+0000","last_became_peered":"2024-04-27T03:17:49.013461+0000","last_unstale":"2024-04-27T03:21:50.217129+0000","last_undegraded":"2024-04-27T03:21:50.217129+0000","last_fullsized":"2024-04-27T03:21:50.217129+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:50.216879+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:50.216879+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.216879+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:27.888741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":35,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.759798+0000","last_change":"2024-04-27T03:22:01.759855+0000","last_active":"2024-04-27T03:22:01.759798+0000","last_peered":"2024-04-27T03:22:01.759798+0000","last_clean":"2024-04-27T03:22:01.759798+0000","last_became_active":"2024-04-27T03:17:47.035694+0000","last_became_peered":"2024-04-27T03:17:47.035694+0000","last_unstale":"2024-04-27T03:22:01.759798+0000","last_undegraded":"2024-04-27T03:22:01.759798+0000","last_fullsized":"2024-04-27T03:22:01.759798+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:01.759596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:01.759596+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.759596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:09:41.922637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3e","version":"31'16","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.088312+0000","last_change":"2024-04-27T03:21:38.088312+0000","last_active":"2024-04-27T03:21:38.088312+0000","last_peered":"2024-04-27T03:21:38.088312+0000","last_clean":"2024-04-27T03:21:38.088312+0000","last_became_active":"2024-04-27T03:17:49.023514+0000","last_became_peered":"2024-04-27T03:17:49.023514+0000","last_unstale":"2024-04-27T03:21:38.088312+0000","last_undegraded":"2024-04-27T03:21:38.088312+0000","last_fullsized":"2024-04-27T03:21:38.088312+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:38.088067+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:38.088067+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.088067+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:12:30.953622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":31,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:10.542137+0000","last_change":"2024-04-27T03:22:10.542203+0000","last_active":"2024-04-27T03:22:10.542137+0000","last_peered":"2024-04-27T03:22:10.542137+0000","last_clean":"2024-04-27T03:22:10.542137+0000","last_became_active":"2024-04-27T03:17:47.013402+0000","last_became_peered":"2024-04-27T03:17:47.013402+0000","last_unstale":"2024-04-27T03:22:10.542137+0000","last_undegraded":"2024-04-27T03:22:10.542137+0000","last_fullsized":"2024-04-27T03:22:10.542137+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:22:10.541930+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:22:10.541930+0000","last_clean_scrub_stamp":"2024-04-27T03:22:10.541930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:40:12.968269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.734857+0000","last_change":"2024-04-27T03:21:37.734857+0000","last_active":"2024-04-27T03:21:37.734857+0000","last_peered":"2024-04-27T03:21:37.734857+0000","last_clean":"2024-04-27T03:21:37.734857+0000","last_became_active":"2024-04-27T03:17:49.019854+0000","last_became_peered":"2024-04-27T03:17:49.019854+0000","last_unstale":"2024-04-27T03:21:37.734857+0000","last_undegraded":"2024-04-27T03:21:37.734857+0000","last_fullsized":"2024-04-27T03:21:37.734857+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:37.734535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:37.734535+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.734535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:00:00.453711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"31'6","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.163767+0000","last_change":"2024-04-27T03:21:40.163767+0000","last_active":"2024-04-27T03:21:40.163767+0000","last_peered":"2024-04-27T03:21:40.163767+0000","last_clean":"2024-04-27T03:21:40.163767+0000","last_became_active":"2024-04-27T03:17:49.018089+0000","last_became_peered":"2024-04-27T03:17:49.018089+0000","last_unstale":"2024-04-27T03:21:40.163767+0000","last_undegraded":"2024-04-27T03:21:40.163767+0000","last_fullsized":"2024-04-27T03:21:40.163767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:40.163506+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:40.163506+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.163506+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:29.161646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.521461+0000","last_change":"2024-04-27T03:21:53.521581+0000","last_active":"2024-04-27T03:21:53.521461+0000","last_peered":"2024-04-27T03:21:53.521461+0000","last_clean":"2024-04-27T03:21:53.521461+0000","last_became_active":"2024-04-27T03:17:47.026628+0000","last_became_peered":"2024-04-27T03:17:47.026628+0000","last_unstale":"2024-04-27T03:21:53.521461+0000","last_undegraded":"2024-04-27T03:21:53.521461+0000","last_fullsized":"2024-04-27T03:21:53.521461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:53.521219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:53.521219+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.521219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:25.993477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.187602+0000","last_change":"2024-04-27T03:21:54.187602+0000","last_active":"2024-04-27T03:21:54.187602+0000","last_peered":"2024-04-27T03:21:54.187602+0000","last_clean":"2024-04-27T03:21:54.187602+0000","last_became_active":"2024-04-27T03:17:49.012560+0000","last_became_peered":"2024-04-27T03:17:49.012560+0000","last_unstale":"2024-04-27T03:21:54.187602+0000","last_undegraded":"2024-04-27T03:21:54.187602+0000","last_fullsized":"2024-04-27T03:21:54.187602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:54.187345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:54.187345+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.187345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:02:44.457897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.771974+0000","last_change":"2024-04-27T03:21:48.772040+0000","last_active":"2024-04-27T03:21:48.771974+0000","last_peered":"2024-04-27T03:21:48.771974+0000","last_clean":"2024-04-27T03:21:48.771974+0000","last_became_active":"2024-04-27T03:17:47.015116+0000","last_became_peered":"2024-04-27T03:17:47.015116+0000","last_unstale":"2024-04-27T03:21:48.771974+0000","last_undegraded":"2024-04-27T03:21:48.771974+0000","last_fullsized":"2024-04-27T03:21:48.771974+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:48.771778+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.771778+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.771778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:10.421790+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"31'10","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:19.288901+0000","last_change":"2024-04-27T03:22:19.288901+0000","last_active":"2024-04-27T03:22:19.288901+0000","last_peered":"2024-04-27T03:22:19.288901+0000","last_clean":"2024-04-27T03:22:19.288901+0000","last_became_active":"2024-04-27T03:17:49.009674+0000","last_became_peered":"2024-04-27T03:17:49.009674+0000","last_unstale":"2024-04-27T03:22:19.288901+0000","last_undegraded":"2024-04-27T03:22:19.288901+0000","last_fullsized":"2024-04-27T03:22:19.288901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:22:19.288530+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:22:19.288530+0000","last_clean_scrub_stamp":"2024-04-27T03:22:19.288530+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:04.035701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.249433+0000","last_change":"2024-04-27T03:21:57.249502+0000","last_active":"2024-04-27T03:21:57.249433+0000","last_peered":"2024-04-27T03:21:57.249433+0000","last_clean":"2024-04-27T03:21:57.249433+0000","last_became_active":"2024-04-27T03:17:47.021887+0000","last_became_peered":"2024-04-27T03:17:47.021887+0000","last_unstale":"2024-04-27T03:21:57.249433+0000","last_undegraded":"2024-04-27T03:21:57.249433+0000","last_fullsized":"2024-04-27T03:21:57.249433+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:57.249215+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:57.249215+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.249215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:06.922093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.33","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.710830+0000","last_change":"2024-04-27T03:21:33.710892+0000","last_active":"2024-04-27T03:21:33.710830+0000","last_peered":"2024-04-27T03:21:33.710830+0000","last_clean":"2024-04-27T03:21:33.710830+0000","last_became_active":"2024-04-27T03:17:49.013917+0000","last_became_peered":"2024-04-27T03:17:49.013917+0000","last_unstale":"2024-04-27T03:21:33.710830+0000","last_undegraded":"2024-04-27T03:21:33.710830+0000","last_fullsized":"2024-04-27T03:21:33.710830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:33.710637+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:33.710637+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.710637+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:36.041679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.34","version":"27'4","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:06.302523+0000","last_change":"2024-04-27T03:22:06.302586+0000","last_active":"2024-04-27T03:22:06.302523+0000","last_peered":"2024-04-27T03:22:06.302523+0000","last_clean":"2024-04-27T03:22:06.302523+0000","last_became_active":"2024-04-27T03:17:47.011498+0000","last_became_peered":"2024-04-27T03:17:47.011498+0000","last_unstale":"2024-04-27T03:22:06.302523+0000","last_undegraded":"2024-04-27T03:22:06.302523+0000","last_fullsized":"2024-04-27T03:22:06.302523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'4","last_scrub_stamp":"2024-04-27T03:22:06.302319+0000","last_deep_scrub":"27'4","last_deep_scrub_stamp":"2024-04-27T03:22:06.302319+0000","last_clean_scrub_stamp":"2024-04-27T03:22:06.302319+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:21:01.297686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.34","version":"31'4","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.716905+0000","last_change":"2024-04-27T03:21:34.716967+0000","last_active":"2024-04-27T03:21:34.716905+0000","last_peered":"2024-04-27T03:21:34.716905+0000","last_clean":"2024-04-27T03:21:34.716905+0000","last_became_active":"2024-04-27T03:17:49.005787+0000","last_became_peered":"2024-04-27T03:17:49.005787+0000","last_unstale":"2024-04-27T03:21:34.716905+0000","last_undegraded":"2024-04-27T03:21:34.716905+0000","last_fullsized":"2024-04-27T03:21:34.716905+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:34.716714+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:34.716714+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.716714+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:06:56.440270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:05.682655+0000","last_change":"2024-04-27T03:22:05.682711+0000","last_active":"2024-04-27T03:22:05.682655+0000","last_peered":"2024-04-27T03:22:05.682655+0000","last_clean":"2024-04-27T03:22:05.682655+0000","last_became_active":"2024-04-27T03:17:47.035760+0000","last_became_peered":"2024-04-27T03:17:47.035760+0000","last_unstale":"2024-04-27T03:22:05.682655+0000","last_undegraded":"2024-04-27T03:22:05.682655+0000","last_fullsized":"2024-04-27T03:22:05.682655+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:05.682454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:05.682454+0000","last_clean_scrub_stamp":"2024-04-27T03:22:05.682454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:26:18.821680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.35","version":"31'14","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.514246+0000","last_change":"2024-04-27T03:21:54.514398+0000","last_active":"2024-04-27T03:21:54.514246+0000","last_peered":"2024-04-27T03:21:54.514246+0000","last_clean":"2024-04-27T03:21:54.514246+0000","last_became_active":"2024-04-27T03:17:49.002993+0000","last_became_peered":"2024-04-27T03:17:49.002993+0000","last_unstale":"2024-04-27T03:21:54.514246+0000","last_undegraded":"2024-04-27T03:21:54.514246+0000","last_fullsized":"2024-04-27T03:21:54.514246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:54.514020+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:54.514020+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.514020+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:37.440770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":34,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:07.604716+0000","last_change":"2024-04-27T03:22:07.604716+0000","last_active":"2024-04-27T03:22:07.604716+0000","last_peered":"2024-04-27T03:22:07.604716+0000","last_clean":"2024-04-27T03:22:07.604716+0000","last_became_active":"2024-04-27T03:17:47.033058+0000","last_became_peered":"2024-04-27T03:17:47.033058+0000","last_unstale":"2024-04-27T03:22:07.604716+0000","last_undegraded":"2024-04-27T03:22:07.604716+0000","last_fullsized":"2024-04-27T03:22:07.604716+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:07.604461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:07.604461+0000","last_clean_scrub_stamp":"2024-04-27T03:22:07.604461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:52:58.924480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"31'38","reported_seq":115,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:14.337089+0000","last_change":"2024-04-27T03:22:14.337175+0000","last_active":"2024-04-27T03:22:14.337089+0000","last_peered":"2024-04-27T03:22:14.337089+0000","last_clean":"2024-04-27T03:22:14.337089+0000","last_became_active":"2024-04-27T03:17:57.088033+0000","last_became_peered":"2024-04-27T03:17:57.088033+0000","last_unstale":"2024-04-27T03:22:14.337089+0000","last_undegraded":"2024-04-27T03:22:14.337089+0000","last_fullsized":"2024-04-27T03:22:14.337089+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'38","last_scrub_stamp":"2024-04-27T03:22:14.336883+0000","last_deep_scrub":"31'38","last_deep_scrub_stamp":"2024-04-27T03:22:14.336883+0000","last_clean_scrub_stamp":"2024-04-27T03:22:14.336883+0000","objects_scrubbed":1,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:03:01.743642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":172,"num_read_kb":172,"num_write":102,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.715106+0000","last_change":"2024-04-27T03:21:51.715106+0000","last_active":"2024-04-27T03:21:51.715106+0000","last_peered":"2024-04-27T03:21:51.715106+0000","last_clean":"2024-04-27T03:21:51.715106+0000","last_became_active":"2024-04-27T03:17:49.015032+0000","last_became_peered":"2024-04-27T03:17:49.015032+0000","last_unstale":"2024-04-27T03:21:51.715106+0000","last_undegraded":"2024-04-27T03:21:51.715106+0000","last_fullsized":"2024-04-27T03:21:51.715106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:51.714857+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:51.714857+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.714857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:02:41.985409+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:14.486660+0000","last_change":"2024-04-27T03:22:14.486660+0000","last_active":"2024-04-27T03:22:14.486660+0000","last_peered":"2024-04-27T03:22:14.486660+0000","last_clean":"2024-04-27T03:22:14.486660+0000","last_became_active":"2024-04-27T03:17:47.016609+0000","last_became_peered":"2024-04-27T03:17:47.016609+0000","last_unstale":"2024-04-27T03:22:14.486660+0000","last_undegraded":"2024-04-27T03:22:14.486660+0000","last_fullsized":"2024-04-27T03:22:14.486660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:14.486397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:14.486397+0000","last_clean_scrub_stamp":"2024-04-27T03:22:14.486397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:45:10.554588+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.758982+0000","last_change":"2024-04-27T03:21:34.759034+0000","last_active":"2024-04-27T03:21:34.758982+0000","last_peered":"2024-04-27T03:21:34.758982+0000","last_clean":"2024-04-27T03:21:34.758982+0000","last_became_active":"2024-04-27T03:17:51.032381+0000","last_became_peered":"2024-04-27T03:17:51.032381+0000","last_unstale":"2024-04-27T03:21:34.758982+0000","last_undegraded":"2024-04-27T03:21:34.758982+0000","last_fullsized":"2024-04-27T03:21:34.758982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:34.758786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:34.758786+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.758786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:06.643052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"31'51","reported_seq":122,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.542440+0000","last_change":"2024-04-27T03:21:17.542518+0000","last_active":"2024-04-27T03:21:17.542440+0000","last_peered":"2024-04-27T03:21:17.542440+0000","last_clean":"2024-04-27T03:21:17.542440+0000","last_became_active":"2024-04-27T03:17:53.038175+0000","last_became_peered":"2024-04-27T03:17:53.038175+0000","last_unstale":"2024-04-27T03:21:17.542440+0000","last_undegraded":"2024-04-27T03:21:17.542440+0000","last_fullsized":"2024-04-27T03:21:17.542440+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'51","last_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_deep_scrub":"31'51","last_deep_scrub_stamp":"2024-04-27T03:21:17.542266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.542266+0000","objects_scrubbed":6,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:24.169053+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.186893+0000","last_change":"2024-04-27T03:21:49.186953+0000","last_active":"2024-04-27T03:21:49.186893+0000","last_peered":"2024-04-27T03:21:49.186893+0000","last_clean":"2024-04-27T03:21:49.186893+0000","last_became_active":"2024-04-27T03:17:55.072040+0000","last_became_peered":"2024-04-27T03:17:55.072040+0000","last_unstale":"2024-04-27T03:21:49.186893+0000","last_undegraded":"2024-04-27T03:21:49.186893+0000","last_fullsized":"2024-04-27T03:21:49.186893+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:49.186698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:49.186698+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.186698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:31.658597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"31'9","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.158141+0000","last_change":"2024-04-27T03:21:53.158222+0000","last_active":"2024-04-27T03:21:53.158141+0000","last_peered":"2024-04-27T03:21:53.158141+0000","last_clean":"2024-04-27T03:21:53.158141+0000","last_became_active":"2024-04-27T03:17:57.059897+0000","last_became_peered":"2024-04-27T03:17:57.059897+0000","last_unstale":"2024-04-27T03:21:53.158141+0000","last_undegraded":"2024-04-27T03:21:53.158141+0000","last_fullsized":"2024-04-27T03:21:53.158141+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:53.157941+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:53.157941+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.157941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:56.275203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"27'2","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.784863+0000","last_change":"2024-04-27T03:21:46.784922+0000","last_active":"2024-04-27T03:21:46.784863+0000","last_peered":"2024-04-27T03:21:46.784863+0000","last_clean":"2024-04-27T03:21:46.784863+0000","last_became_active":"2024-04-27T03:17:49.016827+0000","last_became_peered":"2024-04-27T03:17:49.016827+0000","last_unstale":"2024-04-27T03:21:46.784863+0000","last_undegraded":"2024-04-27T03:21:46.784863+0000","last_fullsized":"2024-04-27T03:21:46.784863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:46.784639+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:46.784639+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.784639+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:29.256538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.740727+0000","last_change":"2024-04-27T03:21:54.740727+0000","last_active":"2024-04-27T03:21:54.740727+0000","last_peered":"2024-04-27T03:21:54.740727+0000","last_clean":"2024-04-27T03:21:54.740727+0000","last_became_active":"2024-04-27T03:17:47.031780+0000","last_became_peered":"2024-04-27T03:17:47.031780+0000","last_unstale":"2024-04-27T03:21:54.740727+0000","last_undegraded":"2024-04-27T03:21:54.740727+0000","last_fullsized":"2024-04-27T03:21:54.740727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:54.740455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:54.740455+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.740455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:29.391194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.767879+0000","last_change":"2024-04-27T03:21:38.767879+0000","last_active":"2024-04-27T03:21:38.767879+0000","last_peered":"2024-04-27T03:21:38.767879+0000","last_clean":"2024-04-27T03:21:38.767879+0000","last_became_active":"2024-04-27T03:17:51.011814+0000","last_became_peered":"2024-04-27T03:17:51.011814+0000","last_unstale":"2024-04-27T03:21:38.767879+0000","last_undegraded":"2024-04-27T03:21:38.767879+0000","last_fullsized":"2024-04-27T03:21:38.767879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:38.767589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:38.767589+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.767589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:55.841408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"31'16","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.287986+0000","last_change":"2024-04-27T03:21:22.288067+0000","last_active":"2024-04-27T03:21:22.287986+0000","last_peered":"2024-04-27T03:21:22.287986+0000","last_clean":"2024-04-27T03:21:22.287986+0000","last_became_active":"2024-04-27T03:17:53.048368+0000","last_became_peered":"2024-04-27T03:17:53.048368+0000","last_unstale":"2024-04-27T03:21:22.287986+0000","last_undegraded":"2024-04-27T03:21:22.287986+0000","last_fullsized":"2024-04-27T03:21:22.287986+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:22.287788+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.287788+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:31.752629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.205494+0000","last_change":"2024-04-27T03:21:23.205547+0000","last_active":"2024-04-27T03:21:23.205494+0000","last_peered":"2024-04-27T03:21:23.205494+0000","last_clean":"2024-04-27T03:21:23.205494+0000","last_became_active":"2024-04-27T03:17:55.057936+0000","last_became_peered":"2024-04-27T03:17:55.057936+0000","last_unstale":"2024-04-27T03:21:23.205494+0000","last_undegraded":"2024-04-27T03:21:23.205494+0000","last_fullsized":"2024-04-27T03:21:23.205494+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.205266+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.205266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:16:32.126473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"27'2","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:19.510708+0000","last_change":"2024-04-27T03:22:19.510847+0000","last_active":"2024-04-27T03:22:19.510708+0000","last_peered":"2024-04-27T03:22:19.510708+0000","last_clean":"2024-04-27T03:22:19.510708+0000","last_became_active":"2024-04-27T03:17:57.074101+0000","last_became_peered":"2024-04-27T03:17:57.074101+0000","last_unstale":"2024-04-27T03:22:19.510708+0000","last_undegraded":"2024-04-27T03:22:19.510708+0000","last_fullsized":"2024-04-27T03:22:19.510708+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:22:19.510479+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:22:19.510479+0000","last_clean_scrub_stamp":"2024-04-27T03:22:19.510479+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:31.128241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"31'16","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.813339+0000","last_change":"2024-04-27T03:21:39.813406+0000","last_active":"2024-04-27T03:21:39.813339+0000","last_peered":"2024-04-27T03:21:39.813339+0000","last_clean":"2024-04-27T03:21:39.813339+0000","last_became_active":"2024-04-27T03:17:49.021062+0000","last_became_peered":"2024-04-27T03:17:49.021062+0000","last_unstale":"2024-04-27T03:21:39.813339+0000","last_undegraded":"2024-04-27T03:21:39.813339+0000","last_fullsized":"2024-04-27T03:21:39.813339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:39.813135+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:39.813135+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.813135+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:03:50.886030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:10.388195+0000","last_change":"2024-04-27T03:22:10.388195+0000","last_active":"2024-04-27T03:22:10.388195+0000","last_peered":"2024-04-27T03:22:10.388195+0000","last_clean":"2024-04-27T03:22:10.388195+0000","last_became_active":"2024-04-27T03:17:47.008499+0000","last_became_peered":"2024-04-27T03:17:47.008499+0000","last_unstale":"2024-04-27T03:22:10.388195+0000","last_undegraded":"2024-04-27T03:22:10.388195+0000","last_fullsized":"2024-04-27T03:22:10.388195+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:10.387899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:10.387899+0000","last_clean_scrub_stamp":"2024-04-27T03:22:10.387899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:53:01.707918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.555607+0000","last_change":"2024-04-27T03:21:18.555657+0000","last_active":"2024-04-27T03:21:18.555607+0000","last_peered":"2024-04-27T03:21:18.555607+0000","last_clean":"2024-04-27T03:21:18.555607+0000","last_became_active":"2024-04-27T03:17:51.033325+0000","last_became_peered":"2024-04-27T03:17:51.033325+0000","last_unstale":"2024-04-27T03:21:18.555607+0000","last_undegraded":"2024-04-27T03:21:18.555607+0000","last_fullsized":"2024-04-27T03:21:18.555607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.555415+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.555415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:17.141075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"31'29","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.786909+0000","last_change":"2024-04-27T03:21:28.786909+0000","last_active":"2024-04-27T03:21:28.786909+0000","last_peered":"2024-04-27T03:21:28.786909+0000","last_clean":"2024-04-27T03:21:28.786909+0000","last_became_active":"2024-04-27T03:17:53.031560+0000","last_became_peered":"2024-04-27T03:17:53.031560+0000","last_unstale":"2024-04-27T03:21:28.786909+0000","last_undegraded":"2024-04-27T03:21:28.786909+0000","last_fullsized":"2024-04-27T03:21:28.786909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'29","last_scrub_stamp":"2024-04-27T03:21:28.786620+0000","last_deep_scrub":"31'29","last_deep_scrub_stamp":"2024-04-27T03:21:28.786620+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.786620+0000","objects_scrubbed":7,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:14:33.411918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.15","version":"31'285","reported_seq":438,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.112035+0000","last_change":"2024-04-27T03:21:37.112095+0000","last_active":"2024-04-27T03:21:37.112035+0000","last_peered":"2024-04-27T03:21:37.112035+0000","last_clean":"2024-04-27T03:21:37.112035+0000","last_became_active":"2024-04-27T03:17:55.068289+0000","last_became_peered":"2024-04-27T03:17:55.068289+0000","last_unstale":"2024-04-27T03:21:37.112035+0000","last_undegraded":"2024-04-27T03:21:37.112035+0000","last_fullsized":"2024-04-27T03:21:37.112035+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:37.111852+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:37.111852+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.111852+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:27.898262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"27'2","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.797908+0000","last_change":"2024-04-27T03:21:13.797972+0000","last_active":"2024-04-27T03:21:13.797908+0000","last_peered":"2024-04-27T03:21:13.797908+0000","last_clean":"2024-04-27T03:21:13.797908+0000","last_became_active":"2024-04-27T03:17:57.067608+0000","last_became_peered":"2024-04-27T03:17:57.067608+0000","last_unstale":"2024-04-27T03:21:13.797908+0000","last_undegraded":"2024-04-27T03:21:13.797908+0000","last_fullsized":"2024-04-27T03:21:13.797908+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:13.797701+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.797701+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:46:15.106812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.17","version":"31'8","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.802531+0000","last_change":"2024-04-27T03:21:45.802590+0000","last_active":"2024-04-27T03:21:45.802531+0000","last_peered":"2024-04-27T03:21:45.802531+0000","last_clean":"2024-04-27T03:21:45.802531+0000","last_became_active":"2024-04-27T03:17:49.016089+0000","last_became_peered":"2024-04-27T03:17:49.016089+0000","last_unstale":"2024-04-27T03:21:45.802531+0000","last_undegraded":"2024-04-27T03:21:45.802531+0000","last_fullsized":"2024-04-27T03:21:45.802531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:45.802345+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:45.802345+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.802345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:23.495705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":20,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:18.332193+0000","last_change":"2024-04-27T03:22:18.332193+0000","last_active":"2024-04-27T03:22:18.332193+0000","last_peered":"2024-04-27T03:22:18.332193+0000","last_clean":"2024-04-27T03:22:18.332193+0000","last_became_active":"2024-04-27T03:17:47.002387+0000","last_became_peered":"2024-04-27T03:17:47.002387+0000","last_unstale":"2024-04-27T03:22:18.332193+0000","last_undegraded":"2024-04-27T03:22:18.332193+0000","last_fullsized":"2024-04-27T03:22:18.332193+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:18.331934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:18.331934+0000","last_clean_scrub_stamp":"2024-04-27T03:22:18.331934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:12:20.140617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.137288+0000","last_change":"2024-04-27T03:21:36.137341+0000","last_active":"2024-04-27T03:21:36.137288+0000","last_peered":"2024-04-27T03:21:36.137288+0000","last_clean":"2024-04-27T03:21:36.137288+0000","last_became_active":"2024-04-27T03:17:51.019377+0000","last_became_peered":"2024-04-27T03:17:51.019377+0000","last_unstale":"2024-04-27T03:21:36.137288+0000","last_undegraded":"2024-04-27T03:21:36.137288+0000","last_fullsized":"2024-04-27T03:21:36.137288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:36.137094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:36.137094+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.137094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:18.002317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"31'43","reported_seq":108,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.228263+0000","last_change":"2024-04-27T03:21:21.228344+0000","last_active":"2024-04-27T03:21:21.228263+0000","last_peered":"2024-04-27T03:21:21.228263+0000","last_clean":"2024-04-27T03:21:21.228263+0000","last_became_active":"2024-04-27T03:17:53.038003+0000","last_became_peered":"2024-04-27T03:17:53.038003+0000","last_unstale":"2024-04-27T03:21:21.228263+0000","last_undegraded":"2024-04-27T03:21:21.228263+0000","last_fullsized":"2024-04-27T03:21:21.228263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'43","last_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_deep_scrub":"31'43","last_deep_scrub_stamp":"2024-04-27T03:21:21.228072+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.228072+0000","objects_scrubbed":5,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:05.540210+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"31'285","reported_seq":418,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.596919+0000","last_change":"2024-04-27T03:21:24.596980+0000","last_active":"2024-04-27T03:21:24.596919+0000","last_peered":"2024-04-27T03:21:24.596919+0000","last_clean":"2024-04-27T03:21:24.596919+0000","last_became_active":"2024-04-27T03:17:55.073990+0000","last_became_peered":"2024-04-27T03:17:55.073990+0000","last_unstale":"2024-04-27T03:21:24.596919+0000","last_undegraded":"2024-04-27T03:21:24.596919+0000","last_fullsized":"2024-04-27T03:21:24.596919+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.596708+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.596708+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:43.207566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"31'32","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.730071+0000","last_change":"2024-04-27T03:22:00.730133+0000","last_active":"2024-04-27T03:22:00.730071+0000","last_peered":"2024-04-27T03:22:00.730071+0000","last_clean":"2024-04-27T03:22:00.730071+0000","last_became_active":"2024-04-27T03:18:34.765857+0000","last_became_peered":"2024-04-27T03:18:34.765857+0000","last_unstale":"2024-04-27T03:22:00.730071+0000","last_undegraded":"2024-04-27T03:22:00.730071+0000","last_fullsized":"2024-04-27T03:22:00.730071+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'32","last_scrub_stamp":"2024-04-27T03:22:00.729884+0000","last_deep_scrub":"31'32","last_deep_scrub_stamp":"2024-04-27T03:22:00.729884+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.729884+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:23:20.945115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":101,"num_read_kb":81,"num_write":41,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:04.277461+0000","last_change":"2024-04-27T03:22:04.277461+0000","last_active":"2024-04-27T03:22:04.277461+0000","last_peered":"2024-04-27T03:22:04.277461+0000","last_clean":"2024-04-27T03:22:04.277461+0000","last_became_active":"2024-04-27T03:17:47.012547+0000","last_became_peered":"2024-04-27T03:17:47.012547+0000","last_unstale":"2024-04-27T03:22:04.277461+0000","last_undegraded":"2024-04-27T03:22:04.277461+0000","last_fullsized":"2024-04-27T03:22:04.277461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:04.277181+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:04.277181+0000","last_clean_scrub_stamp":"2024-04-27T03:22:04.277181+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:09:44.440223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.413670+0000","last_change":"2024-04-27T03:21:31.413670+0000","last_active":"2024-04-27T03:21:31.413670+0000","last_peered":"2024-04-27T03:21:31.413670+0000","last_clean":"2024-04-27T03:21:31.413670+0000","last_became_active":"2024-04-27T03:17:51.026434+0000","last_became_peered":"2024-04-27T03:17:51.026434+0000","last_unstale":"2024-04-27T03:21:31.413670+0000","last_undegraded":"2024-04-27T03:21:31.413670+0000","last_fullsized":"2024-04-27T03:21:31.413670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:31.413366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:31.413366+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.413366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:23.060435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"31'59","reported_seq":130,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.203825+0000","last_change":"2024-04-27T03:21:26.203825+0000","last_active":"2024-04-27T03:21:26.203825+0000","last_peered":"2024-04-27T03:21:26.203825+0000","last_clean":"2024-04-27T03:21:26.203825+0000","last_became_active":"2024-04-27T03:17:53.059749+0000","last_became_peered":"2024-04-27T03:17:53.059749+0000","last_unstale":"2024-04-27T03:21:26.203825+0000","last_undegraded":"2024-04-27T03:21:26.203825+0000","last_fullsized":"2024-04-27T03:21:26.203825+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'59","last_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_deep_scrub":"31'59","last_deep_scrub_stamp":"2024-04-27T03:21:26.203534+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.203534+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:20:44.814392+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":47,"num_write":30,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.260248+0000","last_change":"2024-04-27T03:21:15.260248+0000","last_active":"2024-04-27T03:21:15.260248+0000","last_peered":"2024-04-27T03:21:15.260248+0000","last_clean":"2024-04-27T03:21:15.260248+0000","last_became_active":"2024-04-27T03:17:55.063027+0000","last_became_peered":"2024-04-27T03:17:55.063027+0000","last_unstale":"2024-04-27T03:21:15.260248+0000","last_undegraded":"2024-04-27T03:21:15.260248+0000","last_fullsized":"2024-04-27T03:21:15.260248+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:15.259998+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.259998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:51:59.062074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.4","version":"29'3","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.825848+0000","last_change":"2024-04-27T03:21:25.825848+0000","last_active":"2024-04-27T03:21:25.825848+0000","last_peered":"2024-04-27T03:21:25.825848+0000","last_clean":"2024-04-27T03:21:25.825848+0000","last_became_active":"2024-04-27T03:17:57.085299+0000","last_became_peered":"2024-04-27T03:17:57.085299+0000","last_unstale":"2024-04-27T03:21:25.825848+0000","last_undegraded":"2024-04-27T03:21:25.825848+0000","last_fullsized":"2024-04-27T03:21:25.825848+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'3","last_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_deep_scrub":"29'3","last_deep_scrub_stamp":"2024-04-27T03:21:25.825552+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.825552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:43:06.575600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"31'14","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.537316+0000","last_change":"2024-04-27T03:21:50.537316+0000","last_active":"2024-04-27T03:21:50.537316+0000","last_peered":"2024-04-27T03:21:50.537316+0000","last_clean":"2024-04-27T03:21:50.537316+0000","last_became_active":"2024-04-27T03:17:49.002863+0000","last_became_peered":"2024-04-27T03:17:49.002863+0000","last_unstale":"2024-04-27T03:21:50.537316+0000","last_undegraded":"2024-04-27T03:21:50.537316+0000","last_fullsized":"2024-04-27T03:21:50.537316+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:50.537066+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:50.537066+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.537066+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:53.904312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:16.340806+0000","last_change":"2024-04-27T03:22:16.340806+0000","last_active":"2024-04-27T03:22:16.340806+0000","last_peered":"2024-04-27T03:22:16.340806+0000","last_clean":"2024-04-27T03:22:16.340806+0000","last_became_active":"2024-04-27T03:17:46.994176+0000","last_became_peered":"2024-04-27T03:17:46.994176+0000","last_unstale":"2024-04-27T03:22:16.340806+0000","last_undegraded":"2024-04-27T03:22:16.340806+0000","last_fullsized":"2024-04-27T03:22:16.340806+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:16.340547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:16.340547+0000","last_clean_scrub_stamp":"2024-04-27T03:22:16.340547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:47.868946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.762950+0000","last_change":"2024-04-27T03:21:43.762950+0000","last_active":"2024-04-27T03:21:43.762950+0000","last_peered":"2024-04-27T03:21:43.762950+0000","last_clean":"2024-04-27T03:21:43.762950+0000","last_became_active":"2024-04-27T03:17:51.036445+0000","last_became_peered":"2024-04-27T03:17:51.036445+0000","last_unstale":"2024-04-27T03:21:43.762950+0000","last_undegraded":"2024-04-27T03:21:43.762950+0000","last_fullsized":"2024-04-27T03:21:43.762950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:43.762700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:43.762700+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.762700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:13.310122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"31'44","reported_seq":110,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.518875+0000","last_change":"2024-04-27T03:21:13.518875+0000","last_active":"2024-04-27T03:21:13.518875+0000","last_peered":"2024-04-27T03:21:13.518875+0000","last_clean":"2024-04-27T03:21:13.518875+0000","last_became_active":"2024-04-27T03:17:53.040637+0000","last_became_peered":"2024-04-27T03:17:53.040637+0000","last_unstale":"2024-04-27T03:21:13.518875+0000","last_undegraded":"2024-04-27T03:21:13.518875+0000","last_fullsized":"2024-04-27T03:21:13.518875+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'44","last_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_deep_scrub":"31'44","last_deep_scrub_stamp":"2024-04-27T03:21:13.518592+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.518592+0000","objects_scrubbed":5,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:24:22.864770+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.708291+0000","last_change":"2024-04-27T03:21:32.708291+0000","last_active":"2024-04-27T03:21:32.708291+0000","last_peered":"2024-04-27T03:21:32.708291+0000","last_clean":"2024-04-27T03:21:32.708291+0000","last_became_active":"2024-04-27T03:17:55.074332+0000","last_became_peered":"2024-04-27T03:17:55.074332+0000","last_unstale":"2024-04-27T03:21:32.708291+0000","last_undegraded":"2024-04-27T03:21:32.708291+0000","last_fullsized":"2024-04-27T03:21:32.708291+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:32.708044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:32.708044+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.708044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:43.162536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"31'6","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:11.267314+0000","last_change":"2024-04-27T03:22:11.267381+0000","last_active":"2024-04-27T03:22:11.267314+0000","last_peered":"2024-04-27T03:22:11.267314+0000","last_clean":"2024-04-27T03:22:11.267314+0000","last_became_active":"2024-04-27T03:17:57.064342+0000","last_became_peered":"2024-04-27T03:17:57.064342+0000","last_unstale":"2024-04-27T03:22:11.267314+0000","last_undegraded":"2024-04-27T03:22:11.267314+0000","last_fullsized":"2024-04-27T03:22:11.267314+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:11.267113+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:11.267113+0000","last_clean_scrub_stamp":"2024-04-27T03:22:11.267113+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:09:51.430154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":169,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"31'3","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.774871+0000","last_change":"2024-04-27T03:21:29.774989+0000","last_active":"2024-04-27T03:21:29.774871+0000","last_peered":"2024-04-27T03:21:29.774871+0000","last_clean":"2024-04-27T03:21:29.774871+0000","last_became_active":"2024-04-27T03:18:34.763383+0000","last_became_peered":"2024-04-27T03:18:34.763383+0000","last_unstale":"2024-04-27T03:21:29.774871+0000","last_undegraded":"2024-04-27T03:21:29.774871+0000","last_fullsized":"2024-04-27T03:21:29.774871+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-04-27T03:21:29.774563+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-04-27T03:21:29.774563+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.774563+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:14:41.424401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:09.395668+0000","last_change":"2024-04-27T03:22:09.395668+0000","last_active":"2024-04-27T03:22:09.395668+0000","last_peered":"2024-04-27T03:22:09.395668+0000","last_clean":"2024-04-27T03:22:09.395668+0000","last_became_active":"2024-04-27T03:17:47.025864+0000","last_became_peered":"2024-04-27T03:17:47.025864+0000","last_unstale":"2024-04-27T03:22:09.395668+0000","last_undegraded":"2024-04-27T03:22:09.395668+0000","last_fullsized":"2024-04-27T03:22:09.395668+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:09.395415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:09.395415+0000","last_clean_scrub_stamp":"2024-04-27T03:22:09.395415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:52.077494+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.794520+0000","last_change":"2024-04-27T03:21:45.794520+0000","last_active":"2024-04-27T03:21:45.794520+0000","last_peered":"2024-04-27T03:21:45.794520+0000","last_clean":"2024-04-27T03:21:45.794520+0000","last_became_active":"2024-04-27T03:17:51.036393+0000","last_became_peered":"2024-04-27T03:17:51.036393+0000","last_unstale":"2024-04-27T03:21:45.794520+0000","last_undegraded":"2024-04-27T03:21:45.794520+0000","last_fullsized":"2024-04-27T03:21:45.794520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:45.794255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:45.794255+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.794255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:52.421042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"31'60","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:17.330698+0000","last_change":"2024-04-27T03:22:17.330821+0000","last_active":"2024-04-27T03:22:17.330698+0000","last_peered":"2024-04-27T03:22:17.330698+0000","last_clean":"2024-04-27T03:22:17.330698+0000","last_became_active":"2024-04-27T03:18:34.768957+0000","last_became_peered":"2024-04-27T03:18:34.768957+0000","last_unstale":"2024-04-27T03:22:17.330698+0000","last_undegraded":"2024-04-27T03:22:17.330698+0000","last_fullsized":"2024-04-27T03:22:17.330698+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'60","last_scrub_stamp":"2024-04-27T03:22:17.330500+0000","last_deep_scrub":"31'60","last_deep_scrub_stamp":"2024-04-27T03:22:17.330500+0000","last_clean_scrub_stamp":"2024-04-27T03:22:17.330500+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:53:08.650518+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"31'285","reported_seq":394,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.866425+0000","last_change":"2024-04-27T03:21:20.866425+0000","last_active":"2024-04-27T03:21:20.866425+0000","last_peered":"2024-04-27T03:21:20.866425+0000","last_clean":"2024-04-27T03:21:20.866425+0000","last_became_active":"2024-04-27T03:17:55.062782+0000","last_became_peered":"2024-04-27T03:17:55.062782+0000","last_unstale":"2024-04-27T03:21:20.866425+0000","last_undegraded":"2024-04-27T03:21:20.866425+0000","last_fullsized":"2024-04-27T03:21:20.866425+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:20.866168+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.866168+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:18.678233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.628282+0000","last_change":"2024-04-27T03:21:40.628282+0000","last_active":"2024-04-27T03:21:40.628282+0000","last_peered":"2024-04-27T03:21:40.628282+0000","last_clean":"2024-04-27T03:21:40.628282+0000","last_became_active":"2024-04-27T03:17:57.076209+0000","last_became_peered":"2024-04-27T03:17:57.076209+0000","last_unstale":"2024-04-27T03:21:40.628282+0000","last_undegraded":"2024-04-27T03:21:40.628282+0000","last_fullsized":"2024-04-27T03:21:40.628282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:40.628044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:40.628044+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.628044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:28:28.493954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"31'16","reported_seq":98,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.749288+0000","last_change":"2024-04-27T03:21:40.749288+0000","last_active":"2024-04-27T03:21:40.749288+0000","last_peered":"2024-04-27T03:21:40.749288+0000","last_clean":"2024-04-27T03:21:40.749288+0000","last_became_active":"2024-04-27T03:17:49.007525+0000","last_became_peered":"2024-04-27T03:17:49.007525+0000","last_unstale":"2024-04-27T03:21:40.749288+0000","last_undegraded":"2024-04-27T03:21:40.749288+0000","last_fullsized":"2024-04-27T03:21:40.749288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:40.749042+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:40.749042+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.749042+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:50.213883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":64,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:25.327327+0000","last_change":"2024-04-27T03:22:25.327499+0000","last_active":"2024-04-27T03:22:25.327327+0000","last_peered":"2024-04-27T03:22:25.327327+0000","last_clean":"2024-04-27T03:22:25.327327+0000","last_became_active":"2024-04-27T03:18:34.771773+0000","last_became_peered":"2024-04-27T03:18:34.771773+0000","last_unstale":"2024-04-27T03:22:25.327327+0000","last_undegraded":"2024-04-27T03:22:25.327327+0000","last_fullsized":"2024-04-27T03:22:25.327327+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:25.327078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:25.327078+0000","last_clean_scrub_stamp":"2024-04-27T03:22:25.327078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:08.901919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.756406+0000","last_change":"2024-04-27T03:21:30.756406+0000","last_active":"2024-04-27T03:21:30.756406+0000","last_peered":"2024-04-27T03:21:30.756406+0000","last_clean":"2024-04-27T03:21:30.756406+0000","last_became_active":"2024-04-27T03:17:51.038573+0000","last_became_peered":"2024-04-27T03:17:51.038573+0000","last_unstale":"2024-04-27T03:21:30.756406+0000","last_undegraded":"2024-04-27T03:21:30.756406+0000","last_fullsized":"2024-04-27T03:21:30.756406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:30.756077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:30.756077+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.756077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:41:43.084146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"31'77","reported_seq":167,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.632024+0000","last_change":"2024-04-27T03:21:27.632024+0000","last_active":"2024-04-27T03:21:27.632024+0000","last_peered":"2024-04-27T03:21:27.632024+0000","last_clean":"2024-04-27T03:21:27.632024+0000","last_became_active":"2024-04-27T03:17:53.049684+0000","last_became_peered":"2024-04-27T03:17:53.049684+0000","last_unstale":"2024-04-27T03:21:27.632024+0000","last_undegraded":"2024-04-27T03:21:27.632024+0000","last_fullsized":"2024-04-27T03:21:27.632024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'77","last_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_deep_scrub":"31'77","last_deep_scrub_stamp":"2024-04-27T03:21:27.631741+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.631741+0000","objects_scrubbed":7,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:01.393467+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":34,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.822651+0000","last_change":"2024-04-27T03:21:22.822651+0000","last_active":"2024-04-27T03:21:22.822651+0000","last_peered":"2024-04-27T03:21:22.822651+0000","last_clean":"2024-04-27T03:21:22.822651+0000","last_became_active":"2024-04-27T03:17:55.056893+0000","last_became_peered":"2024-04-27T03:17:55.056893+0000","last_unstale":"2024-04-27T03:21:22.822651+0000","last_undegraded":"2024-04-27T03:21:22.822651+0000","last_fullsized":"2024-04-27T03:21:22.822651+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.822323+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.822323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:31:20.663089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"31'21","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.464841+0000","last_change":"2024-04-27T03:21:32.464841+0000","last_active":"2024-04-27T03:21:32.464841+0000","last_peered":"2024-04-27T03:21:32.464841+0000","last_clean":"2024-04-27T03:21:32.464841+0000","last_became_active":"2024-04-27T03:17:57.061560+0000","last_became_peered":"2024-04-27T03:17:57.061560+0000","last_unstale":"2024-04-27T03:21:32.464841+0000","last_undegraded":"2024-04-27T03:21:32.464841+0000","last_fullsized":"2024-04-27T03:21:32.464841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'21","last_scrub_stamp":"2024-04-27T03:21:32.464430+0000","last_deep_scrub":"31'21","last_deep_scrub_stamp":"2024-04-27T03:21:32.464430+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.464430+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:20.356371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":904,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":42,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'4","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.649809+0000","last_change":"2024-04-27T03:21:42.649809+0000","last_active":"2024-04-27T03:21:42.649809+0000","last_peered":"2024-04-27T03:21:42.649809+0000","last_clean":"2024-04-27T03:21:42.649809+0000","last_became_active":"2024-04-27T03:17:49.009417+0000","last_became_peered":"2024-04-27T03:17:49.009417+0000","last_unstale":"2024-04-27T03:21:42.649809+0000","last_undegraded":"2024-04-27T03:21:42.649809+0000","last_fullsized":"2024-04-27T03:21:42.649809+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:42.649523+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:42.649523+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.649523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:58.314334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.709141+0000","last_change":"2024-04-27T03:21:56.709198+0000","last_active":"2024-04-27T03:21:56.709141+0000","last_peered":"2024-04-27T03:21:56.709141+0000","last_clean":"2024-04-27T03:21:56.709141+0000","last_became_active":"2024-04-27T03:18:34.766054+0000","last_became_peered":"2024-04-27T03:18:34.766054+0000","last_unstale":"2024-04-27T03:21:56.709141+0000","last_undegraded":"2024-04-27T03:21:56.709141+0000","last_fullsized":"2024-04-27T03:21:56.709141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:56.708942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:56.708942+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.708942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:18.804708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.740168+0000","last_change":"2024-04-27T03:21:39.740168+0000","last_active":"2024-04-27T03:21:39.740168+0000","last_peered":"2024-04-27T03:21:39.740168+0000","last_clean":"2024-04-27T03:21:39.740168+0000","last_became_active":"2024-04-27T03:17:51.028479+0000","last_became_peered":"2024-04-27T03:17:51.028479+0000","last_unstale":"2024-04-27T03:21:39.740168+0000","last_undegraded":"2024-04-27T03:21:39.740168+0000","last_fullsized":"2024-04-27T03:21:39.740168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:39.739923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:39.739923+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.739923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:52:10.953090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"31'48","reported_seq":115,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.092719+0000","last_change":"2024-04-27T03:21:30.092719+0000","last_active":"2024-04-27T03:21:30.092719+0000","last_peered":"2024-04-27T03:21:30.092719+0000","last_clean":"2024-04-27T03:21:30.092719+0000","last_became_active":"2024-04-27T03:17:53.048310+0000","last_became_peered":"2024-04-27T03:17:53.048310+0000","last_unstale":"2024-04-27T03:21:30.092719+0000","last_undegraded":"2024-04-27T03:21:30.092719+0000","last_fullsized":"2024-04-27T03:21:30.092719+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'48","last_scrub_stamp":"2024-04-27T03:21:30.092441+0000","last_deep_scrub":"31'48","last_deep_scrub_stamp":"2024-04-27T03:21:30.092441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.092441+0000","objects_scrubbed":6,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:22.386772+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4096,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":63,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.585759+0000","last_change":"2024-04-27T03:21:23.585759+0000","last_active":"2024-04-27T03:21:23.585759+0000","last_peered":"2024-04-27T03:21:23.585759+0000","last_clean":"2024-04-27T03:21:23.585759+0000","last_became_active":"2024-04-27T03:17:55.071813+0000","last_became_peered":"2024-04-27T03:17:55.071813+0000","last_unstale":"2024-04-27T03:21:23.585759+0000","last_undegraded":"2024-04-27T03:21:23.585759+0000","last_fullsized":"2024-04-27T03:21:23.585759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.585484+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.585484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:34.196580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"31'4","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:20.465441+0000","last_change":"2024-04-27T03:22:20.465514+0000","last_active":"2024-04-27T03:22:20.465441+0000","last_peered":"2024-04-27T03:22:20.465441+0000","last_clean":"2024-04-27T03:22:20.465441+0000","last_became_active":"2024-04-27T03:17:57.080114+0000","last_became_peered":"2024-04-27T03:17:57.080114+0000","last_unstale":"2024-04-27T03:22:20.465441+0000","last_undegraded":"2024-04-27T03:22:20.465441+0000","last_fullsized":"2024-04-27T03:22:20.465441+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:20.465212+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:20.465212+0000","last_clean_scrub_stamp":"2024-04-27T03:22:20.465212+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:13.016169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"29'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.193588+0000","last_change":"2024-04-27T03:21:48.193588+0000","last_active":"2024-04-27T03:21:48.193588+0000","last_peered":"2024-04-27T03:21:48.193588+0000","last_clean":"2024-04-27T03:21:48.193588+0000","last_became_active":"2024-04-27T03:17:49.014779+0000","last_became_peered":"2024-04-27T03:17:49.014779+0000","last_unstale":"2024-04-27T03:21:48.193588+0000","last_undegraded":"2024-04-27T03:21:48.193588+0000","last_fullsized":"2024-04-27T03:21:48.193588+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:48.193325+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.193325+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.193325+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:25.886685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:05.317248+0000","last_change":"2024-04-27T03:22:05.317248+0000","last_active":"2024-04-27T03:22:05.317248+0000","last_peered":"2024-04-27T03:22:05.317248+0000","last_clean":"2024-04-27T03:22:05.317248+0000","last_became_active":"2024-04-27T03:17:47.022064+0000","last_became_peered":"2024-04-27T03:17:47.022064+0000","last_unstale":"2024-04-27T03:22:05.317248+0000","last_undegraded":"2024-04-27T03:22:05.317248+0000","last_fullsized":"2024-04-27T03:22:05.317248+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:05.316823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:05.316823+0000","last_clean_scrub_stamp":"2024-04-27T03:22:05.316823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:59:18.115233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.324853+0000","last_change":"2024-04-27T03:21:12.324853+0000","last_active":"2024-04-27T03:21:12.324853+0000","last_peered":"2024-04-27T03:21:12.324853+0000","last_clean":"2024-04-27T03:21:12.324853+0000","last_became_active":"2024-04-27T03:17:51.036270+0000","last_became_peered":"2024-04-27T03:17:51.036270+0000","last_unstale":"2024-04-27T03:21:12.324853+0000","last_undegraded":"2024-04-27T03:21:12.324853+0000","last_fullsized":"2024-04-27T03:21:12.324853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.324599+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.324599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:41:24.652668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"31'45","reported_seq":108,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.787057+0000","last_change":"2024-04-27T03:21:26.787057+0000","last_active":"2024-04-27T03:21:26.787057+0000","last_peered":"2024-04-27T03:21:26.787057+0000","last_clean":"2024-04-27T03:21:26.787057+0000","last_became_active":"2024-04-27T03:17:53.042263+0000","last_became_peered":"2024-04-27T03:17:53.042263+0000","last_unstale":"2024-04-27T03:21:26.787057+0000","last_undegraded":"2024-04-27T03:21:26.787057+0000","last_fullsized":"2024-04-27T03:21:26.787057+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:26.786765+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.786765+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:29:28.251865+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"31'285","reported_seq":421,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.779964+0000","last_change":"2024-04-27T03:21:18.779964+0000","last_active":"2024-04-27T03:21:18.779964+0000","last_peered":"2024-04-27T03:21:18.779964+0000","last_clean":"2024-04-27T03:21:18.779964+0000","last_became_active":"2024-04-27T03:17:55.069933+0000","last_became_peered":"2024-04-27T03:17:55.069933+0000","last_unstale":"2024-04-27T03:21:18.779964+0000","last_undegraded":"2024-04-27T03:21:18.779964+0000","last_fullsized":"2024-04-27T03:21:18.779964+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:18.779620+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.779620+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:58:08.955224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"31'13","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.591263+0000","last_change":"2024-04-27T03:21:44.591263+0000","last_active":"2024-04-27T03:21:44.591263+0000","last_peered":"2024-04-27T03:21:44.591263+0000","last_clean":"2024-04-27T03:21:44.591263+0000","last_became_active":"2024-04-27T03:17:57.077131+0000","last_became_peered":"2024-04-27T03:17:57.077131+0000","last_unstale":"2024-04-27T03:21:44.591263+0000","last_undegraded":"2024-04-27T03:21:44.591263+0000","last_fullsized":"2024-04-27T03:21:44.591263+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'13","last_scrub_stamp":"2024-04-27T03:21:44.590873+0000","last_deep_scrub":"31'13","last_deep_scrub_stamp":"2024-04-27T03:21:44.590873+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.590873+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:31.004553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":180,"num_omap_keys":1,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"27'8","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.745694+0000","last_change":"2024-04-27T03:21:38.745694+0000","last_active":"2024-04-27T03:21:38.745694+0000","last_peered":"2024-04-27T03:21:38.745694+0000","last_clean":"2024-04-27T03:21:38.745694+0000","last_became_active":"2024-04-27T03:17:49.008521+0000","last_became_peered":"2024-04-27T03:17:49.008521+0000","last_unstale":"2024-04-27T03:21:38.745694+0000","last_undegraded":"2024-04-27T03:21:38.745694+0000","last_fullsized":"2024-04-27T03:21:38.745694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'8","last_scrub_stamp":"2024-04-27T03:21:38.745283+0000","last_deep_scrub":"27'8","last_deep_scrub_stamp":"2024-04-27T03:21:38.745283+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.745283+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:11:16.380426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":40,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.772604+0000","last_change":"2024-04-27T03:21:59.772604+0000","last_active":"2024-04-27T03:21:59.772604+0000","last_peered":"2024-04-27T03:21:59.772604+0000","last_clean":"2024-04-27T03:21:59.772604+0000","last_became_active":"2024-04-27T03:17:47.036133+0000","last_became_peered":"2024-04-27T03:17:47.036133+0000","last_unstale":"2024-04-27T03:21:59.772604+0000","last_undegraded":"2024-04-27T03:21:59.772604+0000","last_fullsized":"2024-04-27T03:21:59.772604+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:59.772349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:59.772349+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.772349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:40:42.181739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"31'5","reported_seq":433,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.080869+0000","last_change":"2024-04-27T03:21:14.080869+0000","last_active":"2024-04-27T03:21:14.080869+0000","last_peered":"2024-04-27T03:21:14.080869+0000","last_clean":"2024-04-27T03:21:14.080869+0000","last_became_active":"2024-04-27T03:17:51.036809+0000","last_became_peered":"2024-04-27T03:17:51.036809+0000","last_unstale":"2024-04-27T03:21:14.080869+0000","last_undegraded":"2024-04-27T03:21:14.080869+0000","last_fullsized":"2024-04-27T03:21:14.080869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:21:14.080480+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.080480+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:38:27.352786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":385,"num_read_kb":385,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"31'16","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.621586+0000","last_change":"2024-04-27T03:21:20.621586+0000","last_active":"2024-04-27T03:21:20.621586+0000","last_peered":"2024-04-27T03:21:20.621586+0000","last_clean":"2024-04-27T03:21:20.621586+0000","last_became_active":"2024-04-27T03:17:53.039939+0000","last_became_peered":"2024-04-27T03:17:53.039939+0000","last_unstale":"2024-04-27T03:21:20.621586+0000","last_undegraded":"2024-04-27T03:21:20.621586+0000","last_fullsized":"2024-04-27T03:21:20.621586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:20.621330+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.621330+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:13:04.933468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"31'285","reported_seq":477,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.100400+0000","last_change":"2024-04-27T03:21:33.100400+0000","last_active":"2024-04-27T03:21:33.100400+0000","last_peered":"2024-04-27T03:21:33.100400+0000","last_clean":"2024-04-27T03:21:33.100400+0000","last_became_active":"2024-04-27T03:17:55.068266+0000","last_became_peered":"2024-04-27T03:17:55.068266+0000","last_unstale":"2024-04-27T03:21:33.100400+0000","last_undegraded":"2024-04-27T03:21:33.100400+0000","last_fullsized":"2024-04-27T03:21:33.100400+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:33.100022+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:33.100022+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.100022+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:20.619756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"31'12","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.766241+0000","last_change":"2024-04-27T03:21:17.766241+0000","last_active":"2024-04-27T03:21:17.766241+0000","last_peered":"2024-04-27T03:21:17.766241+0000","last_clean":"2024-04-27T03:21:17.766241+0000","last_became_active":"2024-04-27T03:17:57.074235+0000","last_became_peered":"2024-04-27T03:17:57.074235+0000","last_unstale":"2024-04-27T03:21:17.766241+0000","last_undegraded":"2024-04-27T03:21:17.766241+0000","last_fullsized":"2024-04-27T03:21:17.766241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:17.765988+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.765988+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:51:49.164397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":58,"num_write":31,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.500219+0000","last_change":"2024-04-27T03:21:34.500219+0000","last_active":"2024-04-27T03:21:34.500219+0000","last_peered":"2024-04-27T03:21:34.500219+0000","last_clean":"2024-04-27T03:21:34.500219+0000","last_became_active":"2024-04-27T03:17:49.012651+0000","last_became_peered":"2024-04-27T03:17:49.012651+0000","last_unstale":"2024-04-27T03:21:34.500219+0000","last_undegraded":"2024-04-27T03:21:34.500219+0000","last_fullsized":"2024-04-27T03:21:34.500219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:34.499952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:34.499952+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.499952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:49:24.665238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:17.470490+0000","last_change":"2024-04-27T03:22:17.470548+0000","last_active":"2024-04-27T03:22:17.470490+0000","last_peered":"2024-04-27T03:22:17.470490+0000","last_clean":"2024-04-27T03:22:17.470490+0000","last_became_active":"2024-04-27T03:17:47.031562+0000","last_became_peered":"2024-04-27T03:17:47.031562+0000","last_unstale":"2024-04-27T03:22:17.470490+0000","last_undegraded":"2024-04-27T03:22:17.470490+0000","last_fullsized":"2024-04-27T03:22:17.470490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:17.470291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:17.470291+0000","last_clean_scrub_stamp":"2024-04-27T03:22:17.470291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:57.212808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.628229+0000","last_change":"2024-04-27T03:21:21.628229+0000","last_active":"2024-04-27T03:21:21.628229+0000","last_peered":"2024-04-27T03:21:21.628229+0000","last_clean":"2024-04-27T03:21:21.628229+0000","last_became_active":"2024-04-27T03:17:51.032823+0000","last_became_peered":"2024-04-27T03:17:51.032823+0000","last_unstale":"2024-04-27T03:21:21.628229+0000","last_undegraded":"2024-04-27T03:21:21.628229+0000","last_fullsized":"2024-04-27T03:21:21.628229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:21.627898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.627898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:37.847354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"31'48","reported_seq":112,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.145327+0000","last_change":"2024-04-27T03:21:31.145327+0000","last_active":"2024-04-27T03:21:31.145327+0000","last_peered":"2024-04-27T03:21:31.145327+0000","last_clean":"2024-04-27T03:21:31.145327+0000","last_became_active":"2024-04-27T03:17:53.048227+0000","last_became_peered":"2024-04-27T03:17:53.048227+0000","last_unstale":"2024-04-27T03:21:31.145327+0000","last_undegraded":"2024-04-27T03:21:31.145327+0000","last_fullsized":"2024-04-27T03:21:31.145327+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'48","last_scrub_stamp":"2024-04-27T03:21:31.145012+0000","last_deep_scrub":"31'48","last_deep_scrub_stamp":"2024-04-27T03:21:31.145012+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.145012+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:50:53.240778+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":37,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.235740+0000","last_change":"2024-04-27T03:21:45.235740+0000","last_active":"2024-04-27T03:21:45.235740+0000","last_peered":"2024-04-27T03:21:45.235740+0000","last_clean":"2024-04-27T03:21:45.235740+0000","last_became_active":"2024-04-27T03:17:55.070625+0000","last_became_peered":"2024-04-27T03:17:55.070625+0000","last_unstale":"2024-04-27T03:21:45.235740+0000","last_undegraded":"2024-04-27T03:21:45.235740+0000","last_fullsized":"2024-04-27T03:21:45.235740+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:45.235457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:45.235457+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.235457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:42.520721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"31'9","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.523804+0000","last_change":"2024-04-27T03:21:36.523804+0000","last_active":"2024-04-27T03:21:36.523804+0000","last_peered":"2024-04-27T03:21:36.523804+0000","last_clean":"2024-04-27T03:21:36.523804+0000","last_became_active":"2024-04-27T03:17:57.061573+0000","last_became_peered":"2024-04-27T03:17:57.061573+0000","last_unstale":"2024-04-27T03:21:36.523804+0000","last_undegraded":"2024-04-27T03:21:36.523804+0000","last_fullsized":"2024-04-27T03:21:36.523804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:36.523556+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:36.523556+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.523556+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:56:00.443161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":13,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'8","reported_seq":74,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.191335+0000","last_change":"2024-04-27T03:21:52.191335+0000","last_active":"2024-04-27T03:21:52.191335+0000","last_peered":"2024-04-27T03:21:52.191335+0000","last_clean":"2024-04-27T03:21:52.191335+0000","last_became_active":"2024-04-27T03:17:49.013140+0000","last_became_peered":"2024-04-27T03:17:49.013140+0000","last_unstale":"2024-04-27T03:21:52.191335+0000","last_undegraded":"2024-04-27T03:21:52.191335+0000","last_fullsized":"2024-04-27T03:21:52.191335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:52.191088+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:52.191088+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.191088+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:24.826476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"31'119","reported_seq":213,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:03.477400+0000","last_change":"2024-04-27T03:22:03.477578+0000","last_active":"2024-04-27T03:22:03.477400+0000","last_peered":"2024-04-27T03:22:03.477400+0000","last_clean":"2024-04-27T03:22:03.477400+0000","last_became_active":"2024-04-27T03:17:39.633249+0000","last_became_peered":"2024-04-27T03:17:39.633249+0000","last_unstale":"2024-04-27T03:22:03.477400+0000","last_undegraded":"2024-04-27T03:22:03.477400+0000","last_fullsized":"2024-04-27T03:22:03.477400+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'119","last_scrub_stamp":"2024-04-27T03:22:03.477178+0000","last_deep_scrub":"31'119","last_deep_scrub_stamp":"2024-04-27T03:22:03.477178+0000","last_clean_scrub_stamp":"2024-04-27T03:22:03.477178+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:08:33.020052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:13.313557+0000","last_change":"2024-04-27T03:22:13.313665+0000","last_active":"2024-04-27T03:22:13.313557+0000","last_peered":"2024-04-27T03:22:13.313557+0000","last_clean":"2024-04-27T03:22:13.313557+0000","last_became_active":"2024-04-27T03:17:39.633990+0000","last_became_peered":"2024-04-27T03:17:39.633990+0000","last_unstale":"2024-04-27T03:22:13.313557+0000","last_undegraded":"2024-04-27T03:22:13.313557+0000","last_fullsized":"2024-04-27T03:22:13.313557+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:13.313328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:13.313328+0000","last_clean_scrub_stamp":"2024-04-27T03:22:13.313328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:40:15.739667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.759769+0000","last_change":"2024-04-27T03:22:00.759847+0000","last_active":"2024-04-27T03:22:00.759769+0000","last_peered":"2024-04-27T03:22:00.759769+0000","last_clean":"2024-04-27T03:22:00.759769+0000","last_became_active":"2024-04-27T03:17:47.039242+0000","last_became_peered":"2024-04-27T03:17:47.039242+0000","last_unstale":"2024-04-27T03:22:00.759769+0000","last_undegraded":"2024-04-27T03:22:00.759769+0000","last_fullsized":"2024-04-27T03:22:00.759769+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:00.759559+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:00.759559+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.759559+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:35:38.674495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.278617+0000","last_change":"2024-04-27T03:21:25.278617+0000","last_active":"2024-04-27T03:21:25.278617+0000","last_peered":"2024-04-27T03:21:25.278617+0000","last_clean":"2024-04-27T03:21:25.278617+0000","last_became_active":"2024-04-27T03:17:51.036227+0000","last_became_peered":"2024-04-27T03:17:51.036227+0000","last_unstale":"2024-04-27T03:21:25.278617+0000","last_undegraded":"2024-04-27T03:21:25.278617+0000","last_fullsized":"2024-04-27T03:21:25.278617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.278344+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.278344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:54.825766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"31'94","reported_seq":195,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.155217+0000","last_change":"2024-04-27T03:21:28.155217+0000","last_active":"2024-04-27T03:21:28.155217+0000","last_peered":"2024-04-27T03:21:28.155217+0000","last_clean":"2024-04-27T03:21:28.155217+0000","last_became_active":"2024-04-27T03:17:53.050577+0000","last_became_peered":"2024-04-27T03:17:53.050577+0000","last_unstale":"2024-04-27T03:21:28.155217+0000","last_undegraded":"2024-04-27T03:21:28.155217+0000","last_fullsized":"2024-04-27T03:21:28.155217+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'94","last_scrub_stamp":"2024-04-27T03:21:28.154943+0000","last_deep_scrub":"31'94","last_deep_scrub_stamp":"2024-04-27T03:21:28.154943+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.154943+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:02.805683+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":81,"num_write":34,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.635175+0000","last_change":"2024-04-27T03:21:25.635175+0000","last_active":"2024-04-27T03:21:25.635175+0000","last_peered":"2024-04-27T03:21:25.635175+0000","last_clean":"2024-04-27T03:21:25.635175+0000","last_became_active":"2024-04-27T03:17:55.071045+0000","last_became_peered":"2024-04-27T03:17:55.071045+0000","last_unstale":"2024-04-27T03:21:25.635175+0000","last_undegraded":"2024-04-27T03:21:25.635175+0000","last_fullsized":"2024-04-27T03:21:25.635175+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:25.634932+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.634932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:40.208378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"31'82","reported_seq":177,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.825636+0000","last_change":"2024-04-27T03:21:21.825636+0000","last_active":"2024-04-27T03:21:21.825636+0000","last_peered":"2024-04-27T03:21:21.825636+0000","last_clean":"2024-04-27T03:21:21.825636+0000","last_became_active":"2024-04-27T03:17:57.074702+0000","last_became_peered":"2024-04-27T03:17:57.074702+0000","last_unstale":"2024-04-27T03:21:21.825636+0000","last_undegraded":"2024-04-27T03:21:21.825636+0000","last_fullsized":"2024-04-27T03:21:21.825636+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'82","last_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_deep_scrub":"31'82","last_deep_scrub_stamp":"2024-04-27T03:21:21.825358+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.825358+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:31:11.855020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":215,"num_read_kb":213,"num_write":157,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"31'10","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:04.680259+0000","last_change":"2024-04-27T03:22:04.680259+0000","last_active":"2024-04-27T03:22:04.680259+0000","last_peered":"2024-04-27T03:22:04.680259+0000","last_clean":"2024-04-27T03:22:04.680259+0000","last_became_active":"2024-04-27T03:17:49.005196+0000","last_became_peered":"2024-04-27T03:17:49.005196+0000","last_unstale":"2024-04-27T03:22:04.680259+0000","last_undegraded":"2024-04-27T03:22:04.680259+0000","last_fullsized":"2024-04-27T03:22:04.680259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:22:04.680009+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:22:04.680009+0000","last_clean_scrub_stamp":"2024-04-27T03:22:04.680009+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:52.086768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:15.336414+0000","last_change":"2024-04-27T03:22:15.336470+0000","last_active":"2024-04-27T03:22:15.336414+0000","last_peered":"2024-04-27T03:22:15.336414+0000","last_clean":"2024-04-27T03:22:15.336414+0000","last_became_active":"2024-04-27T03:17:39.636895+0000","last_became_peered":"2024-04-27T03:17:39.636895+0000","last_unstale":"2024-04-27T03:22:15.336414+0000","last_undegraded":"2024-04-27T03:22:15.336414+0000","last_fullsized":"2024-04-27T03:22:15.336414+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:15.336208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:15.336208+0000","last_clean_scrub_stamp":"2024-04-27T03:22:15.336208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:10:59.066912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:08.381664+0000","last_change":"2024-04-27T03:22:08.381725+0000","last_active":"2024-04-27T03:22:08.381664+0000","last_peered":"2024-04-27T03:22:08.381664+0000","last_clean":"2024-04-27T03:22:08.381664+0000","last_became_active":"2024-04-27T03:17:47.012363+0000","last_became_peered":"2024-04-27T03:17:47.012363+0000","last_unstale":"2024-04-27T03:22:08.381664+0000","last_undegraded":"2024-04-27T03:22:08.381664+0000","last_fullsized":"2024-04-27T03:22:08.381664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:08.381446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:08.381446+0000","last_clean_scrub_stamp":"2024-04-27T03:22:08.381446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:13.097476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.174150+0000","last_change":"2024-04-27T03:21:18.174150+0000","last_active":"2024-04-27T03:21:18.174150+0000","last_peered":"2024-04-27T03:21:18.174150+0000","last_clean":"2024-04-27T03:21:18.174150+0000","last_became_active":"2024-04-27T03:17:51.033101+0000","last_became_peered":"2024-04-27T03:17:51.033101+0000","last_unstale":"2024-04-27T03:21:18.174150+0000","last_undegraded":"2024-04-27T03:21:18.174150+0000","last_fullsized":"2024-04-27T03:21:18.174150+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:18.173908+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.173908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:30.769576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"31'297","reported_seq":534,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.209173+0000","last_change":"2024-04-27T03:21:17.209173+0000","last_active":"2024-04-27T03:21:17.209173+0000","last_peered":"2024-04-27T03:21:17.209173+0000","last_clean":"2024-04-27T03:21:17.209173+0000","last_became_active":"2024-04-27T03:17:53.057188+0000","last_became_peered":"2024-04-27T03:17:53.057188+0000","last_unstale":"2024-04-27T03:21:17.209173+0000","last_undegraded":"2024-04-27T03:21:17.209173+0000","last_fullsized":"2024-04-27T03:21:17.209173+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'297","last_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_deep_scrub":"31'297","last_deep_scrub_stamp":"2024-04-27T03:21:17.208885+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.208885+0000","objects_scrubbed":11,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:24.543836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":387,"num_read_kb":371,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.591170+0000","last_change":"2024-04-27T03:21:22.591170+0000","last_active":"2024-04-27T03:21:22.591170+0000","last_peered":"2024-04-27T03:21:22.591170+0000","last_clean":"2024-04-27T03:21:22.591170+0000","last_became_active":"2024-04-27T03:17:55.073895+0000","last_became_peered":"2024-04-27T03:17:55.073895+0000","last_unstale":"2024-04-27T03:21:22.591170+0000","last_undegraded":"2024-04-27T03:21:22.591170+0000","last_fullsized":"2024-04-27T03:21:22.591170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.590921+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.590921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:46:10.482862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"29'7","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:12.258615+0000","last_change":"2024-04-27T03:22:12.258615+0000","last_active":"2024-04-27T03:22:12.258615+0000","last_peered":"2024-04-27T03:22:12.258615+0000","last_clean":"2024-04-27T03:22:12.258615+0000","last_became_active":"2024-04-27T03:17:57.085555+0000","last_became_peered":"2024-04-27T03:17:57.085555+0000","last_unstale":"2024-04-27T03:22:12.258615+0000","last_undegraded":"2024-04-27T03:22:12.258615+0000","last_fullsized":"2024-04-27T03:22:12.258615+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'7","last_scrub_stamp":"2024-04-27T03:22:12.258329+0000","last_deep_scrub":"29'7","last_deep_scrub_stamp":"2024-04-27T03:22:12.258329+0000","last_clean_scrub_stamp":"2024-04-27T03:22:12.258329+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:48:04.136414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"31'8","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.558256+0000","last_change":"2024-04-27T03:21:37.558256+0000","last_active":"2024-04-27T03:21:37.558256+0000","last_peered":"2024-04-27T03:21:37.558256+0000","last_clean":"2024-04-27T03:21:37.558256+0000","last_became_active":"2024-04-27T03:17:49.012212+0000","last_became_peered":"2024-04-27T03:17:49.012212+0000","last_unstale":"2024-04-27T03:21:37.558256+0000","last_undegraded":"2024-04-27T03:21:37.558256+0000","last_fullsized":"2024-04-27T03:21:37.558256+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:37.557861+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:37.557861+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.557861+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:29.852192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.792507+0000","last_change":"2024-04-27T03:21:47.792561+0000","last_active":"2024-04-27T03:21:47.792507+0000","last_peered":"2024-04-27T03:21:47.792507+0000","last_clean":"2024-04-27T03:21:47.792507+0000","last_became_active":"2024-04-27T03:17:39.633380+0000","last_became_peered":"2024-04-27T03:17:39.633380+0000","last_unstale":"2024-04-27T03:21:47.792507+0000","last_undegraded":"2024-04-27T03:21:47.792507+0000","last_fullsized":"2024-04-27T03:21:47.792507+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:47.792322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:47.792322+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.792322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:15:30.180059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:07.414650+0000","last_change":"2024-04-27T03:22:07.414713+0000","last_active":"2024-04-27T03:22:07.414650+0000","last_peered":"2024-04-27T03:22:07.414650+0000","last_clean":"2024-04-27T03:22:07.414650+0000","last_became_active":"2024-04-27T03:17:47.010876+0000","last_became_peered":"2024-04-27T03:17:47.010876+0000","last_unstale":"2024-04-27T03:22:07.414650+0000","last_undegraded":"2024-04-27T03:22:07.414650+0000","last_fullsized":"2024-04-27T03:22:07.414650+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:07.414454+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:07.414454+0000","last_clean_scrub_stamp":"2024-04-27T03:22:07.414454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:59.177596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":8192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"31'2","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.787965+0000","last_change":"2024-04-27T03:21:41.787965+0000","last_active":"2024-04-27T03:21:41.787965+0000","last_peered":"2024-04-27T03:21:41.787965+0000","last_clean":"2024-04-27T03:21:41.787965+0000","last_became_active":"2024-04-27T03:17:51.029895+0000","last_became_peered":"2024-04-27T03:17:51.029895+0000","last_unstale":"2024-04-27T03:21:41.787965+0000","last_undegraded":"2024-04-27T03:21:41.787965+0000","last_fullsized":"2024-04-27T03:21:41.787965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:41.787706+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:41.787706+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.787706+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:24:51.133885+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"31'23","reported_seq":106,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.192744+0000","last_change":"2024-04-27T03:21:27.192744+0000","last_active":"2024-04-27T03:21:27.192744+0000","last_peered":"2024-04-27T03:21:27.192744+0000","last_clean":"2024-04-27T03:21:27.192744+0000","last_became_active":"2024-04-27T03:17:53.050590+0000","last_became_peered":"2024-04-27T03:17:53.050590+0000","last_unstale":"2024-04-27T03:21:27.192744+0000","last_undegraded":"2024-04-27T03:21:27.192744+0000","last_fullsized":"2024-04-27T03:21:27.192744+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'23","last_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_deep_scrub":"31'23","last_deep_scrub_stamp":"2024-04-27T03:21:27.192466+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.192466+0000","objects_scrubbed":8,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:41.765912+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":13,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.232354+0000","last_change":"2024-04-27T03:21:16.232354+0000","last_active":"2024-04-27T03:21:16.232354+0000","last_peered":"2024-04-27T03:21:16.232354+0000","last_clean":"2024-04-27T03:21:16.232354+0000","last_became_active":"2024-04-27T03:17:55.069295+0000","last_became_peered":"2024-04-27T03:17:55.069295+0000","last_unstale":"2024-04-27T03:21:16.232354+0000","last_undegraded":"2024-04-27T03:21:16.232354+0000","last_fullsized":"2024-04-27T03:21:16.232354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.232084+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.232084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:47:05.186600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"31'4","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.711568+0000","last_change":"2024-04-27T03:21:52.711568+0000","last_active":"2024-04-27T03:21:52.711568+0000","last_peered":"2024-04-27T03:21:52.711568+0000","last_clean":"2024-04-27T03:21:52.711568+0000","last_became_active":"2024-04-27T03:17:57.069805+0000","last_became_peered":"2024-04-27T03:17:57.069805+0000","last_unstale":"2024-04-27T03:21:52.711568+0000","last_undegraded":"2024-04-27T03:21:52.711568+0000","last_fullsized":"2024-04-27T03:21:52.711568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:52.711315+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:52.711315+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.711315+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:37:09.185124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"31'10","reported_seq":77,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:45.564265+0000","last_change":"2024-04-27T03:21:45.564265+0000","last_active":"2024-04-27T03:21:45.564265+0000","last_peered":"2024-04-27T03:21:45.564265+0000","last_clean":"2024-04-27T03:21:45.564265+0000","last_became_active":"2024-04-27T03:17:49.009084+0000","last_became_peered":"2024-04-27T03:17:49.009084+0000","last_unstale":"2024-04-27T03:21:45.564265+0000","last_undegraded":"2024-04-27T03:21:45.564265+0000","last_fullsized":"2024-04-27T03:21:45.564265+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:45.564008+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:45.564008+0000","last_clean_scrub_stamp":"2024-04-27T03:21:45.564008+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:34:05.340194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":43,"num_read_kb":39,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:12.360787+0000","last_change":"2024-04-27T03:22:12.360848+0000","last_active":"2024-04-27T03:22:12.360787+0000","last_peered":"2024-04-27T03:22:12.360787+0000","last_clean":"2024-04-27T03:22:12.360787+0000","last_became_active":"2024-04-27T03:17:39.633003+0000","last_became_peered":"2024-04-27T03:17:39.633003+0000","last_unstale":"2024-04-27T03:22:12.360787+0000","last_undegraded":"2024-04-27T03:22:12.360787+0000","last_fullsized":"2024-04-27T03:22:12.360787+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:12.360549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:12.360549+0000","last_clean_scrub_stamp":"2024-04-27T03:22:12.360549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:11.673649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"31'4","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.285465+0000","last_change":"2024-04-27T03:22:02.285533+0000","last_active":"2024-04-27T03:22:02.285465+0000","last_peered":"2024-04-27T03:22:02.285465+0000","last_clean":"2024-04-27T03:22:02.285465+0000","last_became_active":"2024-04-27T03:17:47.013957+0000","last_became_peered":"2024-04-27T03:17:47.013957+0000","last_unstale":"2024-04-27T03:22:02.285465+0000","last_undegraded":"2024-04-27T03:22:02.285465+0000","last_fullsized":"2024-04-27T03:22:02.285465+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:02.285271+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:02.285271+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.285271+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:45.954095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":8,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.367056+0000","last_change":"2024-04-27T03:21:30.367056+0000","last_active":"2024-04-27T03:21:30.367056+0000","last_peered":"2024-04-27T03:21:30.367056+0000","last_clean":"2024-04-27T03:21:30.367056+0000","last_became_active":"2024-04-27T03:17:51.028844+0000","last_became_peered":"2024-04-27T03:17:51.028844+0000","last_unstale":"2024-04-27T03:21:30.367056+0000","last_undegraded":"2024-04-27T03:21:30.367056+0000","last_fullsized":"2024-04-27T03:21:30.367056+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:30.366712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:30.366712+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.366712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:19:52.016724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"31'21","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.748898+0000","last_change":"2024-04-27T03:21:33.748898+0000","last_active":"2024-04-27T03:21:33.748898+0000","last_peered":"2024-04-27T03:21:33.748898+0000","last_clean":"2024-04-27T03:21:33.748898+0000","last_became_active":"2024-04-27T03:17:53.041777+0000","last_became_peered":"2024-04-27T03:17:53.041777+0000","last_unstale":"2024-04-27T03:21:33.748898+0000","last_undegraded":"2024-04-27T03:21:33.748898+0000","last_fullsized":"2024-04-27T03:21:33.748898+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'21","last_scrub_stamp":"2024-04-27T03:21:33.748366+0000","last_deep_scrub":"31'21","last_deep_scrub_stamp":"2024-04-27T03:21:33.748366+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.748366+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:49.413176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":350,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:40.816658+0000","last_change":"2024-04-27T03:21:40.816658+0000","last_active":"2024-04-27T03:21:40.816658+0000","last_peered":"2024-04-27T03:21:40.816658+0000","last_clean":"2024-04-27T03:21:40.816658+0000","last_became_active":"2024-04-27T03:17:55.077945+0000","last_became_peered":"2024-04-27T03:17:55.077945+0000","last_unstale":"2024-04-27T03:21:40.816658+0000","last_undegraded":"2024-04-27T03:21:40.816658+0000","last_fullsized":"2024-04-27T03:21:40.816658+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:40.816420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:40.816420+0000","last_clean_scrub_stamp":"2024-04-27T03:21:40.816420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:44.183666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.8","version":"31'6","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:16.365098+0000","last_change":"2024-04-27T03:22:16.365098+0000","last_active":"2024-04-27T03:22:16.365098+0000","last_peered":"2024-04-27T03:22:16.365098+0000","last_clean":"2024-04-27T03:22:16.365098+0000","last_became_active":"2024-04-27T03:17:57.088637+0000","last_became_peered":"2024-04-27T03:17:57.088637+0000","last_unstale":"2024-04-27T03:22:16.365098+0000","last_undegraded":"2024-04-27T03:22:16.365098+0000","last_fullsized":"2024-04-27T03:22:16.365098+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:16.364832+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:16.364832+0000","last_clean_scrub_stamp":"2024-04-27T03:22:16.364832+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:59.046911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"31'20","reported_seq":73,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.437652+0000","last_change":"2024-04-27T03:21:58.437652+0000","last_active":"2024-04-27T03:21:58.437652+0000","last_peered":"2024-04-27T03:21:58.437652+0000","last_clean":"2024-04-27T03:21:58.437652+0000","last_became_active":"2024-04-27T03:17:49.004818+0000","last_became_peered":"2024-04-27T03:17:49.004818+0000","last_unstale":"2024-04-27T03:21:58.437652+0000","last_undegraded":"2024-04-27T03:21:58.437652+0000","last_fullsized":"2024-04-27T03:21:58.437652+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'20","last_scrub_stamp":"2024-04-27T03:21:58.437355+0000","last_deep_scrub":"31'20","last_deep_scrub_stamp":"2024-04-27T03:21:58.437355+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.437355+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:08.660221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":66,"num_read_kb":50,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.254082+0000","last_change":"2024-04-27T03:21:59.254142+0000","last_active":"2024-04-27T03:21:59.254082+0000","last_peered":"2024-04-27T03:21:59.254082+0000","last_clean":"2024-04-27T03:21:59.254082+0000","last_became_active":"2024-04-27T03:17:47.023053+0000","last_became_peered":"2024-04-27T03:17:47.023053+0000","last_unstale":"2024-04-27T03:21:59.254082+0000","last_undegraded":"2024-04-27T03:21:59.254082+0000","last_fullsized":"2024-04-27T03:21:59.254082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:59.253884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:59.253884+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.253884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:26:21.043164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:04.467449+0000","last_change":"2024-04-27T03:22:04.467585+0000","last_active":"2024-04-27T03:22:04.467449+0000","last_peered":"2024-04-27T03:22:04.467449+0000","last_clean":"2024-04-27T03:22:04.467449+0000","last_became_active":"2024-04-27T03:17:39.635203+0000","last_became_peered":"2024-04-27T03:17:39.635203+0000","last_unstale":"2024-04-27T03:22:04.467449+0000","last_undegraded":"2024-04-27T03:22:04.467449+0000","last_fullsized":"2024-04-27T03:22:04.467449+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:04.467218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:04.467218+0000","last_clean_scrub_stamp":"2024-04-27T03:22:04.467218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:14.140096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"31'2","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.757928+0000","last_change":"2024-04-27T03:21:55.757928+0000","last_active":"2024-04-27T03:21:55.757928+0000","last_peered":"2024-04-27T03:21:55.757928+0000","last_clean":"2024-04-27T03:21:55.757928+0000","last_became_active":"2024-04-27T03:17:51.037302+0000","last_became_peered":"2024-04-27T03:17:51.037302+0000","last_unstale":"2024-04-27T03:21:55.757928+0000","last_undegraded":"2024-04-27T03:21:55.757928+0000","last_fullsized":"2024-04-27T03:21:55.757928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:55.757617+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:55.757617+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.757617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:47:48.305961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"31'87","reported_seq":180,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:18.199826+0000","last_change":"2024-04-27T03:21:18.199826+0000","last_active":"2024-04-27T03:21:18.199826+0000","last_peered":"2024-04-27T03:21:18.199826+0000","last_clean":"2024-04-27T03:21:18.199826+0000","last_became_active":"2024-04-27T03:17:53.053708+0000","last_became_peered":"2024-04-27T03:17:53.053708+0000","last_unstale":"2024-04-27T03:21:18.199826+0000","last_undegraded":"2024-04-27T03:21:18.199826+0000","last_fullsized":"2024-04-27T03:21:18.199826+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'87","last_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_deep_scrub":"31'87","last_deep_scrub_stamp":"2024-04-27T03:21:18.199538+0000","last_clean_scrub_stamp":"2024-04-27T03:21:18.199538+0000","objects_scrubbed":11,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:49.872043+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":89,"num_read_kb":75,"num_write":42,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":292,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.101226+0000","last_change":"2024-04-27T03:21:29.101226+0000","last_active":"2024-04-27T03:21:29.101226+0000","last_peered":"2024-04-27T03:21:29.101226+0000","last_clean":"2024-04-27T03:21:29.101226+0000","last_became_active":"2024-04-27T03:17:55.067924+0000","last_became_peered":"2024-04-27T03:17:55.067924+0000","last_unstale":"2024-04-27T03:21:29.101226+0000","last_undegraded":"2024-04-27T03:21:29.101226+0000","last_fullsized":"2024-04-27T03:21:29.101226+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:29.100967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:29.100967+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.100967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:02.862694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"31'9","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.745242+0000","last_change":"2024-04-27T03:21:15.745242+0000","last_active":"2024-04-27T03:21:15.745242+0000","last_peered":"2024-04-27T03:21:15.745242+0000","last_clean":"2024-04-27T03:21:15.745242+0000","last_became_active":"2024-04-27T03:17:57.066929+0000","last_became_peered":"2024-04-27T03:17:57.066929+0000","last_unstale":"2024-04-27T03:21:15.745242+0000","last_undegraded":"2024-04-27T03:21:15.745242+0000","last_fullsized":"2024-04-27T03:21:15.745242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:15.744992+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.744992+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:00.286559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"29'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.623119+0000","last_change":"2024-04-27T03:21:43.623119+0000","last_active":"2024-04-27T03:21:43.623119+0000","last_peered":"2024-04-27T03:21:43.623119+0000","last_clean":"2024-04-27T03:21:43.623119+0000","last_became_active":"2024-04-27T03:17:49.009511+0000","last_became_peered":"2024-04-27T03:17:49.009511+0000","last_unstale":"2024-04-27T03:21:43.623119+0000","last_undegraded":"2024-04-27T03:21:43.623119+0000","last_fullsized":"2024-04-27T03:21:43.623119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:43.622863+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:43.622863+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.622863+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:25.488085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"31'6","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.510055+0000","last_change":"2024-04-27T03:21:49.510055+0000","last_active":"2024-04-27T03:21:49.510055+0000","last_peered":"2024-04-27T03:21:49.510055+0000","last_clean":"2024-04-27T03:21:49.510055+0000","last_became_active":"2024-04-27T03:17:47.027655+0000","last_became_peered":"2024-04-27T03:17:47.027655+0000","last_unstale":"2024-04-27T03:21:49.510055+0000","last_undegraded":"2024-04-27T03:21:49.510055+0000","last_fullsized":"2024-04-27T03:21:49.510055+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:49.509757+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:49.509757+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.509757+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:04:00.582652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:14.293894+0000","last_change":"2024-04-27T03:22:14.293932+0000","last_active":"2024-04-27T03:22:14.293894+0000","last_peered":"2024-04-27T03:22:14.293894+0000","last_clean":"2024-04-27T03:22:14.293894+0000","last_became_active":"2024-04-27T03:17:39.635169+0000","last_became_peered":"2024-04-27T03:17:39.635169+0000","last_unstale":"2024-04-27T03:22:14.293894+0000","last_undegraded":"2024-04-27T03:22:14.293894+0000","last_fullsized":"2024-04-27T03:22:14.293894+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:14.293773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:14.293773+0000","last_clean_scrub_stamp":"2024-04-27T03:22:14.293773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:19:02.581420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.128384+0000","last_change":"2024-04-27T03:21:16.128384+0000","last_active":"2024-04-27T03:21:16.128384+0000","last_peered":"2024-04-27T03:21:16.128384+0000","last_clean":"2024-04-27T03:21:16.128384+0000","last_became_active":"2024-04-27T03:17:51.035168+0000","last_became_peered":"2024-04-27T03:17:51.035168+0000","last_unstale":"2024-04-27T03:21:16.128384+0000","last_undegraded":"2024-04-27T03:21:16.128384+0000","last_fullsized":"2024-04-27T03:21:16.128384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.128135+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.128135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:44.502862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"31'28","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:21.261734+0000","last_change":"2024-04-27T03:21:21.261734+0000","last_active":"2024-04-27T03:21:21.261734+0000","last_peered":"2024-04-27T03:21:21.261734+0000","last_clean":"2024-04-27T03:21:21.261734+0000","last_became_active":"2024-04-27T03:17:53.048890+0000","last_became_peered":"2024-04-27T03:17:53.048890+0000","last_unstale":"2024-04-27T03:21:21.261734+0000","last_undegraded":"2024-04-27T03:21:21.261734+0000","last_fullsized":"2024-04-27T03:21:21.261734+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'28","last_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_deep_scrub":"31'28","last_deep_scrub_stamp":"2024-04-27T03:21:21.261420+0000","last_clean_scrub_stamp":"2024-04-27T03:21:21.261420+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:52.474587+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"31'285","reported_seq":374,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.251366+0000","last_change":"2024-04-27T03:21:24.251366+0000","last_active":"2024-04-27T03:21:24.251366+0000","last_peered":"2024-04-27T03:21:24.251366+0000","last_clean":"2024-04-27T03:21:24.251366+0000","last_became_active":"2024-04-27T03:17:55.076668+0000","last_became_peered":"2024-04-27T03:17:55.076668+0000","last_unstale":"2024-04-27T03:21:24.251366+0000","last_undegraded":"2024-04-27T03:21:24.251366+0000","last_fullsized":"2024-04-27T03:21:24.251366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:24.250979+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.250979+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:09:52.625705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"31'4","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:25.357796+0000","last_change":"2024-04-27T03:22:25.357796+0000","last_active":"2024-04-27T03:22:25.357796+0000","last_peered":"2024-04-27T03:22:25.357796+0000","last_clean":"2024-04-27T03:22:25.357796+0000","last_became_active":"2024-04-27T03:17:57.090548+0000","last_became_peered":"2024-04-27T03:17:57.090548+0000","last_unstale":"2024-04-27T03:22:25.357796+0000","last_undegraded":"2024-04-27T03:22:25.357796+0000","last_fullsized":"2024-04-27T03:22:25.357796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:25.357501+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:25.357501+0000","last_clean_scrub_stamp":"2024-04-27T03:22:25.357501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:26:40.821592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'6","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:06.636731+0000","last_change":"2024-04-27T03:22:06.636795+0000","last_active":"2024-04-27T03:22:06.636731+0000","last_peered":"2024-04-27T03:22:06.636731+0000","last_clean":"2024-04-27T03:22:06.636731+0000","last_became_active":"2024-04-27T03:17:49.004491+0000","last_became_peered":"2024-04-27T03:17:49.004491+0000","last_unstale":"2024-04-27T03:22:06.636731+0000","last_undegraded":"2024-04-27T03:22:06.636731+0000","last_fullsized":"2024-04-27T03:22:06.636731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:06.636531+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:06.636531+0000","last_clean_scrub_stamp":"2024-04-27T03:22:06.636531+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:49.318611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.761324+0000","last_change":"2024-04-27T03:21:56.761324+0000","last_active":"2024-04-27T03:21:56.761324+0000","last_peered":"2024-04-27T03:21:56.761324+0000","last_clean":"2024-04-27T03:21:56.761324+0000","last_became_active":"2024-04-27T03:17:47.038677+0000","last_became_peered":"2024-04-27T03:17:47.038677+0000","last_unstale":"2024-04-27T03:21:56.761324+0000","last_undegraded":"2024-04-27T03:21:56.761324+0000","last_fullsized":"2024-04-27T03:21:56.761324+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:56.761068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:56.761068+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.761068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:46.693610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.476853+0000","last_change":"2024-04-27T03:22:02.476905+0000","last_active":"2024-04-27T03:22:02.476853+0000","last_peered":"2024-04-27T03:22:02.476853+0000","last_clean":"2024-04-27T03:22:02.476853+0000","last_became_active":"2024-04-27T03:17:39.629061+0000","last_became_peered":"2024-04-27T03:17:39.629061+0000","last_unstale":"2024-04-27T03:22:02.476853+0000","last_undegraded":"2024-04-27T03:22:02.476853+0000","last_fullsized":"2024-04-27T03:22:02.476853+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:02.476650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:02.476650+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.476650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:37:18.950460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.178222+0000","last_change":"2024-04-27T03:21:35.178222+0000","last_active":"2024-04-27T03:21:35.178222+0000","last_peered":"2024-04-27T03:21:35.178222+0000","last_clean":"2024-04-27T03:21:35.178222+0000","last_became_active":"2024-04-27T03:17:51.019321+0000","last_became_peered":"2024-04-27T03:17:51.019321+0000","last_unstale":"2024-04-27T03:21:35.178222+0000","last_undegraded":"2024-04-27T03:21:35.178222+0000","last_fullsized":"2024-04-27T03:21:35.178222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:35.177942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:35.177942+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.177942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:34:50.842753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"31'22","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.688786+0000","last_change":"2024-04-27T03:21:29.688786+0000","last_active":"2024-04-27T03:21:29.688786+0000","last_peered":"2024-04-27T03:21:29.688786+0000","last_clean":"2024-04-27T03:21:29.688786+0000","last_became_active":"2024-04-27T03:17:53.046423+0000","last_became_peered":"2024-04-27T03:17:53.046423+0000","last_unstale":"2024-04-27T03:21:29.688786+0000","last_undegraded":"2024-04-27T03:21:29.688786+0000","last_fullsized":"2024-04-27T03:21:29.688786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'22","last_scrub_stamp":"2024-04-27T03:21:29.688480+0000","last_deep_scrub":"31'22","last_deep_scrub_stamp":"2024-04-27T03:21:29.688480+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.688480+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:50:51.784245+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"31'285","reported_seq":430,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.765171+0000","last_change":"2024-04-27T03:21:36.765171+0000","last_active":"2024-04-27T03:21:36.765171+0000","last_peered":"2024-04-27T03:21:36.765171+0000","last_clean":"2024-04-27T03:21:36.765171+0000","last_became_active":"2024-04-27T03:17:55.074768+0000","last_became_peered":"2024-04-27T03:17:55.074768+0000","last_unstale":"2024-04-27T03:21:36.765171+0000","last_undegraded":"2024-04-27T03:21:36.765171+0000","last_fullsized":"2024-04-27T03:21:36.765171+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:36.764897+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:36.764897+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.764897+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:07.152179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"27'3","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:13.306697+0000","last_change":"2024-04-27T03:22:13.306697+0000","last_active":"2024-04-27T03:22:13.306697+0000","last_peered":"2024-04-27T03:22:13.306697+0000","last_clean":"2024-04-27T03:22:13.306697+0000","last_became_active":"2024-04-27T03:17:57.086177+0000","last_became_peered":"2024-04-27T03:17:57.086177+0000","last_unstale":"2024-04-27T03:22:13.306697+0000","last_undegraded":"2024-04-27T03:22:13.306697+0000","last_fullsized":"2024-04-27T03:22:13.306697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'3","last_scrub_stamp":"2024-04-27T03:22:13.306435+0000","last_deep_scrub":"27'3","last_deep_scrub_stamp":"2024-04-27T03:22:13.306435+0000","last_clean_scrub_stamp":"2024-04-27T03:22:13.306435+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:21:08.301802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"31'14","reported_seq":113,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.791930+0000","last_change":"2024-04-27T03:21:44.791930+0000","last_active":"2024-04-27T03:21:44.791930+0000","last_peered":"2024-04-27T03:21:44.791930+0000","last_clean":"2024-04-27T03:21:44.791930+0000","last_became_active":"2024-04-27T03:17:49.003955+0000","last_became_peered":"2024-04-27T03:17:49.003955+0000","last_unstale":"2024-04-27T03:21:44.791930+0000","last_undegraded":"2024-04-27T03:21:44.791930+0000","last_fullsized":"2024-04-27T03:21:44.791930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'14","last_scrub_stamp":"2024-04-27T03:21:44.791685+0000","last_deep_scrub":"31'14","last_deep_scrub_stamp":"2024-04-27T03:21:44.791685+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.791685+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:37:12.652507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:05.459487+0000","last_change":"2024-04-27T03:22:05.459487+0000","last_active":"2024-04-27T03:22:05.459487+0000","last_peered":"2024-04-27T03:22:05.459487+0000","last_clean":"2024-04-27T03:22:05.459487+0000","last_became_active":"2024-04-27T03:17:47.000484+0000","last_became_peered":"2024-04-27T03:17:47.000484+0000","last_unstale":"2024-04-27T03:22:05.459487+0000","last_undegraded":"2024-04-27T03:22:05.459487+0000","last_fullsized":"2024-04-27T03:22:05.459487+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:05.459218+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:05.459218+0000","last_clean_scrub_stamp":"2024-04-27T03:22:05.459218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:54.776239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:11.379145+0000","last_change":"2024-04-27T03:22:11.379264+0000","last_active":"2024-04-27T03:22:11.379145+0000","last_peered":"2024-04-27T03:22:11.379145+0000","last_clean":"2024-04-27T03:22:11.379145+0000","last_became_active":"2024-04-27T03:17:39.633034+0000","last_became_peered":"2024-04-27T03:17:39.633034+0000","last_unstale":"2024-04-27T03:22:11.379145+0000","last_undegraded":"2024-04-27T03:22:11.379145+0000","last_fullsized":"2024-04-27T03:22:11.379145+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:11.378920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:11.378920+0000","last_clean_scrub_stamp":"2024-04-27T03:22:11.378920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:04:24.878661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.600253+0000","last_change":"2024-04-27T03:21:19.600253+0000","last_active":"2024-04-27T03:21:19.600253+0000","last_peered":"2024-04-27T03:21:19.600253+0000","last_clean":"2024-04-27T03:21:19.600253+0000","last_became_active":"2024-04-27T03:17:51.031812+0000","last_became_peered":"2024-04-27T03:17:51.031812+0000","last_unstale":"2024-04-27T03:21:19.600253+0000","last_undegraded":"2024-04-27T03:21:19.600253+0000","last_fullsized":"2024-04-27T03:21:19.600253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.600004+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.600004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:53:52.295521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"31'108","reported_seq":212,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.121323+0000","last_change":"2024-04-27T03:21:15.121323+0000","last_active":"2024-04-27T03:21:15.121323+0000","last_peered":"2024-04-27T03:21:15.121323+0000","last_clean":"2024-04-27T03:21:15.121323+0000","last_became_active":"2024-04-27T03:17:53.037461+0000","last_became_peered":"2024-04-27T03:17:53.037461+0000","last_unstale":"2024-04-27T03:21:15.121323+0000","last_undegraded":"2024-04-27T03:21:15.121323+0000","last_fullsized":"2024-04-27T03:21:15.121323+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'108","last_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_deep_scrub":"31'108","last_deep_scrub_stamp":"2024-04-27T03:21:15.121034+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.121034+0000","objects_scrubbed":9,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:20.807083+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4316,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":107,"num_read_kb":100,"num_write":44,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.753050+0000","last_change":"2024-04-27T03:21:44.753050+0000","last_active":"2024-04-27T03:21:44.753050+0000","last_peered":"2024-04-27T03:21:44.753050+0000","last_clean":"2024-04-27T03:21:44.753050+0000","last_became_active":"2024-04-27T03:17:55.081313+0000","last_became_peered":"2024-04-27T03:17:55.081313+0000","last_unstale":"2024-04-27T03:21:44.753050+0000","last_undegraded":"2024-04-27T03:21:44.753050+0000","last_fullsized":"2024-04-27T03:21:44.753050+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:44.752732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:44.752732+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.752732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:27.679483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.824647+0000","last_change":"2024-04-27T03:21:12.824647+0000","last_active":"2024-04-27T03:21:12.824647+0000","last_peered":"2024-04-27T03:21:12.824647+0000","last_clean":"2024-04-27T03:21:12.824647+0000","last_became_active":"2024-04-27T03:17:57.067799+0000","last_became_peered":"2024-04-27T03:17:57.067799+0000","last_unstale":"2024-04-27T03:21:12.824647+0000","last_undegraded":"2024-04-27T03:21:12.824647+0000","last_fullsized":"2024-04-27T03:21:12.824647+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.824385+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.824385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:28:25.571276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":31,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.779972+0000","last_change":"2024-04-27T03:21:14.779972+0000","last_active":"2024-04-27T03:21:14.779972+0000","last_peered":"2024-04-27T03:21:14.779972+0000","last_clean":"2024-04-27T03:21:14.779972+0000","last_became_active":"2024-04-27T03:17:57.067544+0000","last_became_peered":"2024-04-27T03:17:57.067544+0000","last_unstale":"2024-04-27T03:21:14.779972+0000","last_undegraded":"2024-04-27T03:21:14.779972+0000","last_fullsized":"2024-04-27T03:21:14.779972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.779705+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.779705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:11:09.535832+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"31'8","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:36.696548+0000","last_change":"2024-04-27T03:21:36.696548+0000","last_active":"2024-04-27T03:21:36.696548+0000","last_peered":"2024-04-27T03:21:36.696548+0000","last_clean":"2024-04-27T03:21:36.696548+0000","last_became_active":"2024-04-27T03:17:49.008983+0000","last_became_peered":"2024-04-27T03:17:49.008983+0000","last_unstale":"2024-04-27T03:21:36.696548+0000","last_undegraded":"2024-04-27T03:21:36.696548+0000","last_fullsized":"2024-04-27T03:21:36.696548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:36.696292+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:36.696292+0000","last_clean_scrub_stamp":"2024-04-27T03:21:36.696292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:12:08.368797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"31'4","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.229499+0000","last_change":"2024-04-27T03:21:56.229499+0000","last_active":"2024-04-27T03:21:56.229499+0000","last_peered":"2024-04-27T03:21:56.229499+0000","last_clean":"2024-04-27T03:21:56.229499+0000","last_became_active":"2024-04-27T03:17:47.022171+0000","last_became_peered":"2024-04-27T03:17:47.022171+0000","last_unstale":"2024-04-27T03:21:56.229499+0000","last_undegraded":"2024-04-27T03:21:56.229499+0000","last_fullsized":"2024-04-27T03:21:56.229499+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:56.229215+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:56.229215+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.229215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:12:37.561633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"31'2","reported_seq":235,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.286722+0000","last_change":"2024-04-27T03:21:26.286722+0000","last_active":"2024-04-27T03:21:26.286722+0000","last_peered":"2024-04-27T03:21:26.286722+0000","last_clean":"2024-04-27T03:21:26.286722+0000","last_became_active":"2024-04-27T03:17:51.031094+0000","last_became_peered":"2024-04-27T03:17:51.031094+0000","last_unstale":"2024-04-27T03:21:26.286722+0000","last_undegraded":"2024-04-27T03:21:26.286722+0000","last_fullsized":"2024-04-27T03:21:26.286722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:26.286462+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.286462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:36:54.147283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":285,"num_read_kb":285,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"31'46","reported_seq":113,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:15.542505+0000","last_change":"2024-04-27T03:21:15.542505+0000","last_active":"2024-04-27T03:21:15.542505+0000","last_peered":"2024-04-27T03:21:15.542505+0000","last_clean":"2024-04-27T03:21:15.542505+0000","last_became_active":"2024-04-27T03:17:53.040517+0000","last_became_peered":"2024-04-27T03:17:53.040517+0000","last_unstale":"2024-04-27T03:21:15.542505+0000","last_undegraded":"2024-04-27T03:21:15.542505+0000","last_fullsized":"2024-04-27T03:21:15.542505+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'46","last_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_deep_scrub":"31'46","last_deep_scrub_stamp":"2024-04-27T03:21:15.542196+0000","last_clean_scrub_stamp":"2024-04-27T03:21:15.542196+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:31.074928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.640360+0000","last_change":"2024-04-27T03:21:28.640422+0000","last_active":"2024-04-27T03:21:28.640360+0000","last_peered":"2024-04-27T03:21:28.640360+0000","last_clean":"2024-04-27T03:21:28.640360+0000","last_became_active":"2024-04-27T03:17:55.074257+0000","last_became_peered":"2024-04-27T03:17:55.074257+0000","last_unstale":"2024-04-27T03:21:28.640360+0000","last_undegraded":"2024-04-27T03:21:28.640360+0000","last_fullsized":"2024-04-27T03:21:28.640360+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:28.640146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:28.640146+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.640146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:44:20.934477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1e","version":"31'3","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:22.409178+0000","last_change":"2024-04-27T03:22:22.409242+0000","last_active":"2024-04-27T03:22:22.409178+0000","last_peered":"2024-04-27T03:22:22.409178+0000","last_clean":"2024-04-27T03:22:22.409178+0000","last_became_active":"2024-04-27T03:17:57.085872+0000","last_became_peered":"2024-04-27T03:17:57.085872+0000","last_unstale":"2024-04-27T03:22:22.409178+0000","last_undegraded":"2024-04-27T03:22:22.409178+0000","last_fullsized":"2024-04-27T03:22:22.409178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-04-27T03:22:22.408986+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-04-27T03:22:22.408986+0000","last_clean_scrub_stamp":"2024-04-27T03:22:22.408986+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:05.983827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"31'10","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.536831+0000","last_change":"2024-04-27T03:21:52.536831+0000","last_active":"2024-04-27T03:21:52.536831+0000","last_peered":"2024-04-27T03:21:52.536831+0000","last_clean":"2024-04-27T03:21:52.536831+0000","last_became_active":"2024-04-27T03:17:49.003186+0000","last_became_peered":"2024-04-27T03:17:49.003186+0000","last_unstale":"2024-04-27T03:21:52.536831+0000","last_undegraded":"2024-04-27T03:21:52.536831+0000","last_fullsized":"2024-04-27T03:21:52.536831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:21:52.536569+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:21:52.536569+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.536569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:49.821834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:52.725275+0000","last_change":"2024-04-27T03:21:52.725328+0000","last_active":"2024-04-27T03:21:52.725275+0000","last_peered":"2024-04-27T03:21:52.725275+0000","last_clean":"2024-04-27T03:21:52.725275+0000","last_became_active":"2024-04-27T03:17:47.021493+0000","last_became_peered":"2024-04-27T03:17:47.021493+0000","last_unstale":"2024-04-27T03:21:52.725275+0000","last_undegraded":"2024-04-27T03:21:52.725275+0000","last_fullsized":"2024-04-27T03:21:52.725275+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:52.725071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:52.725071+0000","last_clean_scrub_stamp":"2024-04-27T03:21:52.725071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:49:42.890358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.749914+0000","last_change":"2024-04-27T03:21:35.749914+0000","last_active":"2024-04-27T03:21:35.749914+0000","last_peered":"2024-04-27T03:21:35.749914+0000","last_clean":"2024-04-27T03:21:35.749914+0000","last_became_active":"2024-04-27T03:17:51.024449+0000","last_became_peered":"2024-04-27T03:17:51.024449+0000","last_unstale":"2024-04-27T03:21:35.749914+0000","last_undegraded":"2024-04-27T03:21:35.749914+0000","last_fullsized":"2024-04-27T03:21:35.749914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:35.749664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:35.749664+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.749664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:49:21.865491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"31'9","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.185455+0000","last_change":"2024-04-27T03:21:19.185455+0000","last_active":"2024-04-27T03:21:19.185455+0000","last_peered":"2024-04-27T03:21:19.185455+0000","last_clean":"2024-04-27T03:21:19.185455+0000","last_became_active":"2024-04-27T03:17:53.050748+0000","last_became_peered":"2024-04-27T03:17:53.050748+0000","last_unstale":"2024-04-27T03:21:19.185455+0000","last_undegraded":"2024-04-27T03:21:19.185455+0000","last_fullsized":"2024-04-27T03:21:19.185455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'9","last_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_deep_scrub":"31'9","last_deep_scrub_stamp":"2024-04-27T03:21:19.185100+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.185100+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:22:12.343058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":250,"num_omap_keys":2,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.823643+0000","last_change":"2024-04-27T03:21:24.823694+0000","last_active":"2024-04-27T03:21:24.823643+0000","last_peered":"2024-04-27T03:21:24.823643+0000","last_clean":"2024-04-27T03:21:24.823643+0000","last_became_active":"2024-04-27T03:17:55.050538+0000","last_became_peered":"2024-04-27T03:17:55.050538+0000","last_unstale":"2024-04-27T03:21:24.823643+0000","last_undegraded":"2024-04-27T03:21:24.823643+0000","last_fullsized":"2024-04-27T03:21:24.823643+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.823449+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.823449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:10:09.300634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1d","version":"27'2","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:15.350279+0000","last_change":"2024-04-27T03:22:15.350338+0000","last_active":"2024-04-27T03:22:15.350279+0000","last_peered":"2024-04-27T03:22:15.350279+0000","last_clean":"2024-04-27T03:22:15.350279+0000","last_became_active":"2024-04-27T03:17:57.088493+0000","last_became_peered":"2024-04-27T03:17:57.088493+0000","last_unstale":"2024-04-27T03:22:15.350279+0000","last_undegraded":"2024-04-27T03:22:15.350279+0000","last_fullsized":"2024-04-27T03:22:15.350279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:22:15.350075+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:22:15.350075+0000","last_clean_scrub_stamp":"2024-04-27T03:22:15.350075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:26:28.489301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"31'10","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.778594+0000","last_change":"2024-04-27T03:22:02.778656+0000","last_active":"2024-04-27T03:22:02.778594+0000","last_peered":"2024-04-27T03:22:02.778594+0000","last_clean":"2024-04-27T03:22:02.778594+0000","last_became_active":"2024-04-27T03:18:34.773038+0000","last_became_peered":"2024-04-27T03:18:34.773038+0000","last_unstale":"2024-04-27T03:22:02.778594+0000","last_undegraded":"2024-04-27T03:22:02.778594+0000","last_fullsized":"2024-04-27T03:22:02.778594+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:22:02.778374+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:22:02.778374+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.778374+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:49.192053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":44,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:08.355889+0000","last_change":"2024-04-27T03:22:08.355955+0000","last_active":"2024-04-27T03:22:08.355889+0000","last_peered":"2024-04-27T03:22:08.355889+0000","last_clean":"2024-04-27T03:22:08.355889+0000","last_became_active":"2024-04-27T03:17:47.011268+0000","last_became_peered":"2024-04-27T03:17:47.011268+0000","last_unstale":"2024-04-27T03:22:08.355889+0000","last_undegraded":"2024-04-27T03:22:08.355889+0000","last_fullsized":"2024-04-27T03:22:08.355889+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:08.355667+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:08.355667+0000","last_clean_scrub_stamp":"2024-04-27T03:22:08.355667+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:26:21.494893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:12.096234+0000","last_change":"2024-04-27T03:21:12.096234+0000","last_active":"2024-04-27T03:21:12.096234+0000","last_peered":"2024-04-27T03:21:12.096234+0000","last_clean":"2024-04-27T03:21:12.096234+0000","last_became_active":"2024-04-27T03:17:51.036850+0000","last_became_peered":"2024-04-27T03:17:51.036850+0000","last_unstale":"2024-04-27T03:21:12.096234+0000","last_undegraded":"2024-04-27T03:21:12.096234+0000","last_fullsized":"2024-04-27T03:21:12.096234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:12.096079+0000","last_clean_scrub_stamp":"2024-04-27T03:21:12.096079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:17:20.095840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"31'47","reported_seq":124,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:30.667010+0000","last_change":"2024-04-27T03:21:30.667010+0000","last_active":"2024-04-27T03:21:30.667010+0000","last_peered":"2024-04-27T03:21:30.667010+0000","last_clean":"2024-04-27T03:21:30.667010+0000","last_became_active":"2024-04-27T03:17:53.046209+0000","last_became_peered":"2024-04-27T03:17:53.046209+0000","last_unstale":"2024-04-27T03:21:30.667010+0000","last_undegraded":"2024-04-27T03:21:30.667010+0000","last_fullsized":"2024-04-27T03:21:30.667010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-04-27T03:21:30.666704+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-04-27T03:21:30.666704+0000","last_clean_scrub_stamp":"2024-04-27T03:21:30.666704+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:05.358118+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":67,"num_read_kb":70,"num_write":48,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":662,"num_omap_keys":2,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.199623+0000","last_change":"2024-04-27T03:21:20.199682+0000","last_active":"2024-04-27T03:21:20.199623+0000","last_peered":"2024-04-27T03:21:20.199623+0000","last_clean":"2024-04-27T03:21:20.199623+0000","last_became_active":"2024-04-27T03:17:55.070890+0000","last_became_peered":"2024-04-27T03:17:55.070890+0000","last_unstale":"2024-04-27T03:21:20.199623+0000","last_undegraded":"2024-04-27T03:21:20.199623+0000","last_fullsized":"2024-04-27T03:21:20.199623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.199413+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.199413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:10:57.834556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:21.453104+0000","last_change":"2024-04-27T03:22:21.453162+0000","last_active":"2024-04-27T03:22:21.453104+0000","last_peered":"2024-04-27T03:22:21.453104+0000","last_clean":"2024-04-27T03:22:21.453104+0000","last_became_active":"2024-04-27T03:17:57.084399+0000","last_became_peered":"2024-04-27T03:17:57.084399+0000","last_unstale":"2024-04-27T03:22:21.453104+0000","last_undegraded":"2024-04-27T03:22:21.453104+0000","last_fullsized":"2024-04-27T03:22:21.453104+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:21.452901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:21.452901+0000","last_clean_scrub_stamp":"2024-04-27T03:22:21.452901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:27:58.668114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"31'6","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.664699+0000","last_change":"2024-04-27T03:21:53.664699+0000","last_active":"2024-04-27T03:21:53.664699+0000","last_peered":"2024-04-27T03:21:53.664699+0000","last_clean":"2024-04-27T03:21:53.664699+0000","last_became_active":"2024-04-27T03:17:49.014464+0000","last_became_peered":"2024-04-27T03:17:49.014464+0000","last_unstale":"2024-04-27T03:21:53.664699+0000","last_undegraded":"2024-04-27T03:21:53.664699+0000","last_fullsized":"2024-04-27T03:21:53.664699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:53.664441+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:53.664441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.664441+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:08:23.207315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.477017+0000","last_change":"2024-04-27T03:21:55.477076+0000","last_active":"2024-04-27T03:21:55.477017+0000","last_peered":"2024-04-27T03:21:55.477017+0000","last_clean":"2024-04-27T03:21:55.477017+0000","last_became_active":"2024-04-27T03:17:47.023348+0000","last_became_peered":"2024-04-27T03:17:47.023348+0000","last_unstale":"2024-04-27T03:21:55.477017+0000","last_undegraded":"2024-04-27T03:21:55.477017+0000","last_fullsized":"2024-04-27T03:21:55.477017+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:55.476808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:55.476808+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.476808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:33.170168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:37.813424+0000","last_change":"2024-04-27T03:21:37.813424+0000","last_active":"2024-04-27T03:21:37.813424+0000","last_peered":"2024-04-27T03:21:37.813424+0000","last_clean":"2024-04-27T03:21:37.813424+0000","last_became_active":"2024-04-27T03:17:51.013349+0000","last_became_peered":"2024-04-27T03:17:51.013349+0000","last_unstale":"2024-04-27T03:21:37.813424+0000","last_undegraded":"2024-04-27T03:21:37.813424+0000","last_fullsized":"2024-04-27T03:21:37.813424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:37.813166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:37.813166+0000","last_clean_scrub_stamp":"2024-04-27T03:21:37.813166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:26.811307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"31'37","reported_seq":97,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:17.159218+0000","last_change":"2024-04-27T03:21:17.159218+0000","last_active":"2024-04-27T03:21:17.159218+0000","last_peered":"2024-04-27T03:21:17.159218+0000","last_clean":"2024-04-27T03:21:17.159218+0000","last_became_active":"2024-04-27T03:17:53.037775+0000","last_became_peered":"2024-04-27T03:17:53.037775+0000","last_unstale":"2024-04-27T03:21:17.159218+0000","last_undegraded":"2024-04-27T03:21:17.159218+0000","last_fullsized":"2024-04-27T03:21:17.159218+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:17.158898+0000","last_clean_scrub_stamp":"2024-04-27T03:21:17.158898+0000","objects_scrubbed":4,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:06.103710+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.871801+0000","last_change":"2024-04-27T03:21:23.871852+0000","last_active":"2024-04-27T03:21:23.871801+0000","last_peered":"2024-04-27T03:21:23.871801+0000","last_clean":"2024-04-27T03:21:23.871801+0000","last_became_active":"2024-04-27T03:17:55.054322+0000","last_became_peered":"2024-04-27T03:17:55.054322+0000","last_unstale":"2024-04-27T03:21:23.871801+0000","last_undegraded":"2024-04-27T03:21:23.871801+0000","last_fullsized":"2024-04-27T03:21:23.871801+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:23.871608+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.871608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:09:14.918299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"31'2","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.137118+0000","last_change":"2024-04-27T03:21:39.137118+0000","last_active":"2024-04-27T03:21:39.137118+0000","last_peered":"2024-04-27T03:21:39.137118+0000","last_clean":"2024-04-27T03:21:39.137118+0000","last_became_active":"2024-04-27T03:17:49.021306+0000","last_became_peered":"2024-04-27T03:17:49.021306+0000","last_unstale":"2024-04-27T03:21:39.137118+0000","last_undegraded":"2024-04-27T03:21:39.137118+0000","last_fullsized":"2024-04-27T03:21:39.137118+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:39.136858+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:39.136858+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.136858+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:15.417970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"31'6","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:09.580393+0000","last_change":"2024-04-27T03:22:09.580463+0000","last_active":"2024-04-27T03:22:09.580393+0000","last_peered":"2024-04-27T03:22:09.580393+0000","last_clean":"2024-04-27T03:22:09.580393+0000","last_became_active":"2024-04-27T03:17:47.031736+0000","last_became_peered":"2024-04-27T03:17:47.031736+0000","last_unstale":"2024-04-27T03:22:09.580393+0000","last_undegraded":"2024-04-27T03:22:09.580393+0000","last_fullsized":"2024-04-27T03:22:09.580393+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:09.580198+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:09.580198+0000","last_clean_scrub_stamp":"2024-04-27T03:22:09.580198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:16:08.893299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":22,"num_write_kb":4098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"31'5","reported_seq":243,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:33.496951+0000","last_change":"2024-04-27T03:21:33.496951+0000","last_active":"2024-04-27T03:21:33.496951+0000","last_peered":"2024-04-27T03:21:33.496951+0000","last_clean":"2024-04-27T03:21:33.496951+0000","last_became_active":"2024-04-27T03:17:51.023307+0000","last_became_peered":"2024-04-27T03:17:51.023307+0000","last_unstale":"2024-04-27T03:21:33.496951+0000","last_undegraded":"2024-04-27T03:21:33.496951+0000","last_fullsized":"2024-04-27T03:21:33.496951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:21:33.496514+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:21:33.496514+0000","last_clean_scrub_stamp":"2024-04-27T03:21:33.496514+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:17:35.575680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":301,"num_read_kb":301,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1d","version":"31'72","reported_seq":155,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:23.270317+0000","last_change":"2024-04-27T03:21:23.270391+0000","last_active":"2024-04-27T03:21:23.270317+0000","last_peered":"2024-04-27T03:21:23.270317+0000","last_clean":"2024-04-27T03:21:23.270317+0000","last_became_active":"2024-04-27T03:17:53.035177+0000","last_became_peered":"2024-04-27T03:17:53.035177+0000","last_unstale":"2024-04-27T03:21:23.270317+0000","last_undegraded":"2024-04-27T03:21:23.270317+0000","last_fullsized":"2024-04-27T03:21:23.270317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'72","last_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_deep_scrub":"31'72","last_deep_scrub_stamp":"2024-04-27T03:21:23.270178+0000","last_clean_scrub_stamp":"2024-04-27T03:21:23.270178+0000","objects_scrubbed":6,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:23:28.956226+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":28,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.743462+0000","last_change":"2024-04-27T03:21:16.743518+0000","last_active":"2024-04-27T03:21:16.743462+0000","last_peered":"2024-04-27T03:21:16.743462+0000","last_clean":"2024-04-27T03:21:16.743462+0000","last_became_active":"2024-04-27T03:17:55.075257+0000","last_became_peered":"2024-04-27T03:17:55.075257+0000","last_unstale":"2024-04-27T03:21:16.743462+0000","last_undegraded":"2024-04-27T03:21:16.743462+0000","last_fullsized":"2024-04-27T03:21:16.743462+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:16.743267+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.743267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:34:18.284825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"31'16","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.724167+0000","last_change":"2024-04-27T03:21:48.724227+0000","last_active":"2024-04-27T03:21:48.724167+0000","last_peered":"2024-04-27T03:21:48.724167+0000","last_clean":"2024-04-27T03:21:48.724167+0000","last_became_active":"2024-04-27T03:17:57.068990+0000","last_became_peered":"2024-04-27T03:17:57.068990+0000","last_unstale":"2024-04-27T03:21:48.724167+0000","last_undegraded":"2024-04-27T03:21:48.724167+0000","last_fullsized":"2024-04-27T03:21:48.724167+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:48.723971+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:48.723971+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.723971+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:58.946837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":21,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"31'6","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.540808+0000","last_change":"2024-04-27T03:21:46.540808+0000","last_active":"2024-04-27T03:21:46.540808+0000","last_peered":"2024-04-27T03:21:46.540808+0000","last_clean":"2024-04-27T03:21:46.540808+0000","last_became_active":"2024-04-27T03:17:48.999909+0000","last_became_peered":"2024-04-27T03:17:48.999909+0000","last_unstale":"2024-04-27T03:21:46.540808+0000","last_undegraded":"2024-04-27T03:21:46.540808+0000","last_fullsized":"2024-04-27T03:21:46.540808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:46.540543+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:46.540543+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.540543+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:37:16.927826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.429062+0000","last_change":"2024-04-27T03:22:01.429118+0000","last_active":"2024-04-27T03:22:01.429062+0000","last_peered":"2024-04-27T03:22:01.429062+0000","last_clean":"2024-04-27T03:22:01.429062+0000","last_became_active":"2024-04-27T03:17:47.017900+0000","last_became_peered":"2024-04-27T03:17:47.017900+0000","last_unstale":"2024-04-27T03:22:01.429062+0000","last_undegraded":"2024-04-27T03:22:01.429062+0000","last_fullsized":"2024-04-27T03:22:01.429062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:01.428879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:01.428879+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.428879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:02:51.699431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":43,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:20.193806+0000","last_change":"2024-04-27T03:21:20.193806+0000","last_active":"2024-04-27T03:21:20.193806+0000","last_peered":"2024-04-27T03:21:20.193806+0000","last_clean":"2024-04-27T03:21:20.193806+0000","last_became_active":"2024-04-27T03:17:51.032413+0000","last_became_peered":"2024-04-27T03:17:51.032413+0000","last_unstale":"2024-04-27T03:21:20.193806+0000","last_undegraded":"2024-04-27T03:21:20.193806+0000","last_fullsized":"2024-04-27T03:21:20.193806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:20.193546+0000","last_clean_scrub_stamp":"2024-04-27T03:21:20.193546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:27:42.782149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"31'45","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.064594+0000","last_change":"2024-04-27T03:21:13.064640+0000","last_active":"2024-04-27T03:21:13.064594+0000","last_peered":"2024-04-27T03:21:13.064594+0000","last_clean":"2024-04-27T03:21:13.064594+0000","last_became_active":"2024-04-27T03:17:53.037361+0000","last_became_peered":"2024-04-27T03:17:53.037361+0000","last_unstale":"2024-04-27T03:21:13.064594+0000","last_undegraded":"2024-04-27T03:21:13.064594+0000","last_fullsized":"2024-04-27T03:21:13.064594+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:13.064511+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.064511+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:06:47.923690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.302195+0000","last_change":"2024-04-27T03:21:14.302256+0000","last_active":"2024-04-27T03:21:14.302195+0000","last_peered":"2024-04-27T03:21:14.302195+0000","last_clean":"2024-04-27T03:21:14.302195+0000","last_became_active":"2024-04-27T03:17:55.069023+0000","last_became_peered":"2024-04-27T03:17:55.069023+0000","last_unstale":"2024-04-27T03:21:14.302195+0000","last_undegraded":"2024-04-27T03:21:14.302195+0000","last_fullsized":"2024-04-27T03:21:14.302195+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:14.302007+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.302007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:00.292019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:23.296135+0000","last_change":"2024-04-27T03:22:23.296187+0000","last_active":"2024-04-27T03:22:23.296135+0000","last_peered":"2024-04-27T03:22:23.296135+0000","last_clean":"2024-04-27T03:22:23.296135+0000","last_became_active":"2024-04-27T03:17:57.069416+0000","last_became_peered":"2024-04-27T03:17:57.069416+0000","last_unstale":"2024-04-27T03:22:23.296135+0000","last_undegraded":"2024-04-27T03:22:23.296135+0000","last_fullsized":"2024-04-27T03:22:23.296135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:23.295943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:23.295943+0000","last_clean_scrub_stamp":"2024-04-27T03:22:23.295943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:26:08.156218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:02.744246+0000","last_change":"2024-04-27T03:22:02.744298+0000","last_active":"2024-04-27T03:22:02.744246+0000","last_peered":"2024-04-27T03:22:02.744246+0000","last_clean":"2024-04-27T03:22:02.744246+0000","last_became_active":"2024-04-27T03:17:49.004091+0000","last_became_peered":"2024-04-27T03:17:49.004091+0000","last_unstale":"2024-04-27T03:22:02.744246+0000","last_undegraded":"2024-04-27T03:22:02.744246+0000","last_fullsized":"2024-04-27T03:22:02.744246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:02.744043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:02.744043+0000","last_clean_scrub_stamp":"2024-04-27T03:22:02.744043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:47:54.622128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:07.343260+0000","last_change":"2024-04-27T03:22:07.343331+0000","last_active":"2024-04-27T03:22:07.343260+0000","last_peered":"2024-04-27T03:22:07.343260+0000","last_clean":"2024-04-27T03:22:07.343260+0000","last_became_active":"2024-04-27T03:17:47.024121+0000","last_became_peered":"2024-04-27T03:17:47.024121+0000","last_unstale":"2024-04-27T03:22:07.343260+0000","last_undegraded":"2024-04-27T03:22:07.343260+0000","last_fullsized":"2024-04-27T03:22:07.343260+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:07.343061+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:07.343061+0000","last_clean_scrub_stamp":"2024-04-27T03:22:07.343061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:02:54.749820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:22.231515+0000","last_change":"2024-04-27T03:21:22.231634+0000","last_active":"2024-04-27T03:21:22.231515+0000","last_peered":"2024-04-27T03:21:22.231515+0000","last_clean":"2024-04-27T03:21:22.231515+0000","last_became_active":"2024-04-27T03:17:51.020278+0000","last_became_peered":"2024-04-27T03:17:51.020278+0000","last_unstale":"2024-04-27T03:21:22.231515+0000","last_undegraded":"2024-04-27T03:21:22.231515+0000","last_fullsized":"2024-04-27T03:21:22.231515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:22.231282+0000","last_clean_scrub_stamp":"2024-04-27T03:21:22.231282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:38.450738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"31'47","reported_seq":114,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:16.572535+0000","last_change":"2024-04-27T03:21:16.572535+0000","last_active":"2024-04-27T03:21:16.572535+0000","last_peered":"2024-04-27T03:21:16.572535+0000","last_clean":"2024-04-27T03:21:16.572535+0000","last_became_active":"2024-04-27T03:17:53.040531+0000","last_became_peered":"2024-04-27T03:17:53.040531+0000","last_unstale":"2024-04-27T03:21:16.572535+0000","last_undegraded":"2024-04-27T03:21:16.572535+0000","last_fullsized":"2024-04-27T03:21:16.572535+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'47","last_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_deep_scrub":"31'47","last_deep_scrub_stamp":"2024-04-27T03:21:16.572262+0000","last_clean_scrub_stamp":"2024-04-27T03:21:16.572262+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:08:05.517074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":39,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:28.304119+0000","last_change":"2024-04-27T03:21:28.304119+0000","last_active":"2024-04-27T03:21:28.304119+0000","last_peered":"2024-04-27T03:21:28.304119+0000","last_clean":"2024-04-27T03:21:28.304119+0000","last_became_active":"2024-04-27T03:17:55.076650+0000","last_became_peered":"2024-04-27T03:17:55.076650+0000","last_unstale":"2024-04-27T03:21:28.304119+0000","last_undegraded":"2024-04-27T03:21:28.304119+0000","last_fullsized":"2024-04-27T03:21:28.304119+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:28.303861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:28.303861+0000","last_clean_scrub_stamp":"2024-04-27T03:21:28.303861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:26.889521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.11","version":"31'5","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:21.327608+0000","last_change":"2024-04-27T03:22:21.327669+0000","last_active":"2024-04-27T03:22:21.327608+0000","last_peered":"2024-04-27T03:22:21.327608+0000","last_clean":"2024-04-27T03:22:21.327608+0000","last_became_active":"2024-04-27T03:17:57.065529+0000","last_became_peered":"2024-04-27T03:17:57.065529+0000","last_unstale":"2024-04-27T03:22:21.327608+0000","last_undegraded":"2024-04-27T03:22:21.327608+0000","last_fullsized":"2024-04-27T03:22:21.327608+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'5","last_scrub_stamp":"2024-04-27T03:22:21.327398+0000","last_deep_scrub":"31'5","last_deep_scrub_stamp":"2024-04-27T03:22:21.327398+0000","last_clean_scrub_stamp":"2024-04-27T03:22:21.327398+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:46.335834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"31'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.698657+0000","last_change":"2024-04-27T03:21:35.698721+0000","last_active":"2024-04-27T03:21:35.698657+0000","last_peered":"2024-04-27T03:21:35.698657+0000","last_clean":"2024-04-27T03:21:35.698657+0000","last_became_active":"2024-04-27T03:17:49.019620+0000","last_became_peered":"2024-04-27T03:17:49.019620+0000","last_unstale":"2024-04-27T03:21:35.698657+0000","last_undegraded":"2024-04-27T03:21:35.698657+0000","last_fullsized":"2024-04-27T03:21:35.698657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:35.698441+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:35.698441+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.698441+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:43.033392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"27'4","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:13.529986+0000","last_change":"2024-04-27T03:22:13.530114+0000","last_active":"2024-04-27T03:22:13.529986+0000","last_peered":"2024-04-27T03:22:13.529986+0000","last_clean":"2024-04-27T03:22:13.529986+0000","last_became_active":"2024-04-27T03:17:47.017062+0000","last_became_peered":"2024-04-27T03:17:47.017062+0000","last_unstale":"2024-04-27T03:22:13.529986+0000","last_undegraded":"2024-04-27T03:22:13.529986+0000","last_fullsized":"2024-04-27T03:22:13.529986+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'4","last_scrub_stamp":"2024-04-27T03:22:13.529690+0000","last_deep_scrub":"27'4","last_deep_scrub_stamp":"2024-04-27T03:22:13.529690+0000","last_clean_scrub_stamp":"2024-04-27T03:22:13.529690+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:54:45.058089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.301687+0000","last_change":"2024-04-27T03:21:27.301740+0000","last_active":"2024-04-27T03:21:27.301687+0000","last_peered":"2024-04-27T03:21:27.301687+0000","last_clean":"2024-04-27T03:21:27.301687+0000","last_became_active":"2024-04-27T03:17:51.028997+0000","last_became_peered":"2024-04-27T03:17:51.028997+0000","last_unstale":"2024-04-27T03:21:27.301687+0000","last_undegraded":"2024-04-27T03:21:27.301687+0000","last_fullsized":"2024-04-27T03:21:27.301687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:27.301481+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.301481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:33.928268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"31'22","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.677168+0000","last_change":"2024-04-27T03:21:31.677168+0000","last_active":"2024-04-27T03:21:31.677168+0000","last_peered":"2024-04-27T03:21:31.677168+0000","last_clean":"2024-04-27T03:21:31.677168+0000","last_became_active":"2024-04-27T03:17:53.041583+0000","last_became_peered":"2024-04-27T03:17:53.041583+0000","last_unstale":"2024-04-27T03:21:31.677168+0000","last_undegraded":"2024-04-27T03:21:31.677168+0000","last_fullsized":"2024-04-27T03:21:31.677168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'22","last_scrub_stamp":"2024-04-27T03:21:31.676892+0000","last_deep_scrub":"31'22","last_deep_scrub_stamp":"2024-04-27T03:21:31.676892+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.676892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:50:19.196626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"31'285","reported_seq":393,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:25.185168+0000","last_change":"2024-04-27T03:21:25.185168+0000","last_active":"2024-04-27T03:21:25.185168+0000","last_peered":"2024-04-27T03:21:25.185168+0000","last_clean":"2024-04-27T03:21:25.185168+0000","last_became_active":"2024-04-27T03:17:55.057930+0000","last_became_peered":"2024-04-27T03:17:55.057930+0000","last_unstale":"2024-04-27T03:21:25.185168+0000","last_undegraded":"2024-04-27T03:21:25.185168+0000","last_fullsized":"2024-04-27T03:21:25.185168+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'285","last_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_deep_scrub":"31'285","last_deep_scrub_stamp":"2024-04-27T03:21:25.184906+0000","last_clean_scrub_stamp":"2024-04-27T03:21:25.184906+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:35.796002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:24.293515+0000","last_change":"2024-04-27T03:22:24.293621+0000","last_active":"2024-04-27T03:22:24.293515+0000","last_peered":"2024-04-27T03:22:24.293515+0000","last_clean":"2024-04-27T03:22:24.293515+0000","last_became_active":"2024-04-27T03:17:57.084298+0000","last_became_peered":"2024-04-27T03:17:57.084298+0000","last_unstale":"2024-04-27T03:22:24.293515+0000","last_undegraded":"2024-04-27T03:22:24.293515+0000","last_fullsized":"2024-04-27T03:22:24.293515+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:24.293282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:24.293282+0000","last_clean_scrub_stamp":"2024-04-27T03:22:24.293282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:01.508495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"31'4","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.686144+0000","last_change":"2024-04-27T03:21:32.686210+0000","last_active":"2024-04-27T03:21:32.686144+0000","last_peered":"2024-04-27T03:21:32.686144+0000","last_clean":"2024-04-27T03:21:32.686144+0000","last_became_active":"2024-04-27T03:17:49.013478+0000","last_became_peered":"2024-04-27T03:17:49.013478+0000","last_unstale":"2024-04-27T03:21:32.686144+0000","last_undegraded":"2024-04-27T03:21:32.686144+0000","last_fullsized":"2024-04-27T03:21:32.686144+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:32.685934+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:32.685934+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.685934+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:17:18.675946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"27'4","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.498152+0000","last_change":"2024-04-27T03:21:51.498216+0000","last_active":"2024-04-27T03:21:51.498152+0000","last_peered":"2024-04-27T03:21:51.498152+0000","last_clean":"2024-04-27T03:21:51.498152+0000","last_became_active":"2024-04-27T03:17:47.027261+0000","last_became_peered":"2024-04-27T03:17:47.027261+0000","last_unstale":"2024-04-27T03:21:51.498152+0000","last_undegraded":"2024-04-27T03:21:51.498152+0000","last_fullsized":"2024-04-27T03:21:51.498152+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'4","last_scrub_stamp":"2024-04-27T03:21:51.497953+0000","last_deep_scrub":"27'4","last_deep_scrub_stamp":"2024-04-27T03:21:51.497953+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.497953+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:38.452327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":10,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:13.332423+0000","last_change":"2024-04-27T03:21:13.332476+0000","last_active":"2024-04-27T03:21:13.332423+0000","last_peered":"2024-04-27T03:21:13.332423+0000","last_clean":"2024-04-27T03:21:13.332423+0000","last_became_active":"2024-04-27T03:17:51.031496+0000","last_became_peered":"2024-04-27T03:17:51.031496+0000","last_unstale":"2024-04-27T03:21:13.332423+0000","last_undegraded":"2024-04-27T03:21:13.332423+0000","last_fullsized":"2024-04-27T03:21:13.332423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:13.332213+0000","last_clean_scrub_stamp":"2024-04-27T03:21:13.332213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:31.014208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"31'73","reported_seq":159,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:32.145202+0000","last_change":"2024-04-27T03:21:32.145296+0000","last_active":"2024-04-27T03:21:32.145202+0000","last_peered":"2024-04-27T03:21:32.145202+0000","last_clean":"2024-04-27T03:21:32.145202+0000","last_became_active":"2024-04-27T03:17:53.039236+0000","last_became_peered":"2024-04-27T03:17:53.039236+0000","last_unstale":"2024-04-27T03:21:32.145202+0000","last_undegraded":"2024-04-27T03:21:32.145202+0000","last_fullsized":"2024-04-27T03:21:32.145202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'73","last_scrub_stamp":"2024-04-27T03:21:32.145018+0000","last_deep_scrub":"31'73","last_deep_scrub_stamp":"2024-04-27T03:21:32.145018+0000","last_clean_scrub_stamp":"2024-04-27T03:21:32.145018+0000","objects_scrubbed":11,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:18:06.836432+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":80,"num_read_kb":67,"num_write":44,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:26.654794+0000","last_change":"2024-04-27T03:21:26.654845+0000","last_active":"2024-04-27T03:21:26.654794+0000","last_peered":"2024-04-27T03:21:26.654794+0000","last_clean":"2024-04-27T03:21:26.654794+0000","last_became_active":"2024-04-27T03:17:55.070733+0000","last_became_peered":"2024-04-27T03:17:55.070733+0000","last_unstale":"2024-04-27T03:21:26.654794+0000","last_undegraded":"2024-04-27T03:21:26.654794+0000","last_fullsized":"2024-04-27T03:21:26.654794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:26.654609+0000","last_clean_scrub_stamp":"2024-04-27T03:21:26.654609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:01.305349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"31'6","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:22.281283+0000","last_change":"2024-04-27T03:22:22.281371+0000","last_active":"2024-04-27T03:22:22.281283+0000","last_peered":"2024-04-27T03:22:22.281283+0000","last_clean":"2024-04-27T03:22:22.281283+0000","last_became_active":"2024-04-27T03:17:57.065744+0000","last_became_peered":"2024-04-27T03:17:57.065744+0000","last_unstale":"2024-04-27T03:22:22.281283+0000","last_undegraded":"2024-04-27T03:22:22.281283+0000","last_fullsized":"2024-04-27T03:22:22.281283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:22:22.281097+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:22:22.281097+0000","last_clean_scrub_stamp":"2024-04-27T03:22:22.281097+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:20:29.619106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"31'2","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:44.195754+0000","last_change":"2024-04-27T03:21:44.195813+0000","last_active":"2024-04-27T03:21:44.195754+0000","last_peered":"2024-04-27T03:21:44.195754+0000","last_clean":"2024-04-27T03:21:44.195754+0000","last_became_active":"2024-04-27T03:17:49.016839+0000","last_became_peered":"2024-04-27T03:17:49.016839+0000","last_unstale":"2024-04-27T03:21:44.195754+0000","last_undegraded":"2024-04-27T03:21:44.195754+0000","last_fullsized":"2024-04-27T03:21:44.195754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:44.195561+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:44.195561+0000","last_clean_scrub_stamp":"2024-04-27T03:21:44.195561+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:26:31.149936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:58.232928+0000","last_change":"2024-04-27T03:21:58.232983+0000","last_active":"2024-04-27T03:21:58.232928+0000","last_peered":"2024-04-27T03:21:58.232928+0000","last_clean":"2024-04-27T03:21:58.232928+0000","last_became_active":"2024-04-27T03:17:47.021860+0000","last_became_peered":"2024-04-27T03:17:47.021860+0000","last_unstale":"2024-04-27T03:21:58.232928+0000","last_undegraded":"2024-04-27T03:21:58.232928+0000","last_fullsized":"2024-04-27T03:21:58.232928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:58.232714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:58.232714+0000","last_clean_scrub_stamp":"2024-04-27T03:21:58.232714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:14:05.190857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.156966+0000","last_change":"2024-04-27T03:21:19.157032+0000","last_active":"2024-04-27T03:21:19.156966+0000","last_peered":"2024-04-27T03:21:19.156966+0000","last_clean":"2024-04-27T03:21:19.156966+0000","last_became_active":"2024-04-27T03:17:51.032670+0000","last_became_peered":"2024-04-27T03:17:51.032670+0000","last_unstale":"2024-04-27T03:21:19.156966+0000","last_undegraded":"2024-04-27T03:21:19.156966+0000","last_fullsized":"2024-04-27T03:21:19.156966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.156758+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.156758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:49.521232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"31'45","reported_seq":110,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:27.820346+0000","last_change":"2024-04-27T03:21:27.820426+0000","last_active":"2024-04-27T03:21:27.820346+0000","last_peered":"2024-04-27T03:21:27.820346+0000","last_clean":"2024-04-27T03:21:27.820346+0000","last_became_active":"2024-04-27T03:17:53.039366+0000","last_became_peered":"2024-04-27T03:17:53.039366+0000","last_unstale":"2024-04-27T03:21:27.820346+0000","last_undegraded":"2024-04-27T03:21:27.820346+0000","last_fullsized":"2024-04-27T03:21:27.820346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'45","last_scrub_stamp":"2024-04-27T03:21:27.820155+0000","last_deep_scrub":"31'45","last_deep_scrub_stamp":"2024-04-27T03:21:27.820155+0000","last_clean_scrub_stamp":"2024-04-27T03:21:27.820155+0000","objects_scrubbed":5,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:39:02.307136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:24.175376+0000","last_change":"2024-04-27T03:21:24.175430+0000","last_active":"2024-04-27T03:21:24.175376+0000","last_peered":"2024-04-27T03:21:24.175376+0000","last_clean":"2024-04-27T03:21:24.175376+0000","last_became_active":"2024-04-27T03:17:55.057784+0000","last_became_peered":"2024-04-27T03:17:55.057784+0000","last_unstale":"2024-04-27T03:21:24.175376+0000","last_undegraded":"2024-04-27T03:21:24.175376+0000","last_fullsized":"2024-04-27T03:21:24.175376+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:24.175164+0000","last_clean_scrub_stamp":"2024-04-27T03:21:24.175164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:23.877311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"31'3","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:23.368859+0000","last_change":"2024-04-27T03:22:23.368929+0000","last_active":"2024-04-27T03:22:23.368859+0000","last_peered":"2024-04-27T03:22:23.368859+0000","last_clean":"2024-04-27T03:22:23.368859+0000","last_became_active":"2024-04-27T03:17:57.086567+0000","last_became_peered":"2024-04-27T03:17:57.086567+0000","last_unstale":"2024-04-27T03:22:23.368859+0000","last_undegraded":"2024-04-27T03:22:23.368859+0000","last_fullsized":"2024-04-27T03:22:23.368859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-04-27T03:22:23.368651+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-04-27T03:22:23.368651+0000","last_clean_scrub_stamp":"2024-04-27T03:22:23.368651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:28:32.851611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"31'10","reported_seq":68,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:18.472576+0000","last_change":"2024-04-27T03:22:18.472693+0000","last_active":"2024-04-27T03:22:18.472576+0000","last_peered":"2024-04-27T03:22:18.472576+0000","last_clean":"2024-04-27T03:22:18.472576+0000","last_became_active":"2024-04-27T03:17:49.011589+0000","last_became_peered":"2024-04-27T03:17:49.011589+0000","last_unstale":"2024-04-27T03:22:18.472576+0000","last_undegraded":"2024-04-27T03:22:18.472576+0000","last_fullsized":"2024-04-27T03:22:18.472576+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'10","last_scrub_stamp":"2024-04-27T03:22:18.472267+0000","last_deep_scrub":"31'10","last_deep_scrub_stamp":"2024-04-27T03:22:18.472267+0000","last_clean_scrub_stamp":"2024-04-27T03:22:18.472267+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:28:43.480703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"31'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:12.573758+0000","last_change":"2024-04-27T03:22:12.573823+0000","last_active":"2024-04-27T03:22:12.573758+0000","last_peered":"2024-04-27T03:22:12.573758+0000","last_clean":"2024-04-27T03:22:12.573758+0000","last_became_active":"2024-04-27T03:17:47.016020+0000","last_became_peered":"2024-04-27T03:17:47.016020+0000","last_unstale":"2024-04-27T03:22:12.573758+0000","last_undegraded":"2024-04-27T03:22:12.573758+0000","last_fullsized":"2024-04-27T03:22:12.573758+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:12.573531+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:12.573531+0000","last_clean_scrub_stamp":"2024-04-27T03:22:12.573531+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:10:56.304235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:29.336891+0000","last_change":"2024-04-27T03:21:29.336949+0000","last_active":"2024-04-27T03:21:29.336891+0000","last_peered":"2024-04-27T03:21:29.336891+0000","last_clean":"2024-04-27T03:21:29.336891+0000","last_became_active":"2024-04-27T03:17:51.029053+0000","last_became_peered":"2024-04-27T03:17:51.029053+0000","last_unstale":"2024-04-27T03:21:29.336891+0000","last_undegraded":"2024-04-27T03:21:29.336891+0000","last_fullsized":"2024-04-27T03:21:29.336891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:29.336698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:29.336698+0000","last_clean_scrub_stamp":"2024-04-27T03:21:29.336698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:15:11.724434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"31'37","reported_seq":100,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:14.538736+0000","last_change":"2024-04-27T03:21:14.538808+0000","last_active":"2024-04-27T03:21:14.538736+0000","last_peered":"2024-04-27T03:21:14.538736+0000","last_clean":"2024-04-27T03:21:14.538736+0000","last_became_active":"2024-04-27T03:17:53.040364+0000","last_became_peered":"2024-04-27T03:17:53.040364+0000","last_unstale":"2024-04-27T03:21:14.538736+0000","last_undegraded":"2024-04-27T03:21:14.538736+0000","last_fullsized":"2024-04-27T03:21:14.538736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'37","last_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_deep_scrub":"31'37","last_deep_scrub_stamp":"2024-04-27T03:21:14.538542+0000","last_clean_scrub_stamp":"2024-04-27T03:21:14.538542+0000","objects_scrubbed":3,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:32.436126+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":33,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.189216+0000","last_change":"2024-04-27T03:21:41.189268+0000","last_active":"2024-04-27T03:21:41.189216+0000","last_peered":"2024-04-27T03:21:41.189216+0000","last_clean":"2024-04-27T03:21:41.189216+0000","last_became_active":"2024-04-27T03:17:55.070210+0000","last_became_peered":"2024-04-27T03:17:55.070210+0000","last_unstale":"2024-04-27T03:21:41.189216+0000","last_undegraded":"2024-04-27T03:21:41.189216+0000","last_fullsized":"2024-04-27T03:21:41.189216+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:41.189019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:41.189019+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.189019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:28.342309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"31'4","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:24.374594+0000","last_change":"2024-04-27T03:22:24.374661+0000","last_active":"2024-04-27T03:22:24.374594+0000","last_peered":"2024-04-27T03:22:24.374594+0000","last_clean":"2024-04-27T03:22:24.374594+0000","last_became_active":"2024-04-27T03:17:57.086226+0000","last_became_peered":"2024-04-27T03:17:57.086226+0000","last_unstale":"2024-04-27T03:22:24.374594+0000","last_undegraded":"2024-04-27T03:22:24.374594+0000","last_fullsized":"2024-04-27T03:22:24.374594+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:24.374397+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:24.374397+0000","last_clean_scrub_stamp":"2024-04-27T03:22:24.374397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:22.145005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"31'16","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.184766+0000","last_change":"2024-04-27T03:21:51.184831+0000","last_active":"2024-04-27T03:21:51.184766+0000","last_peered":"2024-04-27T03:21:51.184766+0000","last_clean":"2024-04-27T03:21:51.184766+0000","last_became_active":"2024-04-27T03:17:49.013062+0000","last_became_peered":"2024-04-27T03:17:49.013062+0000","last_unstale":"2024-04-27T03:21:51.184766+0000","last_undegraded":"2024-04-27T03:21:51.184766+0000","last_fullsized":"2024-04-27T03:21:51.184766+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:51.184588+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:51.184588+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.184588+0000","objects_scrubbed":2,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:01.407454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:06.456721+0000","last_change":"2024-04-27T03:22:06.456783+0000","last_active":"2024-04-27T03:22:06.456721+0000","last_peered":"2024-04-27T03:22:06.456721+0000","last_clean":"2024-04-27T03:22:06.456721+0000","last_became_active":"2024-04-27T03:17:47.000974+0000","last_became_peered":"2024-04-27T03:17:47.000974+0000","last_unstale":"2024-04-27T03:22:06.456721+0000","last_undegraded":"2024-04-27T03:22:06.456721+0000","last_fullsized":"2024-04-27T03:22:06.456721+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:22:06.456530+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:22:06.456530+0000","last_clean_scrub_stamp":"2024-04-27T03:22:06.456530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:18:56.389072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"31'3","reported_seq":30,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:01.767122+0000","last_change":"2024-04-27T03:22:01.767186+0000","last_active":"2024-04-27T03:22:01.767122+0000","last_peered":"2024-04-27T03:22:01.767122+0000","last_clean":"2024-04-27T03:22:01.767122+0000","last_became_active":"2024-04-27T03:18:34.767667+0000","last_became_peered":"2024-04-27T03:18:34.767667+0000","last_unstale":"2024-04-27T03:22:01.767122+0000","last_undegraded":"2024-04-27T03:22:01.767122+0000","last_fullsized":"2024-04-27T03:22:01.767122+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'3","last_scrub_stamp":"2024-04-27T03:22:01.766938+0000","last_deep_scrub":"31'3","last_deep_scrub_stamp":"2024-04-27T03:22:01.766938+0000","last_clean_scrub_stamp":"2024-04-27T03:22:01.766938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:43.632161+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"31'53","reported_seq":126,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:34.140107+0000","last_change":"2024-04-27T03:21:34.140183+0000","last_active":"2024-04-27T03:21:34.140107+0000","last_peered":"2024-04-27T03:21:34.140107+0000","last_clean":"2024-04-27T03:21:34.140107+0000","last_became_active":"2024-04-27T03:17:53.038379+0000","last_became_peered":"2024-04-27T03:17:53.038379+0000","last_unstale":"2024-04-27T03:21:34.140107+0000","last_undegraded":"2024-04-27T03:21:34.140107+0000","last_fullsized":"2024-04-27T03:21:34.140107+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'53","last_scrub_stamp":"2024-04-27T03:21:34.139891+0000","last_deep_scrub":"31'53","last_deep_scrub_stamp":"2024-04-27T03:21:34.139891+0000","last_clean_scrub_stamp":"2024-04-27T03:21:34.139891+0000","objects_scrubbed":7,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:44.594383+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":58,"num_read_kb":52,"num_write":32,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":1138,"num_omap_keys":6,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:19.822080+0000","last_change":"2024-04-27T03:21:19.822219+0000","last_active":"2024-04-27T03:21:19.822080+0000","last_peered":"2024-04-27T03:21:19.822080+0000","last_clean":"2024-04-27T03:21:19.822080+0000","last_became_active":"2024-04-27T03:17:55.063187+0000","last_became_peered":"2024-04-27T03:17:55.063187+0000","last_unstale":"2024-04-27T03:21:19.822080+0000","last_undegraded":"2024-04-27T03:21:19.822080+0000","last_fullsized":"2024-04-27T03:21:19.822080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:19.821853+0000","last_clean_scrub_stamp":"2024-04-27T03:21:19.821853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:20:18.101674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.14","version":"31'4","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:20.310954+0000","last_change":"2024-04-27T03:22:20.311016+0000","last_active":"2024-04-27T03:22:20.310954+0000","last_peered":"2024-04-27T03:22:20.310954+0000","last_clean":"2024-04-27T03:22:20.310954+0000","last_became_active":"2024-04-27T03:17:57.059712+0000","last_became_peered":"2024-04-27T03:17:57.059712+0000","last_unstale":"2024-04-27T03:22:20.310954+0000","last_undegraded":"2024-04-27T03:22:20.310954+0000","last_fullsized":"2024-04-27T03:22:20.310954+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:22:20.310752+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:22:20.310752+0000","last_clean_scrub_stamp":"2024-04-27T03:22:20.310752+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:31:09.349722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.27","version":"29'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:48.475670+0000","last_change":"2024-04-27T03:21:48.475670+0000","last_active":"2024-04-27T03:21:48.475670+0000","last_peered":"2024-04-27T03:21:48.475670+0000","last_clean":"2024-04-27T03:21:48.475670+0000","last_became_active":"2024-04-27T03:17:49.000131+0000","last_became_peered":"2024-04-27T03:17:49.000131+0000","last_unstale":"2024-04-27T03:21:48.475670+0000","last_undegraded":"2024-04-27T03:21:48.475670+0000","last_fullsized":"2024-04-27T03:21:48.475670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-04-27T03:21:48.475413+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-04-27T03:21:48.475413+0000","last_clean_scrub_stamp":"2024-04-27T03:21:48.475413+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:40:05.549233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.20","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.748753+0000","last_change":"2024-04-27T03:21:46.748753+0000","last_active":"2024-04-27T03:21:46.748753+0000","last_peered":"2024-04-27T03:21:46.748753+0000","last_clean":"2024-04-27T03:21:46.748753+0000","last_became_active":"2024-04-27T03:17:47.031276+0000","last_became_peered":"2024-04-27T03:17:47.031276+0000","last_unstale":"2024-04-27T03:21:46.748753+0000","last_undegraded":"2024-04-27T03:21:46.748753+0000","last_fullsized":"2024-04-27T03:21:46.748753+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:46.748445+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:46.748445+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.748445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:44:45.334105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.26","version":"27'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:41.624365+0000","last_change":"2024-04-27T03:21:41.624365+0000","last_active":"2024-04-27T03:21:41.624365+0000","last_peered":"2024-04-27T03:21:41.624365+0000","last_clean":"2024-04-27T03:21:41.624365+0000","last_became_active":"2024-04-27T03:17:49.011325+0000","last_became_peered":"2024-04-27T03:17:49.011325+0000","last_unstale":"2024-04-27T03:21:41.624365+0000","last_undegraded":"2024-04-27T03:21:41.624365+0000","last_fullsized":"2024-04-27T03:21:41.624365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:41.624007+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:41.624007+0000","last_clean_scrub_stamp":"2024-04-27T03:21:41.624007+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:34:52.078499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.21","version":"31'2","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.694164+0000","last_change":"2024-04-27T03:21:57.694164+0000","last_active":"2024-04-27T03:21:57.694164+0000","last_peered":"2024-04-27T03:21:57.694164+0000","last_clean":"2024-04-27T03:21:57.694164+0000","last_became_active":"2024-04-27T03:18:34.772040+0000","last_became_peered":"2024-04-27T03:18:34.772040+0000","last_unstale":"2024-04-27T03:21:57.694164+0000","last_undegraded":"2024-04-27T03:21:57.694164+0000","last_fullsized":"2024-04-27T03:21:57.694164+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:57.693878+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:57.693878+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.693878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:14.849137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"31'6","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.706156+0000","last_change":"2024-04-27T03:21:59.706276+0000","last_active":"2024-04-27T03:21:59.706156+0000","last_peered":"2024-04-27T03:21:59.706156+0000","last_clean":"2024-04-27T03:21:59.706156+0000","last_became_active":"2024-04-27T03:18:34.766145+0000","last_became_peered":"2024-04-27T03:18:34.766145+0000","last_unstale":"2024-04-27T03:21:59.706156+0000","last_undegraded":"2024-04-27T03:21:59.706156+0000","last_fullsized":"2024-04-27T03:21:59.706156+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:59.705879+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:59.705879+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.705879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:41:32.982953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:03.726162+0000","last_change":"2024-04-27T03:22:03.726223+0000","last_active":"2024-04-27T03:22:03.726162+0000","last_peered":"2024-04-27T03:22:03.726162+0000","last_clean":"2024-04-27T03:22:03.726162+0000","last_became_active":"2024-04-27T03:17:47.035821+0000","last_became_peered":"2024-04-27T03:17:47.035821+0000","last_unstale":"2024-04-27T03:22:03.726162+0000","last_undegraded":"2024-04-27T03:22:03.726162+0000","last_fullsized":"2024-04-27T03:22:03.726162+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:03.725962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:03.725962+0000","last_clean_scrub_stamp":"2024-04-27T03:22:03.725962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:20:58.721329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"27'6","reported_seq":76,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:35.518519+0000","last_change":"2024-04-27T03:21:35.518519+0000","last_active":"2024-04-27T03:21:35.518519+0000","last_peered":"2024-04-27T03:21:35.518519+0000","last_clean":"2024-04-27T03:21:35.518519+0000","last_became_active":"2024-04-27T03:17:49.016967+0000","last_became_peered":"2024-04-27T03:17:49.016967+0000","last_unstale":"2024-04-27T03:21:35.518519+0000","last_undegraded":"2024-04-27T03:21:35.518519+0000","last_fullsized":"2024-04-27T03:21:35.518519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'6","last_scrub_stamp":"2024-04-27T03:21:35.518220+0000","last_deep_scrub":"27'6","last_deep_scrub_stamp":"2024-04-27T03:21:35.518220+0000","last_clean_scrub_stamp":"2024-04-27T03:21:35.518220+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:52:50.091666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"29'4","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:49.732999+0000","last_change":"2024-04-27T03:21:49.732999+0000","last_active":"2024-04-27T03:21:49.732999+0000","last_peered":"2024-04-27T03:21:49.732999+0000","last_clean":"2024-04-27T03:21:49.732999+0000","last_became_active":"2024-04-27T03:17:47.015677+0000","last_became_peered":"2024-04-27T03:17:47.015677+0000","last_unstale":"2024-04-27T03:21:49.732999+0000","last_undegraded":"2024-04-27T03:21:49.732999+0000","last_fullsized":"2024-04-27T03:21:49.732999+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'4","last_scrub_stamp":"2024-04-27T03:21:49.732701+0000","last_deep_scrub":"29'4","last_deep_scrub_stamp":"2024-04-27T03:21:49.732701+0000","last_clean_scrub_stamp":"2024-04-27T03:21:49.732701+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:41.379770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.23","version":"31'6","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.819841+0000","last_change":"2024-04-27T03:21:42.819841+0000","last_active":"2024-04-27T03:21:42.819841+0000","last_peered":"2024-04-27T03:21:42.819841+0000","last_clean":"2024-04-27T03:21:42.819841+0000","last_became_active":"2024-04-27T03:17:49.016112+0000","last_became_peered":"2024-04-27T03:17:49.016112+0000","last_unstale":"2024-04-27T03:21:42.819841+0000","last_undegraded":"2024-04-27T03:21:42.819841+0000","last_fullsized":"2024-04-27T03:21:42.819841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:42.819561+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:42.819561+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.819561+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:02:40.104826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.24","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:54.709102+0000","last_change":"2024-04-27T03:21:54.709102+0000","last_active":"2024-04-27T03:21:54.709102+0000","last_peered":"2024-04-27T03:21:54.709102+0000","last_clean":"2024-04-27T03:21:54.709102+0000","last_became_active":"2024-04-27T03:17:47.039060+0000","last_became_peered":"2024-04-27T03:17:47.039060+0000","last_unstale":"2024-04-27T03:21:54.709102+0000","last_undegraded":"2024-04-27T03:21:54.709102+0000","last_fullsized":"2024-04-27T03:21:54.709102+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:54.708841+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:54.708841+0000","last_clean_scrub_stamp":"2024-04-27T03:21:54.708841+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:33:04.381720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"27'2","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:00.443313+0000","last_change":"2024-04-27T03:22:00.443313+0000","last_active":"2024-04-27T03:22:00.443313+0000","last_peered":"2024-04-27T03:22:00.443313+0000","last_clean":"2024-04-27T03:22:00.443313+0000","last_became_active":"2024-04-27T03:17:49.006218+0000","last_became_peered":"2024-04-27T03:17:49.006218+0000","last_unstale":"2024-04-27T03:22:00.443313+0000","last_undegraded":"2024-04-27T03:22:00.443313+0000","last_fullsized":"2024-04-27T03:22:00.443313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:22:00.442911+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:22:00.442911+0000","last_clean_scrub_stamp":"2024-04-27T03:22:00.442911+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:55:03.560173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.25","version":"31'2","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.515012+0000","last_change":"2024-04-27T03:21:47.515012+0000","last_active":"2024-04-27T03:21:47.515012+0000","last_peered":"2024-04-27T03:21:47.515012+0000","last_clean":"2024-04-27T03:21:47.515012+0000","last_became_active":"2024-04-27T03:17:47.023290+0000","last_became_peered":"2024-04-27T03:17:47.023290+0000","last_unstale":"2024-04-27T03:21:47.515012+0000","last_undegraded":"2024-04-27T03:21:47.515012+0000","last_fullsized":"2024-04-27T03:21:47.515012+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:47.514720+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:47.514720+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.514720+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:36:36.512861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.21","version":"31'8","reported_seq":61,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:50.711428+0000","last_change":"2024-04-27T03:21:50.711428+0000","last_active":"2024-04-27T03:21:50.711428+0000","last_peered":"2024-04-27T03:21:50.711428+0000","last_clean":"2024-04-27T03:21:50.711428+0000","last_became_active":"2024-04-27T03:17:49.014970+0000","last_became_peered":"2024-04-27T03:17:49.014970+0000","last_unstale":"2024-04-27T03:21:50.711428+0000","last_undegraded":"2024-04-27T03:21:50.711428+0000","last_fullsized":"2024-04-27T03:21:50.711428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:50.710962+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:50.710962+0000","last_clean_scrub_stamp":"2024-04-27T03:21:50.710962+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:54:53.828224+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":20,"num_write":10,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"31'4","reported_seq":57,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:53.712671+0000","last_change":"2024-04-27T03:21:53.712671+0000","last_active":"2024-04-27T03:21:53.712671+0000","last_peered":"2024-04-27T03:21:53.712671+0000","last_clean":"2024-04-27T03:21:53.712671+0000","last_became_active":"2024-04-27T03:17:47.031509+0000","last_became_peered":"2024-04-27T03:17:47.031509+0000","last_unstale":"2024-04-27T03:21:53.712671+0000","last_undegraded":"2024-04-27T03:21:53.712671+0000","last_fullsized":"2024-04-27T03:21:53.712671+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:53.712385+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:53.712385+0000","last_clean_scrub_stamp":"2024-04-27T03:21:53.712385+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:15.090838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":12,"num_write_kb":4097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"31'16","reported_seq":88,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:46.226679+0000","last_change":"2024-04-27T03:21:46.226679+0000","last_active":"2024-04-27T03:21:46.226679+0000","last_peered":"2024-04-27T03:21:46.226679+0000","last_clean":"2024-04-27T03:21:46.226679+0000","last_became_active":"2024-04-27T03:17:49.014980+0000","last_became_peered":"2024-04-27T03:17:49.014980+0000","last_unstale":"2024-04-27T03:21:46.226679+0000","last_undegraded":"2024-04-27T03:21:46.226679+0000","last_fullsized":"2024-04-27T03:21:46.226679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:46.226388+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:46.226388+0000","last_clean_scrub_stamp":"2024-04-27T03:21:46.226388+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:18.698646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":52,"num_read_kb":48,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.730514+0000","last_change":"2024-04-27T03:21:55.730514+0000","last_active":"2024-04-27T03:21:55.730514+0000","last_peered":"2024-04-27T03:21:55.730514+0000","last_clean":"2024-04-27T03:21:55.730514+0000","last_became_active":"2024-04-27T03:17:47.038144+0000","last_became_peered":"2024-04-27T03:17:47.038144+0000","last_unstale":"2024-04-27T03:21:55.730514+0000","last_undegraded":"2024-04-27T03:21:55.730514+0000","last_fullsized":"2024-04-27T03:21:55.730514+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:55.730240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:55.730240+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.730240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:04:45.047262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2f","version":"31'4","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:39.643685+0000","last_change":"2024-04-27T03:21:39.643685+0000","last_active":"2024-04-27T03:21:39.643685+0000","last_peered":"2024-04-27T03:21:39.643685+0000","last_clean":"2024-04-27T03:21:39.643685+0000","last_became_active":"2024-04-27T03:17:49.011287+0000","last_became_peered":"2024-04-27T03:17:49.011287+0000","last_unstale":"2024-04-27T03:21:39.643685+0000","last_undegraded":"2024-04-27T03:21:39.643685+0000","last_fullsized":"2024-04-27T03:21:39.643685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'4","last_scrub_stamp":"2024-04-27T03:21:39.643340+0000","last_deep_scrub":"31'4","last_deep_scrub_stamp":"2024-04-27T03:21:39.643340+0000","last_clean_scrub_stamp":"2024-04-27T03:21:39.643340+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:37:56.798600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:11.552408+0000","last_change":"2024-04-27T03:22:11.552473+0000","last_active":"2024-04-27T03:22:11.552408+0000","last_peered":"2024-04-27T03:22:11.552408+0000","last_clean":"2024-04-27T03:22:11.552408+0000","last_became_active":"2024-04-27T03:17:47.014293+0000","last_became_peered":"2024-04-27T03:17:47.014293+0000","last_unstale":"2024-04-27T03:22:11.552408+0000","last_undegraded":"2024-04-27T03:22:11.552408+0000","last_fullsized":"2024-04-27T03:22:11.552408+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:11.552206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:11.552206+0000","last_clean_scrub_stamp":"2024-04-27T03:22:11.552206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:59.839853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2e","version":"31'12","reported_seq":85,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:08.617583+0000","last_change":"2024-04-27T03:22:08.617583+0000","last_active":"2024-04-27T03:22:08.617583+0000","last_peered":"2024-04-27T03:22:08.617583+0000","last_clean":"2024-04-27T03:22:08.617583+0000","last_became_active":"2024-04-27T03:17:49.010527+0000","last_became_peered":"2024-04-27T03:17:49.010527+0000","last_unstale":"2024-04-27T03:22:08.617583+0000","last_undegraded":"2024-04-27T03:22:08.617583+0000","last_fullsized":"2024-04-27T03:22:08.617583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:22:08.617332+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:22:08.617332+0000","last_clean_scrub_stamp":"2024-04-27T03:22:08.617332+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:04:22.117072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":50,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:15.518445+0000","last_change":"2024-04-27T03:22:15.518445+0000","last_active":"2024-04-27T03:22:15.518445+0000","last_peered":"2024-04-27T03:22:15.518445+0000","last_clean":"2024-04-27T03:22:15.518445+0000","last_became_active":"2024-04-27T03:17:47.019284+0000","last_became_peered":"2024-04-27T03:17:47.019284+0000","last_unstale":"2024-04-27T03:22:15.518445+0000","last_undegraded":"2024-04-27T03:22:15.518445+0000","last_fullsized":"2024-04-27T03:22:15.518445+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:15.518159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:15.518159+0000","last_clean_scrub_stamp":"2024-04-27T03:22:15.518159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:12:17.326841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"31'2","reported_seq":49,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:42.767075+0000","last_change":"2024-04-27T03:21:42.767075+0000","last_active":"2024-04-27T03:21:42.767075+0000","last_peered":"2024-04-27T03:21:42.767075+0000","last_clean":"2024-04-27T03:21:42.767075+0000","last_became_active":"2024-04-27T03:17:49.005267+0000","last_became_peered":"2024-04-27T03:17:49.005267+0000","last_unstale":"2024-04-27T03:21:42.767075+0000","last_undegraded":"2024-04-27T03:21:42.767075+0000","last_fullsized":"2024-04-27T03:21:42.767075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'2","last_scrub_stamp":"2024-04-27T03:21:42.766832+0000","last_deep_scrub":"31'2","last_deep_scrub_stamp":"2024-04-27T03:21:42.766832+0000","last_clean_scrub_stamp":"2024-04-27T03:21:42.766832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:32:00.664417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:55.196227+0000","last_change":"2024-04-27T03:21:55.196227+0000","last_active":"2024-04-27T03:21:55.196227+0000","last_peered":"2024-04-27T03:21:55.196227+0000","last_clean":"2024-04-27T03:21:55.196227+0000","last_became_active":"2024-04-27T03:17:47.020689+0000","last_became_peered":"2024-04-27T03:17:47.020689+0000","last_unstale":"2024-04-27T03:21:55.196227+0000","last_undegraded":"2024-04-27T03:21:55.196227+0000","last_fullsized":"2024-04-27T03:21:55.196227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:55.195864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:55.195864+0000","last_clean_scrub_stamp":"2024-04-27T03:21:55.195864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:37:11.669674+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:10.301346+0000","last_change":"2024-04-27T03:22:10.301346+0000","last_active":"2024-04-27T03:22:10.301346+0000","last_peered":"2024-04-27T03:22:10.301346+0000","last_clean":"2024-04-27T03:22:10.301346+0000","last_became_active":"2024-04-27T03:17:49.008967+0000","last_became_peered":"2024-04-27T03:17:49.008967+0000","last_unstale":"2024-04-27T03:22:10.301346+0000","last_undegraded":"2024-04-27T03:22:10.301346+0000","last_fullsized":"2024-04-27T03:22:10.301346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:10.300994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:10.300994+0000","last_clean_scrub_stamp":"2024-04-27T03:22:10.300994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:35:48.215930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:09.339067+0000","last_change":"2024-04-27T03:22:09.339067+0000","last_active":"2024-04-27T03:22:09.339067+0000","last_peered":"2024-04-27T03:22:09.339067+0000","last_clean":"2024-04-27T03:22:09.339067+0000","last_became_active":"2024-04-27T03:17:47.015330+0000","last_became_peered":"2024-04-27T03:17:47.015330+0000","last_unstale":"2024-04-27T03:22:09.339067+0000","last_undegraded":"2024-04-27T03:22:09.339067+0000","last_fullsized":"2024-04-27T03:22:09.339067+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:09.338670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:09.338670+0000","last_clean_scrub_stamp":"2024-04-27T03:22:09.338670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:40:51.748061+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2b","version":"31'8","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.217738+0000","last_change":"2024-04-27T03:21:47.217738+0000","last_active":"2024-04-27T03:21:47.217738+0000","last_peered":"2024-04-27T03:21:47.217738+0000","last_clean":"2024-04-27T03:21:47.217738+0000","last_became_active":"2024-04-27T03:17:49.016691+0000","last_became_peered":"2024-04-27T03:17:49.016691+0000","last_unstale":"2024-04-27T03:21:47.217738+0000","last_undegraded":"2024-04-27T03:21:47.217738+0000","last_fullsized":"2024-04-27T03:21:47.217738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'8","last_scrub_stamp":"2024-04-27T03:21:47.217463+0000","last_deep_scrub":"31'8","last_deep_scrub_stamp":"2024-04-27T03:21:47.217463+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.217463+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:56:30.144213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:57.799873+0000","last_change":"2024-04-27T03:21:57.799873+0000","last_active":"2024-04-27T03:21:57.799873+0000","last_peered":"2024-04-27T03:21:57.799873+0000","last_clean":"2024-04-27T03:21:57.799873+0000","last_became_active":"2024-04-27T03:17:47.038461+0000","last_became_peered":"2024-04-27T03:17:47.038461+0000","last_unstale":"2024-04-27T03:21:57.799873+0000","last_undegraded":"2024-04-27T03:21:57.799873+0000","last_fullsized":"2024-04-27T03:21:57.799873+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:57.799593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:57.799593+0000","last_clean_scrub_stamp":"2024-04-27T03:21:57.799593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:23:49.562735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2a","version":"31'6","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:56.509929+0000","last_change":"2024-04-27T03:21:56.509929+0000","last_active":"2024-04-27T03:21:56.509929+0000","last_peered":"2024-04-27T03:21:56.509929+0000","last_clean":"2024-04-27T03:21:56.509929+0000","last_became_active":"2024-04-27T03:17:49.005839+0000","last_became_peered":"2024-04-27T03:17:49.005839+0000","last_unstale":"2024-04-27T03:21:56.509929+0000","last_undegraded":"2024-04-27T03:21:56.509929+0000","last_fullsized":"2024-04-27T03:21:56.509929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'6","last_scrub_stamp":"2024-04-27T03:21:56.509666+0000","last_deep_scrub":"31'6","last_deep_scrub_stamp":"2024-04-27T03:21:56.509666+0000","last_clean_scrub_stamp":"2024-04-27T03:21:56.509666+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:43:38.981565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":28,"num_write":8,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:26.292795+0000","last_change":"2024-04-27T03:22:26.292972+0000","last_active":"2024-04-27T03:22:26.292795+0000","last_peered":"2024-04-27T03:22:26.292795+0000","last_clean":"2024-04-27T03:22:26.292795+0000","last_became_active":"2024-04-27T03:19:35.440068+0000","last_became_peered":"2024-04-27T03:19:35.440068+0000","last_unstale":"2024-04-27T03:22:26.292795+0000","last_undegraded":"2024-04-27T03:22:26.292795+0000","last_fullsized":"2024-04-27T03:22:26.292795+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:26.292570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:26.292570+0000","last_clean_scrub_stamp":"2024-04-27T03:22:26.292570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:28:35.775530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:47.754460+0000","last_change":"2024-04-27T03:21:47.754460+0000","last_active":"2024-04-27T03:21:47.754460+0000","last_peered":"2024-04-27T03:21:47.754460+0000","last_clean":"2024-04-27T03:21:47.754460+0000","last_became_active":"2024-04-27T03:17:49.015427+0000","last_became_peered":"2024-04-27T03:17:49.015427+0000","last_unstale":"2024-04-27T03:21:47.754460+0000","last_undegraded":"2024-04-27T03:21:47.754460+0000","last_fullsized":"2024-04-27T03:21:47.754460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:47.754158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:47.754158+0000","last_clean_scrub_stamp":"2024-04-27T03:21:47.754158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:27:25.426021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:51.774180+0000","last_change":"2024-04-27T03:21:51.774180+0000","last_active":"2024-04-27T03:21:51.774180+0000","last_peered":"2024-04-27T03:21:51.774180+0000","last_clean":"2024-04-27T03:21:51.774180+0000","last_became_active":"2024-04-27T03:17:47.022006+0000","last_became_peered":"2024-04-27T03:17:47.022006+0000","last_unstale":"2024-04-27T03:21:51.774180+0000","last_undegraded":"2024-04-27T03:21:51.774180+0000","last_fullsized":"2024-04-27T03:21:51.774180+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:51.773922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:51.773922+0000","last_clean_scrub_stamp":"2024-04-27T03:21:51.773922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T13:17:53.853087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.28","version":"31'16","reported_seq":89,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:43.173295+0000","last_change":"2024-04-27T03:21:43.173295+0000","last_active":"2024-04-27T03:21:43.173295+0000","last_peered":"2024-04-27T03:21:43.173295+0000","last_clean":"2024-04-27T03:21:43.173295+0000","last_became_active":"2024-04-27T03:17:49.015185+0000","last_became_peered":"2024-04-27T03:17:49.015185+0000","last_unstale":"2024-04-27T03:21:43.173295+0000","last_undegraded":"2024-04-27T03:21:43.173295+0000","last_fullsized":"2024-04-27T03:21:43.173295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'16","last_scrub_stamp":"2024-04-27T03:21:43.173036+0000","last_deep_scrub":"31'16","last_deep_scrub_stamp":"2024-04-27T03:21:43.173036+0000","last_clean_scrub_stamp":"2024-04-27T03:21:43.173036+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:26:46.540282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":59,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:16.516434+0000","last_change":"2024-04-27T03:22:16.516434+0000","last_active":"2024-04-27T03:22:16.516434+0000","last_peered":"2024-04-27T03:22:16.516434+0000","last_clean":"2024-04-27T03:22:16.516434+0000","last_became_active":"2024-04-27T03:17:47.025570+0000","last_became_peered":"2024-04-27T03:17:47.025570+0000","last_unstale":"2024-04-27T03:22:16.516434+0000","last_undegraded":"2024-04-27T03:22:16.516434+0000","last_fullsized":"2024-04-27T03:22:16.516434+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:16.516073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:16.516073+0000","last_clean_scrub_stamp":"2024-04-27T03:22:16.516073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:50:01.263243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":52,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:22:17.330456+0000","last_change":"2024-04-27T03:22:17.330456+0000","last_active":"2024-04-27T03:22:17.330456+0000","last_peered":"2024-04-27T03:22:17.330456+0000","last_clean":"2024-04-27T03:22:17.330456+0000","last_became_active":"2024-04-27T03:17:47.002152+0000","last_became_peered":"2024-04-27T03:17:47.002152+0000","last_unstale":"2024-04-27T03:22:17.330456+0000","last_undegraded":"2024-04-27T03:22:17.330456+0000","last_fullsized":"2024-04-27T03:22:17.330456+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:22:17.330173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:22:17.330173+0000","last_clean_scrub_stamp":"2024-04-27T03:22:17.330173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T12:45:13.398363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.37","version":"31'12","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:31.724438+0000","last_change":"2024-04-27T03:21:31.724508+0000","last_active":"2024-04-27T03:21:31.724438+0000","last_peered":"2024-04-27T03:21:31.724438+0000","last_clean":"2024-04-27T03:21:31.724438+0000","last_became_active":"2024-04-27T03:17:49.013766+0000","last_became_peered":"2024-04-27T03:17:49.013766+0000","last_unstale":"2024-04-27T03:21:31.724438+0000","last_undegraded":"2024-04-27T03:21:31.724438+0000","last_fullsized":"2024-04-27T03:21:31.724438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"31'12","last_scrub_stamp":"2024-04-27T03:21:31.724252+0000","last_deep_scrub":"31'12","last_deep_scrub_stamp":"2024-04-27T03:21:31.724252+0000","last_clean_scrub_stamp":"2024-04-27T03:21:31.724252+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:49.406247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":11,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.31","version":"27'2","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:59.430741+0000","last_change":"2024-04-27T03:21:59.430741+0000","last_active":"2024-04-27T03:21:59.430741+0000","last_peered":"2024-04-27T03:21:59.430741+0000","last_clean":"2024-04-27T03:21:59.430741+0000","last_became_active":"2024-04-27T03:17:47.021427+0000","last_became_peered":"2024-04-27T03:17:47.021427+0000","last_unstale":"2024-04-27T03:21:59.430741+0000","last_undegraded":"2024-04-27T03:21:59.430741+0000","last_fullsized":"2024-04-27T03:21:59.430741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"27'2","last_scrub_stamp":"2024-04-27T03:21:59.430346+0000","last_deep_scrub":"27'2","last_deep_scrub_stamp":"2024-04-27T03:21:59.430346+0000","last_clean_scrub_stamp":"2024-04-27T03:21:59.430346+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T15:11:32.065734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-27T03:21:38.602897+0000","last_change":"2024-04-27T03:21:38.603000+0000","last_active":"2024-04-27T03:21:38.602897+0000","last_peered":"2024-04-27T03:21:38.602897+0000","last_clean":"2024-04-27T03:21:38.602897+0000","last_became_active":"2024-04-27T03:17:49.012563+0000","last_became_peered":"2024-04-27T03:17:49.012563+0000","last_unstale":"2024-04-27T03:21:38.602897+0000","last_undegraded":"2024-04-27T03:21:38.602897+0000","last_fullsized":"2024-04-27T03:21:38.602897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-27T03:21:38.602665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-27T03:21:38.602665+0000","last_clean_scrub_stamp":"2024-04-27T03:21:38.602665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T14:51:00.698430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1541,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":657,"num_read_kb":625,"num_write":560,"num_write_kb":172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":349,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":277,"ondisk_log_size":277,"up":64,"acting":64,"num_store_stats":4},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2280,"ondisk_log_size":2280,"up":64,"acting":64,"num_store_stats":4},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":11674,"num_objects":215,"num_object_clones":0,"num_object_copies":430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":215,"num_whiteouts":0,"num_read":1888,"num_read_kb":1653,"num_write":789,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2692,"num_omap_keys":14,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":7404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1749,"ondisk_log_size":1749,"up":64,"acting":64,"num_store_stats":4},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1800,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":972,"num_read_kb":972,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":4},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":1615,"num_read_kb":1505,"num_write":528,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":483,"ondisk_log_size":483,"up":128,"acting":128,"num_store_stats":4},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":87,"num_write":316,"num_write_kb":58617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":192,"acting":192,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":157,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44072,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327768,"statfs":{"total":96636764160,"available":96591634432,"internally_reserved":0,"allocated":425984,"data_stored":148688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":134,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":983040,"data_stored":736592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705726,"num_pgs":145,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1040384,"data_stored":739335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":158,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39944,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331896,"statfs":{"total":96636764160,"available":96595861504,"internally_reserved":0,"allocated":458752,"data_stored":147935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-27T03:22:34.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-27T03:22:34.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feacbf99640 1 Processor -- start 2024-04-27T03:22:34.935 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feacbf99640 1 -- start start 2024-04-27T03:22:34.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feacbf99640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feac406d4d0 0x7feac406d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:34.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feacbf99640 1 -- --> v1:172.21.15.77:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac409baa0 con 0x7feac4071140 2024-04-27T03:22:34.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feacbf99640 1 -- --> v2:172.21.15.77:3300/0 -- mon_getmap magic: 0 -- 0x7feac4070ad0 con 0x7feac406d4d0 2024-04-27T03:22:34.936 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac9d35640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feac406d4d0 0x7feac406d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:34.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac9d35640 1 --2- >> v2:172.21.15.77:3300/0 conn(0x7feac406d4d0 0x7feac406d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47352/0 (socket says 172.21.15.77:47352) 2024-04-27T03:22:34.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac9d35640 1 -- 172.21.15.77:0/2851468175 learned_addr learned my addr 172.21.15.77:0/2851468175 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:22:34.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac9534640 1 --1- >> v1:172.21.15.77:6789/0 conn(0x7feac4071140 0x7feac4096060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.77:6789/0 says I am v1:172.21.15.77:55130/0 (socket says 172.21.15.77:55130) 2024-04-27T03:22:34.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac9d35640 1 -- 172.21.15.77:0/2851468175 >> v1:172.21.15.77:6789/0 conn(0x7feac4071140 legacy=0x7feac4096060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:34.937 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac9d35640 1 -- 172.21.15.77:0/2851468175 --> v2:172.21.15.77:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac4072000 con 0x7feac406d4d0 2024-04-27T03:22:34.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac9d35640 1 --2- 172.21.15.77:0/2851468175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac406d8b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7feac00097f0 tx=0x7feac001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c489d9f55d2a8d server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:34.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac8d33640 1 -- 172.21.15.77:0/2851468175 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac00ab520 con 0x7feac406d4d0 2024-04-27T03:22:34.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac8d33640 1 -- 172.21.15.77:0/2851468175 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac00b0070 con 0x7feac406d4d0 2024-04-27T03:22:34.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.928+0000 7feac8d33640 1 -- 172.21.15.77:0/2851468175 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac00ab810 con 0x7feac406d4d0 2024-04-27T03:22:34.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/2851468175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 msgr2=0x7feac406d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:34.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 --2- 172.21.15.77:0/2851468175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac406d8b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7feac00097f0 tx=0x7feac001c3d0 comp rx=0 tx=0).stop 2024-04-27T03:22:34.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/2851468175 shutdown_connections 2024-04-27T03:22:34.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 --2- 172.21.15.77:0/2851468175 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac406d8b0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:34.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/2851468175 >> 172.21.15.77:0/2851468175 conn(0x7feac40993d0 msgr2=0x7feac40997e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:22:34.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/2851468175 shutdown_connections 2024-04-27T03:22:34.940 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/2851468175 wait complete. 2024-04-27T03:22:34.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 Processor -- start 2024-04-27T03:22:34.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- start start 2024-04-27T03:22:34.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac41217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 -- 0x7feac41349e0 con 0x7feac406d4d0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feac9d35640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac41217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feac9d35640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac41217c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47368/0 (socket says 172.21.15.77:47368) 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feac9d35640 1 -- 172.21.15.77:0/3791765835 learned_addr learned my addr 172.21.15.77:0/3791765835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feac9d35640 1 -- 172.21.15.77:0/3791765835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac4131260 con 0x7feac406d4d0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feac9d35640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac41217c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7feac001a060 tx=0x7feac001ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac00abe40 con 0x7feac406d4d0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac00be3f0 con 0x7feac406d4d0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac41228d0 con 0x7feac406d4d0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feac00be6d0 con 0x7feac406d4d0 2024-04-27T03:22:34.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feac4121ed0 con 0x7feac406d4d0 2024-04-27T03:22:34.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7feac00c7440 con 0x7feac406d4d0 2024-04-27T03:22:34.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac4062e50 con 0x7feac406d4d0 2024-04-27T03:22:34.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feaba7fc640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7feaa8035d60 0x7feaa8038220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T03:22:34.943 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.932+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6429+0+0 (secure 0 0 0) 0x7feac00e38a0 con 0x7feac406d4d0 2024-04-27T03:22:34.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.936+0000 7feac9534640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7feaa8035d60 0x7feaa8038220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T03:22:34.944 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.936+0000 7feac9534640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7feaa8035d60 0x7feaa8038220 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feab4001600 tx=0x7feab4007a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T03:22:34.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:34.940+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feac00160b0 con 0x7feac406d4d0 2024-04-27T03:22:35.110 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.100+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7feac4123e50 con 0x7feac406d4d0 2024-04-27T03:22:35.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.104+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7feac00bd790 con 0x7feac406d4d0 2024-04-27T03:22:35.113 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.104+0000 7feaba7fc640 1 -- 172.21.15.77:0/3791765835 <== mon.0 v2:172.21.15.77:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7feac00b0020 con 0x7feac406d4d0 2024-04-27T03:22:35.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7feaa8035d60 msgr2=0x7feaa8038220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:35.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7feaa8035d60 0x7feaa8038220 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feab4001600 tx=0x7feab4007a50 comp rx=0 tx=0).stop 2024-04-27T03:22:35.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 msgr2=0x7feac41217c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T03:22:35.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac41217c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7feac001a060 tx=0x7feac001ac70 comp rx=0 tx=0).stop 2024-04-27T03:22:35.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 shutdown_connections 2024-04-27T03:22:35.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.110:6824/2492415371,v1:172.21.15.110:6825/2492415371] conn(0x7feaa8035d60 0x7feaa8038220 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:35.116 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 --2- 172.21.15.77:0/3791765835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7feac406d4d0 0x7feac41217c0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T03:22:35.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 >> 172.21.15.77:0/3791765835 conn(0x7feac40993d0 msgr2=0x7feac4128a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T03:22:35.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 shutdown_connections 2024-04-27T03:22:35.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-04-27T03:22:35.108+0000 7feacbf99640 1 -- 172.21.15.77:0/3791765835 wait complete. 2024-04-27T03:22:35.131 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-27T03:22:35.131 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-27T03:22:35.132 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-27T03:22:35.132 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T03:22:35.223 INFO:tasks.ceph.osd.0:Stopped 2024-04-27T03:22:35.224 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-27T03:22:35.224 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T03:22:35.297 INFO:tasks.ceph.osd.1:Stopped 2024-04-27T03:22:35.297 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-27T03:22:35.298 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T03:22:40.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-27T03:22:41.399 INFO:tasks.ceph.osd.2:Stopped 2024-04-27T03:22:41.399 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-27T03:22:41.399 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T03:22:41.478 INFO:tasks.ceph.osd.3:Stopped 2024-04-27T03:22:41.478 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-27T03:22:41.478 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-27T03:22:41.478 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T03:22:41.508 INFO:tasks.ceph.mgr.x:Stopped 2024-04-27T03:22:41.508 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-27T03:22:41.508 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-27T03:22:41.509 INFO:teuthology.orchestra.run:waiting for 300 2024-04-27T03:22:41.523 INFO:tasks.ceph.mon.a:Stopped 2024-04-27T03:22:41.523 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-27T03:22:41.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | head -n 1 2024-04-27T03:22:41.581 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T03:22:41.581 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-27T03:22:41.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:22:41.947 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-27T03:22:42.245 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:22:42.245 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-27T03:22:42.432 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:22:42.433 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-27T03:22:42.589 INFO:tasks.ceph:Archiving mon data... 2024-04-27T03:22:42.590 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/data/mon.a.tgz 2024-04-27T03:22:42.590 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-04-27T03:22:42.597 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.blVVAFkl3Q 2024-04-27T03:22:42.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.blVVAFkl3Q 2024-04-27T03:22:42.715 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.blVVAFkl3Q 2024-04-27T03:22:42.775 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.blVVAFkl3Q is 268KB 2024-04-27T03:22:42.833 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.blVVAFkl3Q 2024-04-27T03:22:42.838 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-27T03:22:42.838 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-27T03:22:42.880 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-27T03:22:42.939 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-27T03:22:42.945 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-27T03:22:42.945 INFO:tasks.ceph:Archiving crash dumps... 2024-04-27T03:22:42.946 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/remote/smithi077/crash 2024-04-27T03:22:42.947 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-27T03:22:42.996 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/remote/smithi110/crash 2024-04-27T03:22:42.997 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-27T03:22:43.010 INFO:tasks.ceph:Compressing logs... 2024-04-27T03:22:43.010 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T03:22:43.041 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T03:22:43.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23990.log 2024-04-27T03:22:43.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23834.log 2024-04-27T03:22:43.061 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23742.log 2024-04-27T03:22:43.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23990.log: /var/log/ceph/ceph-client.admin.23834.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.23834.log.gz 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24016.log 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.23990.log.gz 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-osd.0.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.23742.log.gz 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24120.log 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23938.log 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.24016.log.gz 2024-04-27T03:22:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23384.log 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.24120.log.gz 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23598.log 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23384.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.23938.log.gz 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24198.log 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.23384.log.gz 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24172.log 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.23598.log.gz 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23964.log 2024-04-27T03:22:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24172.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.24198.log.gz 2024-04-27T03:22:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23808.log 2024-04-27T03:22:43.067 INFO:teuthology.orchestra.run.smithi077.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.24172.log.gz 2024-04-27T03:22:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23486.log 2024-04-27T03:22:43.067 INFO:teuthology.orchestra.run.smithi077.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.23964.log.gz 2024-04-27T03:22:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23808.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-27T03:22:43.067 INFO:teuthology.orchestra.run.smithi077.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.23808.log.gz 2024-04-27T03:22:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24068.log 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-mon.a.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.23486.log.gz 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23782.log 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23209.log 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.24068.log.gz 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23586.log 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.23782.log.gz 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24042.log 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.23209.log.gz 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24146.log 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.23586.log.gz 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23716.log 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24042.log.gz 2024-04-27T03:22:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23590.log 2024-04-27T03:22:43.072 INFO:teuthology.orchestra.run.smithi077.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.24146.log.gz 2024-04-27T03:22:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23389.log 2024-04-27T03:22:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23886.log 2024-04-27T03:22:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23590.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.23716.log.gz 2024-04-27T03:22:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23389.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.23590.log.gz 2024-04-27T03:22:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23690.log 2024-04-27T03:22:43.072 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.23389.log.gz 2024-04-27T03:22:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23912.log 2024-04-27T03:22:43.074 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.23886.log.gz 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24094.log 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.23690.log.gz 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23493.log 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24094.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.23912.log.gz 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- 85.5% /var/log/ceph/ceph.log 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24094.log.gz 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20268.log 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23320.log 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.23493.log.gz 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.tmp-client.admin.20268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21522.log 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20268.log.gz 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23860.log 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.23320.log.gz 2024-04-27T03:22:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.21522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24222.log 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23860.log: gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.21522.log.gz --verbose 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: -- /var/log/ceph/ceph-client.admin.23386.log 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.23860.log.gz 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23234.log 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.24222.log: /var/log/ceph/ceph-client.admin.23386.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.24222.log.gz 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.23386.log.gz 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21633.log 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.23234.log.gz 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23494.log 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.21633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23288.log 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.21633.log.gz 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-27T03:22:43.081 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23482.log 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21592.log 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.23494.log.gz 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.23288.log.gz 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23262.log 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23390.log 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.23482.log.gz 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.21592.log: /var/log/ceph/ceph-client.admin.23262.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.21592.log.gz 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23347.log 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.23262.log.gz 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23597.log 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23390.log: /var/log/ceph/ceph-client.admin.23347.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23390.log.gz 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.23347.log.gz 2024-04-27T03:22:43.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph-client.admin.23597.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.23597.log.gz 2024-04-27T03:22:43.092 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23834.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82002.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.23834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59020.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.23834.log.gz 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.82002.log: gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.59020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69050.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58430.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.82002.log.gz 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63740.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.69050.log: /var/log/ceph/ceph-client.0.58430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76126.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.59020.log.gz 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.63740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38370.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.76126.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.58430.log.gz 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66690.log 2024-04-27T03:22:43.093 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.69050.log.gz 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65510.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.38370.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.76126.log.gz 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.66690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.35420.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph-client.0.65510.log: -5 --verbose -- /var/log/ceph/ceph-client.0.83182.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.63740.log.gz 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.35420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.28346.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.66690.log.gz 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30116.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.83182.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.38370.log.gz 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.28346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47810.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.30116.log: 91.6% 91.9% -- replaced with /var/log/ceph/ceph-client.0.65510.log.gz 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54890.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.0.35420.log.gz 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.47810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61970.log 2024-04-27T03:22:43.094 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.83182.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.54890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34240.log 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.0.28346.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.61970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52530.log 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.30116.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.34240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77878.log 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.52530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.40730.log 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.54890.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.77878.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.47810.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53120.log 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.61970.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.34240.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37190.log 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.40730.log: 90.7% -- replaced with /var/log/ceph/ceph-client.0.77878.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77294.log 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.53120.log: 91.2% -- replaced with /var/log/ceph/ceph-client.0.52530.log.gz 2024-04-27T03:22:43.095 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.37190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59610.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.77294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67280.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.40730.log.gz 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.59610.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.0.77294.log.gz 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.67280.log: gzip 91.4% -- replaced with /var/log/ceph/ceph-client.0.37190.log.gz -5 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.0.31296.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.83772.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.31296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69640.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.53120.log.gz 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.83772.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.67280.log.gz 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31880.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.0.31296.log.gz 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.69640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.29526.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.0.59610.log.gz 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.0.31880.log: -- /var/log/ceph/ceph-client.0.44860.log 2024-04-27T03:22:43.096 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.29526.log: gzip 91.3% -5 -- replaced with /var/log/ceph/ceph-client.0.83772.log.gz --verbose 2024-04-27T03:22:43.097 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.0.86727.log 2024-04-27T03:22:43.097 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.44860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73772.log 2024-04-27T03:22:43.097 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.69640.log.gz 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.86727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.28936.log 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.0.29526.log.gz 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.73772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39550.log 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr: 92.1% 91.6% -- replaced with /var/log/ceph/ceph-client.0.44860.log.gz 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.28936.log: -- replaced with /var/log/ceph/ceph-client.0.31880.log.gz 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50170.log 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.39550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45450.log 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.50170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60790.log 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.73772.log.gz 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.45450.log: 91.6% -- replaced with /var/log/ceph/ceph-client.0.28936.log.gz 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.39550.log.gz 2024-04-27T03:22:43.101 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.50170.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr: 93.4% -- replaced with /var/log/ceph/ceph-client.0.86727.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74362.log 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.0.45450.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.60790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67870.log 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.74362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75542.log 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.67870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74952.log 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.75542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41320.log 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.74952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72590.log 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.0.75542.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.67870.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.0.60790.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.41320.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.74362.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.74952.log.gz 2024-04-27T03:22:43.102 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.41320.log.gz 2024-04-27T03:22:43.105 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27630.log 2024-04-27T03:22:43.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.72590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30706.log 2024-04-27T03:22:43.105 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.27630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85542.log 2024-04-27T03:22:43.105 INFO:teuthology.orchestra.run.smithi110.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.27630.log.gz 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.30706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48400.log 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36600.log/var/log/ceph/ceph-client.0.85542.log: 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.48400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79642.log 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.36600.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6%/var/log/ceph/ceph-client.0.79642.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.72590.log.gz 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.0.30706.log.gz 91.3% -- replaced with /var/log/ceph/ceph-client.0.85542.log.gz 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68460.log 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.48400.log.gz 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.3.log: 91.5% -- replaced with /var/log/ceph/ceph-client.0.36600.log.gz 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70230.log 2024-04-27T03:22:43.106 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53710.log 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.68460.log: /var/log/ceph/ceph-client.0.70230.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.79642.log.gz 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54300.log 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34830.log 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.53710.log: /var/log/ceph/ceph-client.0.54300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49580.log 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.68460.log.gz 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.34830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27654.log 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.49580.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.53710.log.gz 2024-04-27T03:22:43.108 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.70230.log.gz 2024-04-27T03:22:43.109 INFO:teuthology.orchestra.run.smithi110.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.54300.log.gz 2024-04-27T03:22:43.109 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.34830.log.gz 2024-04-27T03:22:43.110 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64920.log 2024-04-27T03:22:43.110 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.49580.log.gz 2024-04-27T03:22:43.110 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.27654.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-27T03:22:43.111 INFO:teuthology.orchestra.run.smithi110.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.27654.log.gz 2024-04-27T03:22:43.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.64920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72000.log 2024-04-27T03:22:43.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43090.log 2024-04-27T03:22:43.111 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.72000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23882.log 2024-04-27T03:22:43.112 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.43090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43680.log 2024-04-27T03:22:43.112 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6%/var/log/ceph/ceph-client.admin.23882.log: -- replaced with /var/log/ceph/ceph-client.0.64920.log.gz 2024-04-27T03:22:43.112 INFO:teuthology.orchestra.run.smithi110.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.23882.log.gz 2024-04-27T03:22:43.112 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.84362.log 2024-04-27T03:22:43.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.43680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50760.log 2024-04-27T03:22:43.113 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.43090.log.gz 2024-04-27T03:22:43.113 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.84362.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.72000.log.gz 2024-04-27T03:22:43.114 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70820.log 2024-04-27T03:22:43.114 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.43680.log.gz 2024-04-27T03:22:43.115 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph-client.0.50760.log: -5 --verbose -- /var/log/ceph/ceph-client.0.61380.log 2024-04-27T03:22:43.115 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.70820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57250.log 2024-04-27T03:22:43.115 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.84362.log.gz/var/log/ceph/ceph-client.0.61380.log: 2024-04-27T03:22:43.115 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.50760.log.gz 2024-04-27T03:22:43.116 INFO:teuthology.orchestra.run.smithi110.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.0.70820.log.gz 2024-04-27T03:22:43.116 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46630.log 2024-04-27T03:22:43.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.57250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64330.log 2024-04-27T03:22:43.117 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.61380.log.gz 2024-04-27T03:22:43.117 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.46630.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-27T03:22:43.118 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82592.log 2024-04-27T03:22:43.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.64330.log: /var/log/ceph/ceph-mgr.x.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.57250.log.gz 2024-04-27T03:22:43.119 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.46630.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.0.51940.log 2024-04-27T03:22:43.119 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-04-27T03:22:43.119 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.82592.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.64330.log.gz 2024-04-27T03:22:43.120 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37780.log 2024-04-27T03:22:43.120 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76710.log 2024-04-27T03:22:43.121 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.51940.log: /var/log/ceph/ceph-client.0.37780.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.82592.log.gz 2024-04-27T03:22:43.121 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60200.log 2024-04-27T03:22:43.122 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.76710.log: 91.5% -- replaced with /var/log/ceph/ceph-client.0.37780.log.gz 2024-04-27T03:22:43.122 INFO:teuthology.orchestra.run.smithi110.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.51940.log.gz 2024-04-27T03:22:43.122 INFO:teuthology.orchestra.run.smithi110.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.0.76710.log.gz 2024-04-27T03:22:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44270.log 2024-04-27T03:22:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.60200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36010.log 2024-04-27T03:22:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.44270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56660.log 2024-04-27T03:22:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.36010.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.44270.log.gz 2024-04-27T03:22:43.125 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.36010.log.gz 2024-04-27T03:22:43.126 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51350.log 2024-04-27T03:22:43.126 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.56660.log: 93.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63150.log 2024-04-27T03:22:43.126 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.0.60200.log.gz 2024-04-27T03:22:43.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.51350.log: 91.5% -- replaced with /var/log/ceph/ceph-client.0.56660.log.gz 2024-04-27T03:22:43.128 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.0.51350.log.gz 2024-04-27T03:22:43.128 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27582.log 2024-04-27T03:22:43.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.63150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78462.log 2024-04-27T03:22:43.130 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.27582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.32470.log 2024-04-27T03:22:43.130 INFO:teuthology.orchestra.run.smithi110.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.27582.log.gz 2024-04-27T03:22:43.131 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.78462.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.78462.log.gz 2024-04-27T03:22:43.132 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.0.63150.log.gz 2024-04-27T03:22:43.132 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ops-log-ceph-client.0.log 2024-04-27T03:22:43.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.32470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66100.log 2024-04-27T03:22:43.133 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ops-log-ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57840.log 2024-04-27T03:22:43.134 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.66100.log: 91.8% -- replaced with /var/log/ceph/ops-log-ceph-client.0.log.gz 2024-04-27T03:22:43.134 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.32470.log.gz 2024-04-27T03:22:43.135 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33060.log 2024-04-27T03:22:43.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.57840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56070.log 2024-04-27T03:22:43.135 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.33060.log: 91.9% -- replaced with /var/log/ceph/ceph-client.0.66100.log.gz 2024-04-27T03:22:43.135 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41910.log 2024-04-27T03:22:43.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.56070.log: 92.0% -- replaced with /var/log/ceph/ceph-client.0.57840.log.gz 2024-04-27T03:22:43.137 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.33060.log.gz 2024-04-27T03:22:43.137 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23858.log 2024-04-27T03:22:43.137 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.41910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86132.log 2024-04-27T03:22:43.138 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.23858.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.23858.log.gz 2024-04-27T03:22:43.138 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.56070.log.gz 2024-04-27T03:22:43.138 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.40140.log 2024-04-27T03:22:43.139 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.41910.log.gz 2024-04-27T03:22:43.139 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.81412.log 2024-04-27T03:22:43.139 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.86132.log: /var/log/ceph/ceph-client.0.40140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.71410.log 2024-04-27T03:22:43.141 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.81412.log: 91.6% 91.4% -- replaced with /var/log/ceph/ceph-client.0.86132.log.gz 2024-04-27T03:22:43.141 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.0.40140.log.gz 2024-04-27T03:22:43.141 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55480.log 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.81412.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27606.log 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.71410.log: 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr: 92.3% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.55480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23810.log 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.27606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48990.log 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.27606.log.gz/var/log/ceph/ceph-client.admin.23810.log: 91.3% 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.0.71410.log.gz 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.23810.log.gz 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47220.log 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.48990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80822.log 2024-04-27T03:22:43.144 INFO:teuthology.orchestra.run.smithi110.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.55480.log.gz 2024-04-27T03:22:43.145 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.47220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79052.log 2024-04-27T03:22:43.146 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33650.log 2024-04-27T03:22:43.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.80822.log: /var/log/ceph/ceph-client.0.79052.log: 91.2% -- replaced with /var/log/ceph/ceph-client.0.48990.log.gz 2024-04-27T03:22:43.146 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73182.log 2024-04-27T03:22:43.146 INFO:teuthology.orchestra.run.smithi110.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.47220.log.gz 2024-04-27T03:22:43.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.33650.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.79052.log.gz 2024-04-27T03:22:43.146 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.80822.log.gz 2024-04-27T03:22:43.148 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62560.log 2024-04-27T03:22:43.148 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.84952.log 2024-04-27T03:22:43.148 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.73182.log: /var/log/ceph/ceph-client.0.62560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38960.log 2024-04-27T03:22:43.148 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.33650.log.gz 2024-04-27T03:22:43.148 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.84952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42500.log 2024-04-27T03:22:43.150 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.38960.log: 91.2% -- replaced with /var/log/ceph/ceph-client.0.62560.log.gz 2024-04-27T03:22:43.150 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.73182.log.gz 2024-04-27T03:22:43.150 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.84952.log.gz 2024-04-27T03:22:43.150 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27678.log 2024-04-27T03:22:43.150 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.38960.log.gz 2024-04-27T03:22:43.150 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80232.log 2024-04-27T03:22:43.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.42500.log: /var/log/ceph/ceph-client.admin.27678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46040.log 2024-04-27T03:22:43.151 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.27678.log.gz/var/log/ceph/ceph-client.0.80232.log: 2024-04-27T03:22:43.151 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.0.46040.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.42500.log.gz 2024-04-27T03:22:43.152 INFO:teuthology.orchestra.run.smithi110.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.0.46040.log.gz 2024-04-27T03:22:43.152 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.0.80232.log.gz 2024-04-27T03:22:43.169 INFO:teuthology.orchestra.run.smithi110.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-27T03:22:43.252 INFO:teuthology.orchestra.run.smithi077.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-27T03:22:43.914 INFO:teuthology.orchestra.run.smithi110.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-27T03:22:44.108 INFO:teuthology.orchestra.run.smithi110.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-27T03:22:44.112 INFO:teuthology.orchestra.run.smithi077.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-27T03:22:44.114 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-04-27T03:22:44.114 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m1.071s 2024-04-27T03:22:44.114 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m1.242s 2024-04-27T03:22:44.114 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.075s 2024-04-27T03:22:44.145 INFO:teuthology.orchestra.run.smithi110.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-27T03:22:44.146 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-04-27T03:22:44.147 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m1.090s 2024-04-27T03:22:44.147 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m3.065s 2024-04-27T03:22:44.147 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.195s 2024-04-27T03:22:44.147 INFO:tasks.ceph:Archiving logs... 2024-04-27T03:22:44.147 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/remote/smithi077/log 2024-04-27T03:22:44.148 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T03:22:44.352 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/remote/smithi110/log 2024-04-27T03:22:44.352 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T03:22:44.821 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-27T03:22:44.834 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-27T03:22:44.834 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T03:22:44.837 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T03:22:44.892 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T03:22:44.902 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T03:22:44.903 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T03:22:44.905 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T03:22:44.915 INFO:teuthology.orchestra.run.smithi077.stdout: remote refid st t when poll reach delay offset jitter 2024-04-27T03:22:44.915 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================== 2024-04-27T03:22:44.916 INFO:teuthology.orchestra.run.smithi077.stdout:+hv01.front.sepi 67.205.162.81 3 u 60 64 177 0.137 -2.523 2.477 2024-04-27T03:22:44.916 INFO:teuthology.orchestra.run.smithi077.stdout:+hv02.front.sepi 63.231.80.2 3 u 58 64 177 0.122 +1.907 2.544 2024-04-27T03:22:44.916 INFO:teuthology.orchestra.run.smithi077.stdout:*hv03.front.sepi 74.6.168.72 3 u 63 64 177 0.125 -0.556 2.675 2024-04-27T03:22:44.916 INFO:teuthology.orchestra.run.smithi077.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:22:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2024-04-27T03:22:44.918 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2024-04-27T03:22:44.918 INFO:teuthology.orchestra.run.smithi110.stdout:+hv01.front.sepi 67.205.162.81 3 u 51 64 177 0.101 -2.942 0.581 2024-04-27T03:22:44.918 INFO:teuthology.orchestra.run.smithi110.stdout:*hv02.front.sepi 63.231.80.2 3 u 58 64 177 0.083 +2.640 1.249 2024-04-27T03:22:44.918 INFO:teuthology.orchestra.run.smithi110.stdout:+hv03.front.sepi 74.6.168.72 3 u 53 64 177 0.087 -1.642 0.923 2024-04-27T03:22:44.918 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-27T03:22:44.919 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T03:22:44.930 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T03:22:44.931 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T03:22:44.973 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T03:22:45.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T03:22:45.058 INFO:teuthology.task.internal:Duration was 884.171793 seconds 2024-04-27T03:22:45.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T03:22:45.099 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T03:22:45.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T03:22:45.102 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T03:22:45.141 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-27T03:22:45.141 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-04-27T03:22:45.141 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T03:22:45.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-04-27T03:22:45.192 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T03:22:45.203 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-27T03:22:45.203 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T03:22:45.236 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T03:22:45.270 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-27T03:22:45.270 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T03:22:45.304 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T03:22:45.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T03:22:45.374 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T03:22:45.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T03:22:45.387 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T03:22:45.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T03:22:45.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T03:22:45.432 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T03:22:45.441 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-04-27T03:22:45.455 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-04-27T03:22:45.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-27T03:22:45.647 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T03:22:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:22:45.695 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T03:22:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T03:22:45.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T03:22:45.711 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T03:22:45.712 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/remote/smithi077 2024-04-27T03:22:45.712 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T03:22:45.760 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-26_21:40:02-rgw-squid-distro-default-smithi/7675650/remote/smithi110 2024-04-27T03:22:45.760 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T03:22:45.782 INFO:teuthology.task.internal:Removing archive directory... 2024-04-27T03:22:45.782 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T03:22:45.800 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T03:22:45.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T03:22:45.836 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T03:22:45.836 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T03:22:45.868 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T03:22:45.868 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T03:22:45.871 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T03:22:45.875 INFO:teuthology.orchestra.run.smithi077.stdout: 658464 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 03:22 /home/ubuntu/cephtest 2024-04-27T03:22:45.877 INFO:teuthology.orchestra.run.smithi110.stdout: 658628 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 03:22 /home/ubuntu/cephtest 2024-04-27T03:22:45.878 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T03:22:45.910 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T03:22:45.993 INFO:teuthology.run:Summary data: description: rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} duration: 884.1717925071716 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-27T03:22:45.993 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T03:22:46.046 INFO:teuthology.run:pass